THE SUPREME COURT: This man whom James Madison tried to put on the court declined, then followed him into the White House