“Are judges highly skilled mechanics who make rational and logical decisions? Or are they intuitive hunch-makers who feel their way to decisions that they later justify with deliberation?”
Chief Justice John Roberts used a now-famous umpire analogy during his confirmation hearing in September 2005 to describe the role of a judge, or justice, as he saw it: “Judges are like umpires. Umpires don’t make the rules they apply them.” This view of a judge’s role, which has been heavily debated, paints a picture of the judge as a neutral and impartial arbiter. It brings to mind the image of a judge who decides disputes entirely rationally and consistently, based only on a case’s objective merits.
However, a growing amount of research suggests that this picture of a judge—as an impartial and wholly rational arbiter—is not quite correct, or at minimum, incomplete. Rather, empirical evidence suggests that judges are in fact “constrained by the boundaries of human cognition.”
For instance, Peer and Gamliel reviewed several studies whose findings suggest that “irrelevant factors that should not affect judgment might cause systemic and predictable biases in judges’ decision-making processes in a way that could be explained using known cognitive heuristics and biases.” Rachlinski and Wistrich, the latter a former federal judge himself, conducted research involving one thousand judges and found that judicial decisions run the risk of being less consistent than one might think, due in part to a “susceptibility to framing … caused by uncritical reliance on heuristic processing, which can be exacerbated by time pressure.”
The extent to which judges rely on intuition, and are thus susceptible to heuristic-based persuasion, can be predicted fairly well by cognitive reflection tests (CRTs).The purpose of the CRT is to “test a respondent’s ability to suppress intuition in favor of deliberation in a setting where intuition is misleading.” Essentially, a CRT contains questions that are simple but require some deliberate processing; a reliance on mental shortcuts and intuition will typically lead to a wrong—but a predictably wrong—answer.
For example: A bat and a ball cost $1.10 in total. The bat costs $1.00 more than the ball. How much does the ball cost?  Most people immediately answer that the ball costs 10 cents.  Intuitively this question seems obvious, but the point is that here, intuition leads to the wrong answer. If the ball costs 10 cents, the bat must cost $1.10, leading to a total of $1.20. Instead, getting to the correct answer (the ball costs 5 cents) requires suppression of the initial intuitive response. A typical CRT has three of these sorts of questions on it. 
Studies on judges showed that “judges performed about as well as the most educated adults on the CRT” —not terribly well, answering on average just over 1 out of 3 questions correctly. 
A low CRT score suggests that judges, when making decisions in a generalized setting at least, often make intuitive decisions when they should be reflective.
Of course, a judge may make decisions differently in court than when making ordinary, non-legal decisions. And, judges can be forgiven for being bad at math. 
But, the CRT is remarkably robust, and it does actually predict fairly well the extent to which those who score well (engineers), or poorly (judges), use deliberate or intuitive reasoning in other contexts.  In recent years, studies have been done showing that judges are inclined to rely on intuitive judgments and only have a mild disposition toward overriding them.
Behavioral Science, Democratized
We make 35,000 decisions each day, often in environments that aren’t conducive to making sound choices.
At TDL, we work with organizations in the public and private sectors—from new startups, to governments, to established players like the Gates Foundation—to debias decision-making and create better outcomes for everyone.
1. Guthrie et al., Judicial Intuition, at 2.
2. 9/12/05: John Roberts’ Baseball Analogy, ABC News (Sep. 12, 2005), https://abcnews.go.com/Archives/video/sept-12-2005-john-roberts-baseball-analogy-10628259.
3. Compare Getting Beyond Balls and Strikes, NY Times (Oct. 23, 2018), https://www.nytimes.com/2018/10/23/opinion/getting-beyond-balls-and-strikes.html (arguing against the view that judges are umpires) and Kagan: ‘Umpire’ metaphor suggests judges are robots, Politico (June 30, 2010, 9:50 AM), https://www.politico.com/blogs/politico-now/2010/06/kagan-umpire-metaphor-suggests-judges-are-robots-027873 (arguing against the view that judges are umpires) with Is a Judge a Player or an Umpire? Heritage Foundation (Aug. 9, 2018), https://www.heritage.org/courts/commentary/judge-player-or-umpire (arguing in favor of the view that judges are umpires).
4. See Peer & Gamliel, Heuristics and Biases in Judicial Decisions, at 114.
5. See generally Peer & Gamliel; Rachlinski & Wistrich, Judging the Judiciary.
6. Peer & Gamliel, Heuristics and Biases in Judicial Decisions, at 114.
8. Jeffrey J. Rachlinski & Andrew J. Wistrich, Gains, Losses, and Judges: Framing and the Judiciary, 94 Notre Dame L. Rev. 521, 573 (2018).
9. Guthrie et al., The Hidden Judiciary, at 1498.
11. See id. at 1497.
12. Id. at 13.
15. Guthrie et al., The Hidden Judiciary, at 1498.
17. Rachlinski & Wistrich, Judging the Judiciary, at 13–14.
About the Author
Tom is a Co-Founder and Managing Director at The Decision Lab. He is interested in the intersection of decision-science and the law, with a focus on leveraging behavioral research to shape more effective public and legal policy. Tom graduated from Georgetown Law with honors. Prior to law school, Tom attended McGill University where he graduated with First Class Honors with majors in Philosophy and Psychology.