The Cognitive Reflection Test: Answers and What They Mean

The three questions

These are the items exactly as Shane Frederick published them in 2005. Try them before you read on — the article is more interesting if you have your own answers to compare against.

1. A bat and a ball cost $1.10 in total. The bat costs $1.00 more than the ball. How much does the ball cost? ___ cents
2. If it takes 5 machines 5 minutes to make 5 widgets, how long would it take 100 machines to make 100 widgets? ___ minutes
3. In a lake, there is a patch of lily pads. Every day, the patch doubles in size. If it takes 48 days for the patch to cover the entire lake, how long would it take for the patch to cover half of the lake? ___ days

Cognitive Reflection Test answers

The correct answers are 5 cents, 5 minutes and 47 days.

Why the intuitive answers are wrong

Bat and ball. If the ball cost 10 cents, the bat — costing $1.00 more — would cost $1.10, and the pair would total $1.20, not $1.10. The gap between $1.00 and 10 cents is only 90 cents. Work it properly: if the ball is x, the bat is x + 1.00, so 2x + 1.00 = 1.10, and x = 0.05. The ball costs 5 cents and the bat costs $1.05.

Widgets. Five machines take five minutes to make five widgets, so each machine makes one widget in five minutes. Adding machines adds output, not time. A hundred machines each make one widget in the same five minutes. The number 100 in the question is a distraction; the answer never changes.

Lily pads. The patch doubles every day. Whatever covers the lake on day 48 must have covered exactly half of it on day 47, because doubling half gives you the whole. The answer is 47. Halving 48 gives 24, which is the intuitive response and would only be right if the patch grew at a constant rate rather than doubling.

What does the Cognitive Reflection Test measure?

Not arithmetic. Every item is solvable by an eleven-year-old. What the CRT measures, in Frederick's own framing, is “the ability or disposition to reflect on a question and resist reporting the first response that comes to mind.”

Each item is engineered so that an incorrect answer is generated automatically and impulsively, while the correct answer requires you to notice that the automatic one is wrong and suppress it. Psychologists describe this as the difference between two modes of thinking: a fast, effortless, associative process, and a slower, deliberate, effortful one — the distinction Daniel Kahneman popularised as System 1 and System 2.

That is why it works even on people who could trivially do the algebra. Knowing how to check is not the same as checking.

What is a good CRT score?

Your score is simply how many of the three you answered correctly, from 0 to 3. Frederick administered the CRT to 3,428 respondents across 35 separate studies over 26 months beginning in January 2003, mostly undergraduates at universities in the American midwest and northeast.

The overall mean was 1.24 out of 3. Only 17% of respondents answered all three correctly. Fully 33% — a third of a heavily university-educated sample — answered none correctly.

Two things stand out. Even at MIT, fewer than half the students got all three right. And a group of people picnicking by the Charles River before a fireworks display outperformed Harvard undergraduates. Small samples explain some of that, but the broader lesson holds: this is not a test that education reliably defeats.

Is the Cognitive Reflection Test an IQ test?

No, though it is related to one. Frederick compared CRT scores against several established measures of cognitive ability, including the Wonderlic Personnel Test, self-reported SAT and ACT scores, and the Need for Cognition scale. The CRT correlated with SAT scores at about r = .44, and with SAT-math at about r = .46 — a real relationship, but far from identity.

A proper IQ test takes an hour, samples many distinct abilities, and is normed on a representative population. The CRT takes two minutes, has three items, and measures one narrow disposition. It sits in a wider family of cognitive tests, most of which measure something quite different again. What makes it remarkable is how much it predicts for its length: CRT scores relate to patience in choosing between rewards, to risk preferences, and to susceptibility to a range of judgement biases.

I already knew the answers. Is my score meaningless?

Almost every article that publishes the CRT answers adds a warning that you have now ruined the test for yourself. The evidence does not clearly support that.

It is true that people who have encountered the items before score higher on them. The disputed question is whether that inflation destroys what the test predicts. Bialek and Pennycook (2018) examined precisely this across six studies with roughly 2,500 participants and seventeen outcome variables — including numeracy, susceptibility to heuristics and biases, and receptivity to pseudo-profound nonsense. They found similar correlations between CRT scores and those outcomes for experienced and inexperienced participants alike, and concluded that the CRT is robust to repeated exposure and need not be abandoned as an individual-difference measure.

For your own purposes, though, the honest position is simpler: once you know that the bat costs $1.05, your answer to that item no longer tells you anything about your own reflectiveness. The finding above concerns what CRT scores predict across large samples, not what a remembered answer tells one person about themselves.

Why do men score higher on the CRT?

They do, and Frederick reported it plainly: men scored significantly higher than women on the CRT. This finding is often quoted without its context, so here is the context.

In the same samples there were no significant sex differences on any other cognitive measure, except a modest difference on SAT-math corresponding to national averages. Women scored slightly higher than men on the Wonderlic, administered under identical conditions. The gap on the CRT persisted even after controlling for SAT-math scores.

Frederick's own reading was that the CRT's items have mathematical content, and that the difference may reflect mathematical ability or interest rather than general reflectiveness. What the result does not show is a difference in intelligence. Research on general intelligence finds no meaningful difference in average ability between men and women — only greater variability among men. A three-item test with mathematical framing is a narrow instrument, and a sex difference on it is a fact about that instrument, not about intelligence.

Criticisms of the CRT, and the CRT-2

The test has two well-documented weaknesses.

• Overexposure. The original three items are among the most famous in psychology. Large numbers of participants in common research pools have seen them before, which adds noise even if it does not destroy validity.
• Numeracy confound. All three items are mathematical. Scores are therefore entangled with numerical ability, which is not what the test claims to measure.

Thomson and Oppenheimer (2016) developed the CRT-2 in response, using items that trigger an intuitive lure without relying on arithmetic. Their alternate form appears to depend less on numeracy while measuring a closely related construct.

None of this makes the original CRT worthless. It makes it a short, sharp, well-validated measure of one specific thing — with known limits, which is more than can be said for most tests circulating online. If you want to try reasoning problems that have not been in every psychology textbook for twenty years, our practice IQ questions are a better place to start.

Myths and facts about the Cognitive Reflection Test