the bad news

You are a terrible guesser.

I'm also a terrible guesser. As are most people, with the exception of some well-trained meteorologists and professional gamblers.

'Guessing' here refers to a properly calibrated probability estimate. If you were to guess the gross domestic earnings of The Goonies, your answer would probably be wrong, not through personal faults, but because the question tests exact domain knowledge. Rephrasing the question as:

“What are your high and low estimates for the gross domestic earnings of the The Goonies? What interval estimate of confidence would you give this range?”

The answer might be:

“Between $100,000 to $100,000,000, inclusive, with 90% confidence.”^[note]

Given enough data points in a sufficiently ideal world, the estimated confidence interval would match the observed accuracy. In other words, the range estimates given with 90% confidence should – on average and with minimal variance – be correct 90% of the time. This changes the exercise from a measurement of trivia knowledge to a measurement of ability to gauge uncertainty. However, much fewer than 90% of answers with 90% confidence will contain the actual value because humans habitually display overconfident estimates. What we know is about 10-35 percentage points less than what we think we know in this example.

cognitive bugs

Plenty of empirical research explores the limits and prejudices of probability estimates, and a few present theories on the potential causes, most of which can be found among other lists of cognitive biases.

• Anchoring: the wording and structure of the questions can decrease epistemic accuracy by up to 53 percentage points.
• Pattern-seeking habits of humans create the illusion of signal where there is only noisy data ^12..
• Prospect theory: small probabilities are overweighed, especially when attached to high-consequence events ^100-101..
• Our educational training emphasizes algebra and calculus as the end goals, not probability and statistics.
• We ignore prior probabilities.
• Many social and professional systems reward overconfidence.
• In situations determined primarily by chance, we often build narratives^[note] to coherently explain the events, giving the illusion of control. E.g., market fluctuations due to labor reports, portfolio performance due to investment strategies or combat effectiveness predicted by training exercises^[note].

These biases are to estimation as optical illusions are to psychometrics, where a simple change of the problem context causes a predictable change in the perceived reality. In general, humans have a very troubled relationship with uncertainty. We don't understand it instinctually, we don't communicate it well and we're willing to pay ^{Part I, Chapter II. 100-101.} to avoid it.

you should care

Even if you don't live in a region with legalized gambling or work in a forecasting profession, everyday failures of estimation hurt your quality of life, whether due to inaccurate project estimates, poor investments or being late to the next appointment. We make decisions based on uncertainty and imperfect knowledge, knowing much less than we think we know. As far as ubiquitous problems of human existence, it's right up there with communicable disease^[note].

More importantly, inability to accurately estimate closes the door to powerful tools of probabilistic thinking. With accurate prior probabilities, Bayesian prediction^[note] avoids the nuances of frequentist statistics, while allowing your mental model to adapt as the facts change. It's something which the Army and Air Force train, and M.D.s understand through years of experience. Along with the distance-rate-time equation, time-value equation and logical equalities, Bayes' Theorem is one of the those unreasonably effective structures of math, which internalizing will vastly improve your thinking.

mensa mea bona est

Overconfidence follows a predictable pattern. It is usual for difficult assessments (although slightly less for true/false tests ^p64.). In some cases, very easy questions inspire underconfidence. Two simple calibration techniques can help to correct this:

• Consider the reasons why your judgment might be wrong ^p228..
• Range estimates can reduce the anchoring effect of a point estimate, particularly by working towards a narrow range from an absurdly large range ^p64-5..

Things that don't fix overconfidence:

• More information. Paradoxically, providing more information to the problem increases one's confidence in the answer, but not accuracy.

Most importantly, feedback and iterative practice allow us to improve our estimation techniques, which is the purpose of this project. Select the number of questions you want and the quiz will give you instant feedback on your progress.

The more questions have been answered, the more reliable the results will be.

When choosing your confidence level, 50% confidence indicates that you have no idea which answer is correct. 100% indicates absolute certainty of the correct answer. The more questions you answer, the more accurate your calibration will be.

20 questions 40 questions 70 questions 100 questions

other examples

If you want to try other types of calibrated probability assessments:

• A list of them.
• Aumann Game: a manually scoring assessment.
• An automated quiz that produces a nifty graph.
• A range estimate quiz reproduced from Decision Traps, and also in Plous's book.

sources