Superforecasting

Category: Decision-Making & Strategic Thinking Source: Philip Tetlock & Dan Gardner - "Superforecasting: The Art and Science of Prediction" (2015) Practitioner Score: 46/50 (Tier 1 Canonical)

Overview

Superforecasting is a systematic methodology for making accurate probabilistic predictions, developed through Philip Tetlock's Good Judgment Project. The research identified "superforecasters" - individuals who consistently outperform experts, pundits, and even intelligence analysts by 30% or more. The framework codifies their techniques into 10 actionable commandments plus deliberate practice protocols.

Core Insight: Prediction accuracy is a learnable skill. By combining rigorous process (break problems down, update beliefs incrementally, balance outside/inside views) with calibrated probabilistic thinking and error analysis, anyone can dramatically improve forecasting ability.

Evidence: Good Judgment Project participants trained in these techniques beat CIA analysts with classified information access.

When to Use

Strategic decisions - Market entry, product launches, competitive moves
Resource allocation - Investment decisions, hiring plans, capacity planning
Risk assessment - Project timelines, crisis likelihood, threat analysis
Competitive intelligence - Predicting competitor actions, market shifts
Long-range planning - Technology adoption curves, regulatory changes

Anti-patterns:

Decisions already made (confirmation bias trap)
Binary yes/no thinking without probability ranges
One-shot predictions without learning feedback
High-emotion situations without cooling-off period

How to Execute

Step 1: Triage - Choose Forecast-Worthy Questions

Action: Focus effort on questions where hard work pays off

Skip "clocklike" questions: Simple rules/trends suffice (e.g., "Will sun rise tomorrow?")
Skip "cloud-like" questions: Too random even for models (e.g., "Will World War III start?")
Target Goldilocks zone: Difficult but tractable with analysis
Output: Prioritized list of forecastable questions

Step 2: Fermi-Style Decomposition

Action: Break intractable problems into tractable sub-problems

Identify knowable parts: What can be researched or estimated?
Expose unknowables: Flush ignorance into the open
Example: "Will product X succeed?" → market size × conversion rate × pricing × competition
Output: Hierarchical breakdown with researchable components

Step 3: Balance Outside View (Base Rates) and Inside View

Action: Start with reference class, adjust for specifics

Outside view first: How often do things of this sort happen in situations of this sort?
Find comparison class: Similar products, markets, technologies
Inside view adjustment: What makes this case unique?
Output: Base rate probability + reasoned adjustments

Step 4: Incremental Belief Updating

Action: Update forecasts as new evidence arrives - not too much, not too little

Avoid under-reacting: Ignoring genuinely new information
Avoid over-reacting: Jumping to conclusions from noisy signals
Bayesian mindset: P(H|E) = P(E|H) × P(H) / P(E)
Output: Revised probability with explicit reasoning

Step 5: Consider Clashing Causal Forces

Action: Map arguments for AND against your thesis

Steel-man opposition: Understand counterarguments deeply
Force interaction: How do conflicting factors balance?
Example: "AI adoption" → Cost savings (pro) vs. Implementation complexity (con)
Output: Two-column list of forces with relative weights

Step 6: Granular Probability Estimates

Action: Translate vague hunches into numeric probabilities

Avoid vague language: "Likely" means what exactly?
Use fine gradations: 55% vs. 60% forces precision
Calibration practice: Track how often your 70% predictions come true
Output: Numeric probability (e.g., 68%) with confidence range

Step 7: Balance Under/Overconfidence

Action: Manage trade-off between decisiveness and humility

Calibration: Are your 80% predictions correct 80% of the time?
Resolution: Can you distinguish 60% from 80% events?
Avoid extremes: "Definitely" (99%+) and "No way" (1%-) rarely justified
Output: Calibrated probability that neither overstates nor understates certainty

Step 8: Learn from Errors Without Hindsight Bias

Action: Analyze mistakes while resisting "I knew it all along"

Pre-mortem: Before outcome, write why forecast might fail
Post-mortem: After outcome, compare to pre-mortem (not current knowledge)
Brier score tracking: Measure accuracy over time
Output: Error log with root cause analysis

Step 9: Leverage Team Wisdom

Action: Master collaborative forecasting dynamics

Perspective-taking: Reproduce others' arguments to their satisfaction
Precision questioning: Help clarify without judgment
Constructive confrontation: Disagree without being disagreeable
Output: Team forecast incorporating diverse viewpoints

Step 10: Master the Error-Balancing Bicycle

Action: Treat commandments as guidelines requiring constant judgment

No rigid rules: Every situation is unique
Deliberate practice: Forecasting is skill built through repetition
Feedback loops: Clear, unambiguous results inform learning
Output: Continuous improvement trajectory

Real-World Examples

Good Judgment Project (2011-2015):

Superforecasters beat intelligence analysts by 30%
Ordinary people trained in these methods outperformed experts
Result: Validated that forecasting is a learnable skill

Prediction Markets (Metaculus, Good Judgment Open):

Calibrated forecasters consistently identify probability ranges
Aggregated predictions outperform individual experts
Result: Operational use in policy, business, research

Tech Industry Product Forecasting:

Decompose adoption rates into addressable market × conversion × retention
Update predictions as beta data arrives
Result: Better resource allocation, realistic roadmaps

Integration Points

Complements:

Brier Score: Measures superforecasting accuracy quantitatively
Fermi Estimation: Powers Step 2 decomposition
Bayes' Theorem: Mathematical foundation for belief updating
Calibration: Essential skill for Step 7 confidence management
Base Rate Analysis: Core of Step 3 outside view

Contrasts with:

Expert Intuition: Systematic process vs. gut feel
Punditry: Probabilistic humility vs. confident pronouncements
Binary Thinking: 65% vs. "yes/no"

Common Pitfalls

Pitfall 1: Anchoring on Initial Estimate

Warning sign: Forecast barely moves despite major news
Fix: Explicit belief updating protocol after each information update

Pitfall 2: Ignoring Base Rates

Warning sign: "This time is different" without evidence
Fix: Always start with outside view reference class

Pitfall 3: Overconfidence in Extremes

Warning sign: Many forecasts at 5% or 95%
Fix: Force justification for extreme probabilities, track calibration

Pitfall 4: Confirmation Bias in Research

Warning sign: Only seeking evidence supporting initial view
Fix: Actively search for disconfirming evidence (Step 5)

Pitfall 5: No Feedback Loop

Warning sign: Making predictions but never tracking outcomes
Fix: Maintain prediction log with dates, probabilities, and resolutions

Validation Checklist

[ ] Question is in Goldilocks zone (neither trivial nor impossible)
[ ] Problem decomposed into researchable sub-components
[ ] Base rate identified from reference class
[ ] Both supporting and opposing forces mapped
[ ] Probability is numeric and granular (not vague language)
[ ] Calibration tracked over time (70% predictions = 70% accuracy)
[ ] Forecast updated as new information arrives
[ ] Pre-mortem written before outcome known
[ ] Team input incorporated through structured dialogue

Key Metrics

Brier Score: Primary accuracy measure (0 = perfect, 2 = worst)

Formula: (1/N) Σ(forecast - outcome)²
Target: < 0.20 for well-calibrated forecaster

Calibration: Do your X% predictions happen X% of the time?

Plot predicted probability vs. observed frequency
Perfect calibration = diagonal line

Resolution: Can you distinguish different probability levels?

Difference in outcomes between 60% and 80% forecasts
Higher resolution = better discrimination