Expert Forecasts and Calibration
Reduce optimism with reference-class forecasts and small expert ensembles, calibrate simulations with pilots, and track the gap between predictions and outcomes.
Written and reviewed by Eric Han · Co-founder & CTO, reopt
Key takeaways
- Published effects are inflated. For nudges, journal effects averaged 8.7 percentage points versus 1.4 in operational RCTs; selective publication and low statistical power explain about 70% of the gap.
- Forecasters are also influenced by published effects (NBER version). Median forecasts of nudge-unit effects were 4 points among academics and others and 1.95 among nudge practitioners, versus 1.38 observed.
- Averaging experts is powerful. Averaging five forecasts reduced error by about 33% compared with individual forecasts; averaging the full group reduced it by 45% (169 → 114 → 93 mean absolute error in a real-effort task; calculated).
- Optimistic demand forecasts are a longstanding problem. Rail ridership was overestimated by an average of 105.6%, with no improvement over 30 years.
- Simulations do not improve without recording forecasts and comparing them with outcomes.
An Outside View to Reduce Optimism
The inside view starts with “our plan”; the outside view starts with “the distribution of outcomes from similar efforts” (Kahneman and Lovallo, 1993). Reference-class forecasting has three steps (Flyvbjerg, 2006).
This book's conversion atlases and quantile tables serve as marketing reference classes. However, most numerical examples in the reference-class literature concern infrastructure costs; we found no empirical validation in marketing conversion. Its use here is an adaptation from another field.
| Evidence | Figures | Grade |
|---|---|---|
| Flyvbjerg et al., 2005 | Rail ridership overestimated by 105.6% on average; 84% of projects fell short by more than 20%. Half of road projects deviated by more than ±20%. No improvement over 30 years | B |
| UK Department for Transport cost-overrun distribution | Rail P50 40%, P80 57% | D (case) |
Discount Publicized Effects
DellaVigna and Linos (Econometrica, 2022) compare similar interventions (grade A).
- Journal papers8.7pp
- Median forecast: academics and others4pp
- Median forecast: nudge practitioners1.95pp
- Observed nudge-unit RCTs1.38pp
| Sample | Mean effect | Scale |
|---|---|---|
| Journal papers | 8.7 percentage points (33.4% lift) | 505,337 people |
| Nudge-unit RCTs | 1.4 percentage points (8.0% lift) | 23 million people |
Selective publication and low power explain about 70% of the gap. A reanalysis by Maier et al. (2022) reduced a nudge meta-analysis effect from d=0.43 to d=0.04 (CI 0.00–0.14) after correcting for publication bias.
These are government-nudge results, so the roughly sixfold ratio is not a universal rule. Borrow the reasoning: vendor case collections and conference presentations select for success.
Averaging a Small Group of Experts
DellaVigna and Pope (JPE, 2018) asked 208 academics to predict experimental results (grade A).
- Individual169
- Average of 5114
- Average of 2099
- Full-group average93
The correlation between the average forecast and outcomes was 0.77; 96% of individual experts performed worse than the average. Citation counts and seniority were unrelated to accuracy. Applying these real-effort results to marketing conversion is an analogy. We found no study directly evaluating experts' marketing-conversion forecasts.
How to Apply It
- Ask about five people across marketing, sales, and product to independently write down each stage's 10th percentile, median, and 90th percentile without seeing each other's estimates.
- Use their average as the prior's center and reflect disagreement in the interval.
- Compare with benchmarks. Ask for evidence if estimates are substantially higher.
- Record the forecast and compare it with outcomes next quarter.
Pilot Calibration Workflow
Pilot Size
These are approximate sample sizes for measuring a conversion rate at a specified precision (95% confidence; calculated). Cost is sample size × your CPC.
| Expected conversion | Margin of error | Required sample |
|---|---|---|
| 1% | ±0.5 percentage points | About 1,500 |
| 3% | ±1 percentage point | About 1,100 |
| 5% | ±1 percentage point | About 1,800 |
| 10% | ±2 percentage points | About 900 |
Smaller samples are still useful. The update formula works at any sample size; the interval simply narrows less. Use the sample-size calculator for A/B comparisons.
When You Lack Scale for an Experiment
- Keep a wide interval for the reported-to-incremental multiplier.
- Do not treat before/after comparisons as incremental evidence.
- Start on/off tests with channels you can switch off, such as affiliates or branded search.
A Forecast Scorecard
Each quarter, count how often outcomes fall inside the predicted 90% interval. Compare against the actual-count interval, not the expected-count interval.
| Coverage | Meaning | Action |
|---|---|---|
| Near 90% | Well calibrated | Maintain |
| Much lower | Intervals too narrow or center misplaced | Lower mean × and inspect the direction of bias |
| Almost 100% | Intervals too wide | Increase using your own data |
Do not tune concentration from coverage when counts are small
The simulator's expected-count 90% interval need not contain actual counts 90% of the time, even with perfect inputs. Under a correct model, wins from 10,000 cold emails have 0% coverage because no integer lies between 0.01 and 0.90; 100,000 emails give about 67%, and early SaaS paying customers about 75% (this book's calculations). Lowering k does not fix this. Compare actual-count intervals, and do not tune concentration from coverage below 20 outcomes. Score incremental forecasts only when a holdout is available.
What to Record
| Field | Example |
|---|---|
| Forecast date and owner | 2026-11-02, growth team |
| Target and budget | Meta traffic, $10,000 |
| Inputs and evidence | CVR mean 1.25%, Littledata fashion |
| Prior concentration | (mean × = 1.6) |
| Forecast interval | 16–344 incremental purchases, alongside the actual-count interval |
| Actual result | Fill in after measurement |
| Reason for the gap | Fill in after measurement |
After several quarters, this becomes an internal reference class more useful than public benchmarks.
Desktop App Conversion
Review downloadable-software funnels, Steam wishlist sales ratios, refunds, and installation warnings with explicit dates and samples.
Choosing a Forecasting Model
Compare Bass diffusion, BG/NBD, agent-based models, and MMM by their equations, data requirements, appropriate uses, and limitations.