Reported and Incremental Conversions
Use randomized experiments at eBay, Meta, and Google to assess the gap between platform reporting and actual effects, and choose adjustment multipliers for simulations.
Written and reviewed by Eric Han · Co-founder & CTO, reopt
Key takeaways
- Meta seven-day last-click conversions were 1.2 times experimental incremental conversions for ecommerce, 1.5 for retail, and 2.0 for travel: averages for large US advertisers in 2019–20.
- Causal adjustment with observational data still leaves large errors. Machine-learning lift estimates were 2.9–4.8 times RCT estimates; propensity-score matching gave 6.0–12.8 times (ratios of medians, calculated).
- eBay nonbrand-search ROI exceeded 1,000% in regressions but was −63% experimentally. New and infrequent customers did show positive effects.
- Branded-search conclusions depend on context. Effects are small without competitors; when only competitors bid, they may take 18–42% of clicks, versus 1–5% when the brand also advertises.
- Vendor experiments can point in the opposite direction. A YouTube estimate found 3.4 times platform-reported results, so use channel-specific multipliers. For channels other than Meta, treat them as unknown until you run internal experiments.
Why Adjust?
People likely to see an ad are often already likely to buy. Platform-attributed conversions count purchases by exposed people, not additional purchases caused by advertising. Using benchmark CVR directly in a simulation overstates the actual effect.
The Most Direct Multiplier: Meta
Gordon, Moakler, and Zettelmeyer, PIE compare last-click conversions per dollar with incremental conversions per dollar across 2,226 RCTs from 998 Meta experiments (Meta-affiliated coauthor, grade A).
- Travel2.0×
- Retail1.5×
- Ecommerce1.2×
For travel, roughly half of last-click conversions would have happened without the ads. The sample covers US advertisers from November 2019 to March 2020, with at least one million test-group users and 5,000 conversions; purchase, cart, and page-view events are mixed. It predates iOS tracking restrictions and does not represent small advertisers.
The issue is systematic inflation, not an absence of signal. Industry-specific calibration can therefore help. The overall sample multiplier is 1.33.
- Using raw last-click values as incremental estimates gives out-of-sample of 0.19 (0.90 for ecommerce; worse than predicting the mean for retail and travel). Feeding the same last-click metric into an RCT-calibrated model raises it to 0.85.
- Last-click and incremental outcomes have high rank correlation, ρ=0.89.
- Last-click-based continue/stop decisions disagreed with RCT decisions 12–20% of the time.
An estimated incremental-CAC distribution is also available. Across more than 70,000 Meta purchase-optimized campaigns, the median was $38.16 and the 10th–90th percentiles spanned $5.86–$241.95. Optimizing for clicks instead of purchases raised CAC 31% to $49.93 (Wernerfelt et al., Marketing Science 2025, two Meta-affiliated coauthors, grade A).
How Wrong Are Observational Estimates?
Gordon, Moakler, and Zettelmeyer, Close Enough? estimated the same campaigns with RCTs and observational methods across 663 experiments (Meta-affiliated coauthor, grade A). These multipliers compare statistically adjusted models with RCTs, not platform reports, and measure lift rather than conversion counts. Do not mix them with the previous table.
- Propensity-score matching64%
- Double machine learning24%
- Randomized experiment5%
| Metric | Median RCT lift | Double machine learning | Propensity-score matching |
|---|---|---|---|
| Page views | 29% | 83% (about 2.9×) | 173% (about 6.0×) |
| Add to cart | 18% | 58% (about 3.2×) | 176% (about 9.8×) |
| Purchases | 5% | 24% (about 4.8×) | 64% (about 12.8×) |
These errors remain even with more than 5,000 user features. Lift also declines deeper in the funnel: estimated purchase lift was not statistically distinguishable from zero in 40% of RCTs.
Search: Context Can Reverse the Conclusion
| Evidence | Conditions | Result | Grade |
|---|---|---|---|
| eBay nonbrand | Ads stopped for 60 days in regions representing 30% of US traffic | Regression ROI 4,173% and 1,632%; experimental −63% (95% CI −124% to −3%) | A |
| eBay branded | Yahoo! / MSN branded search stopped | 99.5% of traffic retained through organic search | A |
| eBay customer frequency | At least three transactions in the prior year | No effect; positive effects only among new/infrequent customers | A |
| Bing branded | No competitors | Clicks +1–4% | A |
| Bing branded | Brand absent; competitors bidding | Competitors capture 18–42% of clicks | A |
| Bing branded | Brand advertising; competitors bidding | Competitors capture 1–5% of clicks | A |
| Google incremental clicks | Cross-industry internal research, 2011, over 400 pause studies | 89% of paid clicks incremental; clicks, not conversions | B |
| Google by organic rank | No organic result versus rank 1 | 99% versus 50% incremental clicks | B |
Do not transfer eBay's result directly
eBay was already a widely known brand, a condition the authors acknowledge. Do not apply −63% to an unknown startup. Model brand awareness, competitor bidding, and new-customer share separately.
Display and Retargeting
Johnson, Lewis, and Nubbemeyer analyze 432 Google display experiments (Google-affiliated coauthor, grade A).
- Median site-visit lift: 16.6% (10th–90th percentiles −1.1–215.6%); conversion lift: 8.1% (−8.9–83.4%).
- Positive significant conversion lift in 53 of 184 experiments, or 29% (calculated).
- A 10% visit lift translates into 5.1% conversion lift. Incremental visitors convert less well than existing visitors.
When Academic and Vendor Results Disagree
| Source | Channel | Incrementality versus reporting | Sample |
|---|---|---|---|
| PIE (academic) | Meta seven-day last click | Reporting overstates by 1.2–2.0× | 2,226 user-level RCTs |
| Haus | Meta seven-day click, D2C | Reporting understates by 15% | 640 geographic experiments, own customers |
| Haus | YouTube | 3.4 times reported results | 190 geographic experiments, conversion-optimized campaigns, post-campaign window included, own customers |
Experimental units, outcomes (platform conversions versus D2C revenue), attribution windows, customer mix, and measurement periods differ; PIE uses 2019–20 data. These differences do not establish that either side is wrong. Haus sells incrementality measurement. Use ranges instead of choosing one figure.
Simulation Inputs
| Channel | Input: reported ÷ incremental | Evidence and conditions |
|---|---|---|
| Meta conversion campaigns | 1.2–2.0 | PIE. Including Haus at the lower end gives about 0.85–2.0 (calculated). Interpreting Haus's 15% understatement as 1.15 gives a lower bound of 0.87 |
| Nonbrand search, well-known brand | Substantial discount indicated | eBay; separately calculate new-customer share |
| Nonbrand search, new brand | Unknown until internal experiment | No public multiplier |
| Branded search | Separate by competitor bidding | Bing, eBay |
| YouTube / TikTok / LinkedIn | Unknown until internal experiment | No academic RCT multiplier found |
| YouTube reference only | Vendor value about 0.29, inverse of 3.4 (calculated) | Haus customers, post-campaign window included; not an input |
These ranges are author judgments, not universal laws. For channel ordering see Stella's incremental ROAS, applying the author's recommended 15–20% discount.
Do not use eBay's −63% for new brands, or machine-learning / propensity-score-matching lift as an input.
Limits of Measurement Itself
Lewis and Rao (QJE, 2015) analyze 25 RCTs totaling $2.8 million. Median ROI standard errors were 26.1% in retail and 115% in brokerage. The text states that distinguishing +50% ROI from zero requires a campaign nine times the median size, and distinguishing a 10-point gap requires 62 times (grade A). The original table's retail medians are 2.5 times and 62 times. With small budgets, even RCT estimates remain uncertain. This is why the book recommends intervals.
Paid Channel Baselines
Establish pre-spend baselines using industry CTR, CPC, CVR, and CPL for Google search, Meta, TikTok, and LinkedIn, and identify gaps for Korean channels.
Creative, Retargeting, and Saturation
Review experimental effect sizes for targeting fit, messaging, retargeting timing, frequency, and budget saturation, and apply them to simulations.