LLM Synthetic Customers
Review how well LLM customers reproduce surveys and experiments, and distinguish message shortlisting from unsupported uses in sample sizing, segmentation, and conversion forecasts.
Written and reviewed by Eric Han · Co-founder & CTO, reopt
Key takeaways
- Synthetic customers have mostly been compared with survey responses. Few studies compare them with actual A/B tests, and we found none establishing accurate absolute conversion rates.
- Individual-level correlation for digital twins is around 0.2. Accuracy of 0.748 closes about 63% of the gap from random performance at 0.629 to retest accuracy at 0.817 (calculated).
- Twin standard deviations are lower than human ones for 93.9% of 164 outcomes. Treating synthetic responses as a real sample invalidates confidence intervals.
- Correlation with survey-experiment effects reaches 0.85 (Nature, 2026), but field megastudies average 0.33, with five studies ranging from 0.05 to 0.65. Those field studies concern vaccines, gyms, and nudges, not advertising.
- Use synthetic customers to shortlist messages and ideas before human testing. Do not use them for absolute conversion rates or segment differences.
Synthetic customers are LLMs instructed to answer surveys or experiments as customers. Digital twins receive a real person's responses and attempt to answer as that person would.
Three Levels of Prediction
“Accurate” can mean three different things. Evidence strength varies substantially by level.
- Survey-experiment effects (Nature 2026)0.85
- Survey megastudy average0.43
- Field megastudy average0.33
- Individual digital twins0.20
| Level | Representative figures | Assessment |
|---|---|---|
| Aggregate direction and ranking | Overall survey experiments r=0.85 (Nature 2026); field average r=0.33; direction accuracy 64% | Conditional use |
| Absolute effect magnitude | RMSE 10.9 percentage points; overestimation | Do not use directly |
| Segments and individuals | Individual r≈0.2 | Do not use |
Individual-Level Accuracy
| Study | Figures | Conditions | Grade |
|---|---|---|---|
| Peng, Toubia et al., megastudy | Accuracy 0.748, random 0.629, retest 0.817, correlation r≈0.20 | 19 studies, 164 outcomes, GPT-4.1 | B |
| Toubia et al., Twin-2K-500 | 71.72%, random 59.17%, retest 81.72% | N=2,058, 500 questions | B |
| Park et al. | GSS normalized accuracy 0.83 (raw 65.67%, self-consistency 79.53%); demographics only 0.74 | N=1,052, two-hour interviews | B |
Normalized accuracy of 0.83 or 88% is relative to people's own consistency. Include the random baseline: Peng's data give , and Toubia's give 56%.
Even twins built from roughly 500 questions are quickly matched by machine learning trained on a small sample. In Peng's study, XGBoost reached the twins' correlation with data from about 180 people.
Aggregate and Experimental Effects
| Study | Figures | Grade |
|---|---|---|
| Ashokkumar et al., Nature 2026 | 70 survey experiments, 469 effects: r=0.85 (adjusted 0.92), laypeople 0.84, combined 0.89. Systematic overestimation of effect sizes | B |
| Same paper's underlying data, 15 megastudies | Survey average r=0.43; field average r=0.33 (gym 0.05; vaccines 0.17, 0.49, 0.65; nudge unit 0.27) | B |
| Hewitt et al., 2024 preprint | Direction accuracy: surveys 79%, field 64%. Magnitude RMSE 10.9 points (laypeople 8.4); shrinking by 0.56 reduces it to 5.3. Published-version figures unverified | B |
| Chen, Hu, and Lu, 2025 | Direction accuracy 78% across 276 economics field-experiment papers. Preprint; only published conclusions | B |
| Cui et al. | Replication of main effects in 156 scenario experiments: 73–81%; interactions: 46–63%. Effect sizes 2–3 times larger. Significant results in 68–83% of originally null cases | B |
| Argyle et al., 2023 | Overall voting-choice correlations 0.90–0.94, but 0.02 among independents in 2020 | B |
Even the field results concern public health and behavioral nudges, not ads or purchases. The 0.56 shrinkage factor was estimated from that archive and should not be transferred to marketing. Estimate an adjustment from your own experiments.
Comparisons with Actual A/B Tests
| Study | Findings | Limits | Grade |
|---|---|---|---|
| Amazon persona agents | Direction accuracy 75–80% across 40 A/B tests | Ambiguous tests excluded; Amazon-affiliated authors; preprint | C |
| Upworthy headlines, LOLA | Across 17,681 A/B tests, an LLM alone struggled to choose the best headline; combining it with online learning was useful | Media headlines | B |
Evidence for shortlisting is growing, but these studies do not establish accurate absolute conversion rates.
Marketing Research
| Study | What worked | What did not | Grade |
|---|---|---|---|
| Brand, Israeli, and Ngwe | Fluoride toothpaste WTP within 50% of human estimates. Within-category fine-tuning improved predictions for new features | MacBook brand WTP had the opposite sign; tablet storage, RAM, and screen-size WTP overestimated by 70–312%; cross-category generalization worsened results | B |
| Maier et al., PyMC Labs and Colgate | Purchase-intent correlation attainment 90.2–90.6% with image stimuli and transformed free-text responses | Direct Likert responses 80–82%, distribution agreement KS 0.26–0.39; text stimuli 73.0%. One personal-care category | C |
| ConsumerSimBench | At best, captured 47.8% of real social responses | Missed more than half; Chinese-language setting | B |
| Toubia et al. | Downward-sloping demand at positive prices | Failed to reproduce 4 of 10 classic effects replicated by humans, including outcome bias, sunk-cost reversal, and Allais-type effects | B |
Segment failures are especially clear. In Brand et al., human RAM WTP was $17.80 for low-income and $19.40 for high-income respondents; GPT estimated −$4.40 and $99.60. Bisbee et al. report that 48% of regression coefficients differed significantly, and 32% of those even reversed sign.
Collapsing Variance
- Peng: twin SD was smaller than human SD in 154/164 outcomes (93.9%).
- Bisbee: pooled SD was 31.4 for humans versus 16.1 for synthetic respondents. Power calculations using synthetic SD and effect sizes suggested only 33 respondents for 99% power, almost an order of magnitude below the actual requirement.
- Tigre and Souto, 2026: even after calibration with human data, outcome-dependent response (MNAR) reduced 95% CI coverage for Twin-2K purchase rates to as low as 4.20%.
Do not treat the number of synthetic responses as a sample size.
Decisions by Use Case
| Use | Assessment | Evidence |
|---|---|---|
| Shortlisting copy and ideas before human tests | Conditional use | Brand, Maier |
| Ranking treatment directions | Conditional. Stronger for survey experiments, but field r=0.33 and direction accuracy 64%; weak evidence for ranking similar candidates. Combine with human forecasts | Nature 2026 (combined r=0.89), Hewitt et al., Upworthy |
| Absolute effect sizes | Do not use | Systematic overestimation, often 2–3 times |
| Absolute conversion rates | Do not use | No supporting evidence |
| Sample sizes and confidence intervals | Do not use | Bisbee, Peng |
| Segment differences | Do not use | Argyle, Brand; preprint interaction-effect r from −0.03 to 0.16 (adjusted 0.17–0.55) |
| Individual targeting | Do not use | r≈0.2 |
| Korean customers | Do not use before internal validation | Only survey/voting replications found, with large errors; no purchase-behavior replication found (Korean replications) |
Korean Replications
Studies reproducing Korean surveys and voting have begun to appear. We found no purchase-behavior replication, so do not use them for absolute conversion rates. Grades appear at the end of the first cell.
| Study | Findings |
|---|---|
| Kim and Cho, 2026, B | Overall MAE 13.8–17.0 percentage points. Binary-question correlations 0.81–0.91 coexist with large level errors. Gender/age-cell MAE of 15.1–18.3 points fell to 4.4–4.9 using 30% real responses for calibration, still worse than 3.6 using those responses directly. Synthetic-panel advantages remained only below about 250–860 real responses |
| Jang et al., 2026, B | Average group-level gap 29 points; between-group direction matched in 49 of 110 cases. Two personas in the same cell chose the same answer with 46–47% probability versus 4–8% for real respondents. Within-group variance disappeared. Preprint |
| Lee et al., 2025, B | Minimum-wage agreement among very progressive respondents: 45.2% actual versus 0% synthetic. Ideological extremes were exaggerated |
| Kang, 2026, B | Correct presidential winner in 3 elections (95% CI 29.2–100%); calibrated prompts reduced MAE from 36.8 to 7.1 points. Public election results may overlap training data; this is not consumer conversion. Preprint |
| Bae, 2026, B | Female representation in male-dominated occupations was 2.1–3.0 times official statistics in synthetic persona data. Correct marginal distributions can coexist with incorrect joint distributions. Preprint |
Operating Rules
A baseline worth comparing
Averaging five experts reduces individual forecast error by about 33% (169 → 114 mean absolute error in a real-effort task; calculated). Before building a synthetic-customer system, compare it against a small expert ensemble. In the Nature paper's megastudies, LLM and expert predictions were similar.
Will Better Models Solve This?
The evidence spans GPT-3 through GPT-5.2. Peng's study still found variance collapse and mean bias with 2025 models. Do not assume the problem will soon disappear. Repeat these replication tests when a new model becomes available.
Choosing a Forecasting Model
Compare Bass diffusion, BG/NBD, agent-based models, and MMM by their equations, data requirements, appropriate uses, and limitations.
Korean Cases: Advertising and Commerce
Review Naver log analyses, advertising-platform cases, store conversion, repeat-purchase filings, company experiments, and live commerce with denominators and evidence grades.