Effect Sizes of Ecommerce Levers

Key takeaways

• Estimates from Qubit customer experiments place 90% of effects within ±1.2%, described by the authors as a rough indicator. Average effects are small enough that scarcity at +2.9% is the top category. Researchers disagree on how common unusually large effects are.
• Speed effects differ more than tenfold by evidence source. A Bing RCT implies +0.6% revenue per 0.1-second improvement; Google-commissioned observational work reports +9.1% product-detail-to-cart conversion and +9.2% spending.
• Free-shipping evidence is mixed. Offer/message experiments average −0.2% revenue per visitor, while a natural experiment increased revenue and orders.
• Coupon emails increase spending outside redemption. Randomized experiments found +37% spending, with about 90% of the increase unrelated to coupon use.
• Detecting a 1% relative difference requires about 120,000 buyers per variant; most stores cannot reliably judge small effects.

The Reality of Experimental Effects

Browne and Jones (2017) analyze about 6,700 retail and travel experiments. Below are category-level mean effects among roughly 2,600 classified experiments (Qubit-affiliated authors; independent verification by PwC UK stated; no journal publication details; grade B).

• Scarcity messages2.9%
• Social proof2.3%
• Urgency1.5%
• Exit prevention1.1%
• Product recommendations0.4%
• Color0.0%
• Free-shipping offers / messages-0.2%
• Buttons-0.2%
• CTA wording-0.3%
• Mobile navigation-1.7%

Only eight of 29 categories had more than a 50% probability of a positive effect. The authors find little evidence of double-digit revenue gains from a single treatment. The top 0.1% of customers were excluded, and the sample consists of experimentation-tool customers.

In 2,101 Optimizely experiments, Berman et al. (2018) report a mean lift of 11.2% but a median of 0.15% (click-based, December 2018 version). A few extreme values drive the mean. Azevedo et al. (JPE, 2020) also find extremely heavy tails in Bing experiments, whereas Browne and Jones find thin tails. Most effects being small is common ground; the magnitude of rare big wins remains debated.

Limits for Small Sites

Browne and Jones estimate about 120,000 buyers in the control and each variant to detect a 1% relative conversion lift. This book's sample formula, using 2% baseline conversion, 1% relative lift, and 80% power, gives about 154,000 buyers: the same order of magnitude (calculated). Few stores can support this. Small stores should prioritize potentially larger changes in pricing, assortment, and offers over experiments intended to distinguish ±1% effects.

Page Speed

Bing is not ecommerce, and Deloitte did not randomize. Faster sites may also have stronger brands or higher-income customers, confounding observational estimates. Google commissioned the research. Public evidence for ecommerce RCT effect sizes is limited. Plan with a range and narrow it through internal experiments. If the store lacks the sample to judge the effect, set the planning value to zero and do not book the upper bound as expected profit (author judgment).

Reviews

Observational results from Spiegel Research Center and PowerReviews concern one premium gift retailer (grade C).

Popularity may drive both reviews and purchases. These are product-level relative effects for the transition from zero to five reviews, not sitewide conversion lifts. Use these findings only as directional evidence.

Shipping and Additional Costs

Effects depend on format, thresholds, and category. Run your own experiment rather than applying one effect size.

Blake et al., StubHub found spending about 21% higher when fees appeared at checkout rather than upfront (grade A). This demonstrates a revenue effect of hiding costs, but it can be a regulated dark pattern. It is not a recommendation to delay total-price disclosure.

Discounts and Email

Across 70 randomized experiments at a ticket platform, Sahni, Zou, and Chintagunta (2017) found promotional-period spending +37%, with about 90% of the increase outside coupon redemption. Spending remained +25% in the week after expiry (institutional summary, grade A). Redemption rate alone misses much of the effect.

Where Levers Work

In 2,732 experiments across 252 companies, Miller and Hosanagar (2020) found the largest effects for price promotions and category/product-listing pages. Pricing worked earlier in the funnel; shipping later. Only rankings were published (C).

Simulation Inputs

Planning values are author judgments informed by evidence. Do not simply add effects from multiple levers: overlapping exposure to the same people often makes the combined effect smaller.

Do not use Spiegel's +270%, Baymard's 35%, or Vodafone's +8% single case as direct inputs. Company-reported Korean experiments appear in Korean Cases.