Free A/B test significance calculator: check your test result

It runs an unpooled two-proportion z-test, the standard approach for comparing two proportions. Its null hypothesis is that A and B convert equally. The formula divides the gap between the rates by its standard error, giving a z-score, then reads it off the bell curve: how surprising are the observed results if nothing differs? It's a normal approximation of binomial data, so distrust small samples.

Say it's a landing page headline test with the figures above. A converts 400 of 5,000 visitors (8.0%), B 460 (9.2%): in this example, a 15.0% relative lift, z of 2.14 and about 98.4% confidence, significant at 95%. If each version had 2,500 visitors at the same rates, confidence would drop to about 93.5%; at 1,000, to about 83.1%. Same lift, smaller sample, weaker verdict.

Confidence is one minus the one-sided p-value: the example's 98.4% means p of about 0.016, the chance of a gap this big if the null hypothesis were true. The calculator runs a one-sided test, a directional question: is the better version really better? A two-sided test asks if A and B differ significantly either way, which raises the bar for z from 1.645 to 1.96. Agree one-tailed or two-tailed before launch.

Set the required sample size before launch from your baseline conversion rate, the minimum detectable effect (MDE, the smallest lift worth finding) and statistical power, usually 80% (the chance of catching a real effect if there is one; missing it is a type II error). For example, an 8% baseline and a 15% relative MDE need about 6,750 visitors per variant (one-sided, 95%). Statista's 2025 data puts the average e-commerce conversion rate at under 2%; at 2%, the same lift needs roughly 29,000 per variant.

Peeking is the worst. If you check daily and stop the first time it shows 95%, your false positive rate ends up far above 5%: the z-test assumes a fixed sample size, and random chance gets many tries. With many variants, one can win due to chance alone, so test one variant at a time or apply a Bonferroni correction (divide the significance threshold by the number of comparisons). And watch a guardrail metric, one that mustn't get worse, such as refunds.

HubSpot's 2026 State of Marketing report found that 50% of marketers use conversion rate optimisation (CRO), their second most-used optimisation technique, and that visual assets are what they test most. Without a log, the same image test gets rerun next year. Low-traffic B2B pages are harder: on a demo page with 400 visitors a month converting at 5%, for example, the visitors needed to detect a 10% lift come to about 49,000, roughly ten years. Test bolder changes there, or ask customers directly.