A/B test significance
Is that conversion-rate lift real? Two-proportion z-test with a plain-language verdict.
p = 0.0769, above the 0.05 threshold. Keep running, or accept there may be no real difference.
Percentage points
The interval includes zero — no reliable difference
To detect this size of effect at 80% power
How it works
An A/B test asks whether the difference you observed is bigger than the noise you would expect from random assignment alone. The two-proportion z-test answers that directly.
p̂ = (cA + cB) / (nA + nB)
SE = √( p̂(1 − p̂) · (1/nA + 1/nB) )
z = (pB − pA) / SE What the p-value means — and what it does not
The p-value is the probability of seeing a difference at least this large if the two variants were actually identical. It is not the probability that B is better, and it is not the size of the effect. A tiny p-value on a 0.1% lift is still a 0.1% lift.
Stopping early is the classic mistake
Checking the p-value every day and stopping the moment it dips below 0.05 inflates the false positive rate dramatically — you are effectively running many tests and reporting the luckiest. Fix a sample size in advance and run to it. The “visitors needed” figure above is that number, for the effect size you are currently seeing.
Confidence interval over p-value
The interval tells you the range of true differences consistent with your data. If it spans zero, you cannot rule out “no difference”. If it runs from +0.1 to +8 percentage points, you have a winner but not much idea how big — which is itself useful to know.
Assumptions
The normal approximation needs a reasonable number of conversions in each arm — at least 5 expected successes and 5 failures is the usual rule. Visitors must be assigned randomly and counted once each; repeat visits and shared devices break the arithmetic.