What test does this calculator use?

A two-proportion z-test (also called a two-sample z-test for proportions). It compares conversion rates between two independent groups using a pooled standard error and reports the two-tailed p-value.

What is statistical significance?

A result is statistically significant when the probability of observing the difference by random chance (the p-value) is below your chosen threshold. At 95% confidence the threshold is p < 0.05, meaning there's less than a 5% chance the difference is noise.

What confidence level should I use?

95% is the industry standard for A/B testing and survey comparisons. Use 99% when the cost of a wrong decision is very high (medical, financial). Use 90% only for early-stage exploration where false positives are cheap.

My result is significant but the lift is tiny — should I ship?

Statistical significance is not the same as practical significance. With a large enough sample, even a 0.1% lift can become significant. Always weigh the absolute lift, implementation cost and downstream risk against the p-value before shipping.

Is a two-tailed or one-tailed test better?

Two-tailed (used here) is the default and the safer choice — it tests whether B is different from A in either direction. Only use one-tailed tests when you have a strong, pre-registered hypothesis that B can only be better, never worse.

Can I peek at results before the test is done?

No — that's the most common A/B testing mistake. Peeking and stopping early inflates false positive rates well above your stated 5%. Pre-register your sample size, run the full test, and only evaluate significance once you've hit it.

SpaceForms / Tools / Statistical Significance Calculator

Statistical Significance Calculator (Free, 2026)

Compare two groups — A/B test variants, survey segments, before/after campaigns — and get the p-value, z-score, lift % and a clear significance verdict instantly. Uses a two-proportion z-test with a pooled standard error, the textbook approach for comparing conversion rates.

Group A (control)

Conversions

Total visitors / sample

Rate: 5.00%

Group B (variant)

Conversions

Total visitors / sample

Rate: 6.46%

Confidence level

Significance at 95% confidence

Yes — significant

Variant B wins with +29.17% lift

p-value

0.0297

z-score

2.174

Lift

+29.17%

Test used: Two-proportion z-test with pooled standard error. z = (p_B − p_A) / √(p̂(1−p̂) × (1/n_A + 1/n_B)), where p̂ is the pooled conversion rate. Two-tailed p-value compared against α = 1 − confidence.

Reading the result

p-value < α: the observed difference is unlikely to be due to chance. Reject the null hypothesis ("the two groups are identical") and declare a winner.
p-value ≥ α: you don't have enough evidence to declare a winner — either the lift is real but the sample is too small, or there's no real lift. Either way, don't ship based on this data.
Lift %: relative change, (p_B − p_A) / p_A × 100. A 2% → 3% conversion rate is a +50% relative lift, even though absolute change is 1pp.
z-score: how many standard errors apart the two rates are. |z| above 1.96 ≈ 95% significance; above 2.576 ≈ 99% significance.

Minimum sample sizes for A/B testing

The required sample size per variant depends on baseline rate and the minimum detectable effect (MDE). Smaller lifts need much larger samples to detect with 95% confidence and 80% power.

Baseline rate	MDE +5%	MDE +10%	MDE +20%
2% (e.g., signup CTR)	~78,000	~19,500	~4,900
5%	~30,000	~7,500	~1,900
10%	~14,200	~3,500	~900
25% (e.g., survey CTR)	~4,700	~1,200	~300
50%	~1,600	~400	~100

Per variant, 95% confidence, 80% power, two-tailed. Use the sample size calculator for custom inputs.

When to use this calculator

The two-proportion z-test is the right tool whenever you're comparing two independent groups on a binary outcome — conversion vs no conversion, clicked vs didn't click, satisfied vs not. Common use cases:

• Email subject line A vs B (open rate)
• Landing page variants (signup rate)
• Survey invitation copy (response rate — Refiner's 2026 average is 27.5%, so test against that)
• Pricing page experiments (purchase rate)
• In-app prompt placement (engagement rate — 36.1% average for mobile in-app per Refiner)
• Before vs after a UI change (with caveats about confounding)
• Segment comparisons in survey data (e.g., "is mobile CSAT lower than desktop CSAT?")

When NOT to use this test

Small samples (n < 30 per group). The z-test assumes the normal approximation holds. For small samples use Fisher's exact test instead.
Continuous outcomes. If you're comparing mean revenue, mean CSAT score on a 1-5 scale, or mean time-on-page, use a t-test, not this calculator.
More than two groups. Running multiple pairwise z-tests inflates false positives. Use a chi-square test for the whole table, then post-hoc comparisons with Bonferroni correction.
Paired data. Same users measured twice (before/after) require a paired test (McNemar's for binary outcomes), not a two-sample z-test.
Sequential / peeked experiments. If you watched the test daily and decided to stop the moment p < 0.05, the stated significance is wrong. Use sequential testing methods (mSPRT, Bayesian) or commit to a fixed sample size up front.

Worked example

Scenario

You tested two survey invitation subject lines. Group A (control) got 120 responses from 2,400 sends. Group B (variant) got 155 responses from 2,400 sends. Is B's higher rate real or noise?

Calculation

• p_A = 120/2400 = 5.00%, p_B = 155/2400 = 6.46%
• Pooled p̂ = (120+155) / (2400+2400) = 5.73%
• SE = √(0.0573 × 0.9427 × (1/2400 + 1/2400)) = 0.00671
• z = (0.0646 − 0.0500) / 0.00671 = 2.18
• Two-tailed p-value ≈ 0.0293

Verdict

p = 0.0293 < 0.05, so the result is statistically significant at 95% confidence. Variant B wins with a +29.2% relative lift. Ship B for future survey sends — and if 29% lift on response rate seems too good to be true, run a confirmation test before rolling out fully.

Run your A/B tests with SpaceForms

Test invitation copy, question wording, scale formats — and compare segments directly in your survey results dashboard. Unlimited responses on the free tier means you can actually reach significance instead of running underpowered tests.

Build a free survey Calculate sample size

FAQs

What does the p-value actually mean?

The probability of observing a difference at least as extreme as yours, assuming the two groups are truly identical. A p-value of 0.03 means there's a 3% chance you'd see this big a difference purely by random sampling.

My result is barely significant — should I trust it?

Borderline results (p between 0.04 and 0.05) replicate poorly. If the decision matters, run a confirmation test. The closer p is to 0.05, the less robust the finding.

Why use a pooled standard error?

Under the null hypothesis (the two groups are identical), the best estimate of the shared rate comes from pooling both groups. Using a pooled SE produces a more accurate test than calculating SE separately for each group.

What if my sample is unbalanced?

The z-test handles unequal sample sizes fine — n_A and n_B don't need to match. But heavily unbalanced designs are less efficient: you get the most statistical power for a fixed total budget when n_A ≈ n_B.

Is this a frequentist or Bayesian test?

Frequentist. It returns a p-value, not a posterior probability. If you want "probability variant B is actually better," you need a Bayesian A/B testing tool — the math is different and the answers to different questions.