Free A/B test analysis tool

Statistical Significance Calculator

A statistical significance calculator tells you whether the difference between two conversion or survey response rates is likely real or just random noise. Enter the sample size and successes for each variant to get the lift, p-value, confidence interval, and a clear significant or not significant verdict.

After your test ends

Enter your results

What are you comparing?

Users or sessions assigned to each variant, and how many of them converted.

Variant A (control)
Variant B (treatment)
Confidence level

Two-sided test. Choose the level before you look at the results.

Two-proportion z-test

Your result

Statistically significant

Statistically significant at 95% confidence (p = 0.0118). New checkout's conversion rate of 11.60% is higher than Control's 10.00%.

Control (A)

10.00%

480 of 4,800 visitors

New checkout (B)

11.60%

551 of 4,750 visitors

Absolute change
+1.60 pp
Relative lift
+16.00%
p-value (two-sided)
0.0118
z-score
2.519
Confidence level
95% (alpha 0.05)
95% CI for the difference
+0.36 pp to +2.84 pp

Statistical significance is not practical significance. Check whether the low end of the interval is still a change worth shipping.

Privacy: every calculation runs in your browser. Nothing you enter is sent to a server or stored.

Planning a test or analyzing one?

Before the test: sample size

Use the A/B Test Sample Size Calculator to decide how many visitors each variant needs and how long to run. Fix that number up front.

After the test: significance

Once you reach the planned sample, use this calculator to check whether the observed difference is statistically significant and how large it plausibly is.

Worked example: a checkout test

The control checkout had 480 purchases from 4,800 visitors (10.00%). The new checkout had 551 from 4,750 (11.60%).

  • Absolute change: +1.60 pp, a relative lift of +16.00%.
  • Pooled rate 10.80%, z = 2.519, two-sided p = 0.0118.
  • At 95% confidence, p is below 0.05, so the lift is significant. The 95% interval runs from +0.36 pp to +2.84 pp.
  • At 99% confidence the same data is not significant, because p is above 0.01. The confidence level you pick in advance matters.

How the calculation works

Rates
pA = xA / nA and pB = xB / nB
Pooled standard error
p = (xA + xB) / (nA + nB), SE = √(p(1 - p)(1/nA + 1/nB))
z-score and p-value
z = (pB - pA) / SE. The two-sided p-value is the chance of a |z| at least this large if there is no real difference.
Confidence interval
(pB - pA) ± z* × √(pA(1 - pA)/nA + pB(1 - pB)/nB), with z* = 1.645, 1.960, or 2.576.

The test uses a pooled standard error and the interval uses an unpooled one, which is standard practice. Near the cutoff they can occasionally disagree.

Assumptions and limitations

  • Independent observations

    Each visitor or respondent is counted once and belongs to only one group. Repeat sessions from the same person break this.

  • Random assignment or representative samples

    Users must be randomly split between variants, or survey groups must be representative of the people you want to describe.

  • Adequate expected counts

    The normal approximation needs roughly 5 or more expected successes and non-successes per group. The calculator warns you when this fails.

  • One planned comparison

    Testing many variants, metrics, or segments, or checking results repeatedly and stopping early, inflates false positives. Adjust alpha or use a sequential method.

  • Significance is not importance

    With enough traffic, a tiny change can be significant. Look at the interval and ask whether the smallest plausible lift is worth shipping.

  • Binary outcomes only

    This compares proportions. For averages such as revenue per user, or for three or more groups, use a t-test or ANOVA.

Statistical significance questions

What does a p-value tell me?

The p-value is the probability of seeing a difference at least as large as yours if the two variants truly performed the same. A small p-value means your result would be unusual under no real difference. It is not the probability that Variant B is better, and it says nothing about how big or valuable the difference is.

Which confidence level should I use?

95% is the common default for product A/B tests. Use 90% for low-risk, easily reversed changes where speed matters, and 99% for costly or hard-to-undo decisions. Pick the level before you look at results, and use the same level you planned with in the sample size calculator.

How is this different from the A/B test sample size calculator?

The sample size calculator is for planning: it tells you how many visitors you need before a test starts. This statistical significance calculator is for analysis: it checks the data you already collected and tells you whether the observed difference is statistically significant.

Can I use this for survey results?

Yes. Switch to survey responses and enter how many people answered in each group and how many chose the answer you care about, such as Yes or Agree. The groups must be independent samples, such as two customer segments or two survey waves with different respondents.

Is my data uploaded anywhere?

No. Every calculation runs in your browser with plain JavaScript. Nothing you enter is sent to a server or saved, and Reset clears the form.

Related product tools

A/B Test Sample Size Calculator

Plan how many visitors each variant needs before you launch the test.

Open tool

ANOVA Calculator

Compare the means of three or more groups instead of two proportions.

Open tool

Feature Adoption Rate Calculator

Track how many users adopt the feature you just shipped.

Open tool

User Feedback Survey Builder

Design the survey whose answers you will compare here.

Open tool

Decide what to test next

FeatureVote gives you a public feedback board where customers submit and vote on ideas, so your next experiment starts from what users actually ask for.

Try FeatureVote free