Tools list

en

A/B Test Significance Checker (Chi-Square)

Enter sample size (visits, clicks) and conversions (purchases, sign-ups) to check whether the difference is likely random. No advanced stats knowledge needed.
Use Simple for beginners and Detailed for power users.

How to use (Beginner)

Click to expand
1) Enter the sample size and conversion count for A and B.
2) If the result says “Significant”, the difference is unlikely to be random.
3) “Not significant” does not prove “no difference”; it may be too early to conclude.

Common notes


・Peeking and stopping at a convenient moment increases false positives (set a test period in advance if possible).
・Check other key metrics too (revenue, churn, CS, etc.).
This tool provides reference information for decision support.
Final decisions should consider assumptions, context, and risk.

1) Input

Enter sample size and conversions for A and B. Sample size should be unique users if possible.
Enter in this order (30 sec)
STEP1: Sample size (visits/clicks)
STEP2: Conversions (purchase/sign-up, etc.)
STEP3: Check A vs B and review results
Sample size is ideally unique users. Conversions are successful outcomes.
-
-
Yates correction (chi-square only)
“Significant” means the probability of this difference happening by chance is very low.
“Not significant” does not prove no difference; it may be too early to decide.

2) Results

You can start by checking the top verdict.
A
-
B
-
-
Detailed results (p-value, method, 2×2 table) -
Method used
-
Min expected count: -
Statistics: -
95% CI (B−A)
-
p-value (chi-square): -
p-value (z): -
p-value (Fisher): -
The p-value is the two-sided probability of observing a difference at least this large if the CVRs were equal.
Success (CV) Failure Total
A - - -
B - - -
Total - - -
💡 Was this tool helpful?
If someone you know is stuck, please share it with them.
日本語版 →
Copied