Free A/B Test Significance Calculator
Variant B is winning by 30 percent and you want to know whether that is real or whether you are looking at a coin landing heads a few times. Put the numbers in below and find out. Nothing you type ever leaves your browser.
Variant A
Variant B
How sure you want to be before you call it. 95 is the convention, and it is a convention rather than a law.
Is it real?
The p-value is the probability of seeing a gap at least this big if the two ads were actually identical. It is not the probability that B is better. Nearly every A/B calculator gets this backwards.
One thing that quietly ruins tests: checking every day and stopping the moment you see significance. Do that and you will find a winner in almost any test, including one where both ads are identical. Decide your sample size before you start, and look when you get there.
Free. No signup, no email. Nothing you type here leaves your browser.
Automated comment moderation for Facebook and Instagram
We monitor your Facebook and Instagram pages 24/7 and automatically hide bad comments and spam in your ads and organic posts.
7 days free. No credit card required. Setup in 2 minutes.
What this tool does
Two ads, one is ahead, and the only question that matters is whether it would still be ahead tomorrow. That is what significance testing answers, and it answers it in a narrower way than most people think.
Here is the thing almost every A/B calculator gets wrong, including expensive ones. They tell you 95 percent confidence means a 95 percent chance B is better. It does not. It means that if the two ads were identical, you would see a gap this big about 5 percent of the time anyway. That is a statement about the coin, not about which side wins, and the difference matters the moment you bet a budget on it.
This runs a proper two-tailed z-test. Two-tailed because the real question is whether B is different, not whether B is better: one-tailed tests reach significance sooner by quietly doubling your false positive rate, which is why the tools that want you to feel decisive use them.
It is free, there is no signup, and the whole thing runs in your browser.
How to run an ad test that tells you something
The calculator is the last five seconds of this. Everything that decides whether the answer is worth having happens before you open it.
- 1Change one thingNew image and new headline and new audience is not a test, it is a new ad. If it wins you have learned that something worked and you will never know what.
- 2Decide the sample size before you startThis is the step everyone skips and it is the one that makes the rest valid. Pick how many conversions you will wait for, write it down, and do not look before then.
- 3Let it run for whole weeksTuesday and Saturday are different businesses. A test that ran Monday to Thursday has measured your weekday audience and will mislead you about the weekend.
- 4Put the numbers in and read the verdict, not the liftA 40 percent lift on 30 conversions is noise. A 4 percent lift on 30,000 is money. The lift is the headline and the confidence is the story.
- 5Take not significant seriouslyIt means keep running, not call it a draw. Most ad tests that get declared a tie were simply stopped too early to see the difference.
What significance does not mean
This is where the money gets lost, and none of it is about the arithmetic.
- 1It is not the chance that B wins95 percent confidence means: if the ads were identical, a gap this big would turn up about 5 percent of the time by luck. It is a statement about how surprising your data is under a boring assumption, not a probability that you picked right.
- 2Not significant does not mean no differenceIt means you cannot tell yet. Absence of evidence is not evidence of absence, and the two get confused in almost every marketing meeting where a test gets shut down.
- 3Significant does not mean importantWith enough traffic, a 0.1 percent difference becomes statistically certain and remains commercially pointless. Statistics tells you the gap is real. It has no idea whether the gap is worth anything.
- 4Peeking manufactures winnersCheck daily and stop when you hit 95 percent and you will hit 95 percent eventually in almost any test, including one where the variants are literally the same ad. The threshold assumes you looked once.
What this test tells you, and what it does not
The arithmetic is standard and correct. What it means is narrower than people want it to be.
What this tells you
- Whether the gap is surprisingGiven the sample size, is this difference the kind of thing that happens by luck? That is a real, useful question and it is the one being answered.
- When you do not have enough dataBelow a handful of conversions it says so rather than printing a confident number that the underlying maths cannot support.
- That your numbers are impossibleMore conversions than visitors catches the most common paste error before it becomes a decision.
- A verdict that moves with your barThe same data is significant at 90 and not at 99. Seeing that flip is the fastest way to understand that the threshold is a convention, not a fact.
What it cannot tell you
- Whether B is actually betterThis is the big one. The p-value is about how likely your data is if the ads were identical. It is not the probability that B wins, and no amount of confidence turns it into that.
- Whether you peekedThe test assumes you decided the sample size in advance and looked once. If you checked every morning and stopped on a good one, this number is decoration.
- Whether the difference mattersSignificance is not size. A certain 0.2 percent lift is real and probably not worth the meeting.
- Anything about whyIt compares two counts. It does not know one ad had a scam comment under it for three days of the test, which is the sort of thing that decides ad tests and never appears in the data.
- Whether the test was fairDifferent budgets, different start dates, different audiences. Garbage in, statistically significant garbage out.
- NoveltyNew ads often do well because they are new. A winner at week one is sometimes just the thing your audience has not got bored of yet.
Testing ads vs Sweep Inbox
One tells you which ad won. The other is about a variable your test cannot see. Here is the split.
| This calculator | Sweep Inbox | |
|---|---|---|
| Price | Free, no signup | Paid, with a 7 day free trial |
| What it compares | Two sets of numbers you typed | Nothing. It changes what your buyers read |
| The uncontrolled variable | A scam comment under one variant skews the whole test, invisibly | Removed within seconds, so the test measures the ads |
| What it can act on | Nothing. It is arithmetic | Hides the comment before the impression is wasted |
| When it works | When you open it | Around the clock |
Questions people actually ask
- It means that if your two ads performed identically, a gap as big as the one you are looking at would be unusual. At 95 percent confidence, it would happen by luck about 5 percent of the time. That is all it means. It is not the probability that B is better, and that misreading is the single most common mistake in marketing statistics.
More free tools
Every tool here is free, runs in your browser, and never asks for an email.
Hidden Words Generator
Build a blocked words list for Instagram and Facebook in one click, then paste it into your Meta settings. No signup, no email.
Open toolComment Spam Checker
Paste the comments from under your posts and ads, and see which ones are spam and why. No signup, no email.
Open toolNegative Comment Reply Generator
Ready to post replies for complaints, scam accusations, trolls, and late deliveries. Pick the situation and the tone, then make it yours.
Open toolComment Moderation ROI Calculator
See what moderating your comments by hand costs every month, in hours and in money. Your numbers, our arithmetic, shown in full.
Open toolComment Section Health Score
Ten questions, one grade, and the specific fix for every weak spot in how you handle your comments. Most of the fixes are free.
Open toolProfanity Block List Generator
Build a swear word block list in English, French, Spanish, Arabic, or German, and choose how strict it is. Paste it straight into Meta.
Open toolEmoji Spam Blocked List
Emoji spam has no words, so word filters never see it. Meta's blocked list takes emojis too. Build one and paste it in.
Open toolArabic Hidden Words List
An Arabic blocked words list written as Arabic, not translated. Gulf, Egyptian, Levantine, and Maghrebi dialects included.
Open toolComment Policy Generator
The post you point at when someone asks why their comment was hidden. Pick your industry and strictness, then pin it.
Open toolFacebook Review Response Templates
Replies for angry one star reviews, fake ones, mixed, and glowing. Written for the next customer reading, not the reviewer.
Open toolModeration Cost Savings Calculator
Covering every hour is 4.2 full time people. See the gap between the coverage you promised and the team you have.
Open toolFacebook Ad Policy Checker
Paste your ad copy and see which phrases look risky, and which Meta policy each one belongs to. It cannot approve anything, and it says so.
Open toolAd Comment Cost Calculator
You paid for the impression and a scam reply was underneath it. See what that is worth, with the guesswork clearly labelled.
Open toolAd Character Limits Counter
Facebook and Instagram do not have the same limits, whatever the blogs say. Live counter with a truncation preview.
Open toolCPM, CPC and CPA Calculator
Fill in whatever you have from Ads Manager and it works out the rest. Anything it cannot compute says so instead of showing zero.
Open toolBreakeven ROAS Calculator
The 1 divided by margin shortcut is wrong and it flatters you. This one counts fees, shipping, and fulfilment.
Open toolAd Frequency Calculator
Frequency on its own tells you nothing. See how much of your audience you have actually burned through.
Open toolUTM Builder
Tagged campaign links with the Meta presets already right, plus the dynamic parameters almost nobody uses.
Open toolAd Budget Calculator
Work backwards from a revenue goal to the spend it needs, with the ROAS your plan quietly assumes shown next to it.
Open toolEngagement Bait Checker
Tag a friend, like if you agree, share to win. Meta demotes all of it. Paste your caption and get a rewrite.
Open toolInstagram Engagement Rate Calculator
The rate everyone quotes divides your likes by people who never saw the post. This gives you that one and the honest one.
Open toolInstagram Comment Picker
Pick a giveaway winner with a genuinely random draw. No post link, no upload, and deliberately no reroll.
Open toolBanned Hashtag Checker
The ones that catch you out are words like desk and adulting. Honest about the fact that no list is official or current.
Open toolInstagram Fonts Generator
Every style at once, ready to copy. Also the only one that tells you which styles your blind followers cannot read.
Open toolInstagram Character Counter
Captions can be 2,200 characters and get cut off at 125. See exactly which words disappear behind the more link.
Open toolFollower Growth Rate Calculator
8 percent in a week and 8 percent in a quarter are not the same. This normalises properly so you can compare them.
Open toolInstagram Reach Rate Calculator
A 500 percent reach rate can mean your own followers never saw the post. Split it and find out which it is.
Open toolFacebook Engagement Rate Calculator
Dividing by page likes is close to meaningless when barely any of them see your posts. Add reach and get the real number.
Open toolFacebook Comment Picker
Draw a giveaway winner from your Facebook comments. No post link, nothing uploaded, and deliberately no reroll.
Open toolFacebook Character Counter
63,206 characters is trivia. Your post gets cut at 125 on a phone, and you cannot see that from your desktop.
Open toolYour test has a variable you never controlled for.
Variant B lost, and nobody noticed the scam reply that sat under it for three days. Connect your pages and let Sweep Inbox make sure your test is measuring the ads.
7 days free. No credit card. Cancel anytime.
