Sample ratio mismatch checker
If a 50/50 test didn't split close to 50/50, something in assignment or tracking is broken. Check the split before you read the result.
Split check
Chi-square goodness of fitEnter visitor counts to check the split.
- p-value
- —
- Chi-square
- —
- Total visitors
- —
| Group | Observed | Expected | Share | Intended |
|---|
Common causes of a mismatch
- Redirect or slow-loading variants. Visitors who leave before the variant finishes loading never get counted, so the slower arm looks smaller.
- Tracking that fires in one arm only. A new template missing the analytics tag, or an event renamed in the variant.
- Bot and internal-traffic filtering that treats the arms differently, for example because the variant changes the URL or user agent pattern.
- Assignment at different points. Bucketing on page load in one arm and on click in another, or excluding visitors after they were assigned.
- Ramp changes mid-test. Changing the split from 10/90 to 50/50 partway through, then analyzing everything as one period.
- Caching or CDN rules that serve one arm's page to visitors assigned to the other.
Why a strict threshold
A healthy test is still never exactly 50/50, so the check asks whether the gap is bigger than chance allows. This page flags a mismatch at p < 0.001. On large tests even a small, real skew clears that bar, and a skew that size is enough to bias the lift.
When the check fails, don't adjust the numbers. Find the cause, fix it, and rerun the test. Results from a mismatched test can point the wrong way entirely.
Ready to read the outcome? Use the significance calculator, which runs this check automatically.
Catch broken tests before the meeting.
Loupe checks every result for sample ratio problems before it reaches your weekly roundtable, so nobody spends the meeting debating a broken test.
Seeing mismatches often?
That usually points to a setup problem across tests, not one bad test. An audit traces assignment and tracking end to end and fixes the root cause. From $2,000.