Funnel experiments use a Beta model to evaluate the win probabilities and credible intervals for an experiment. Read the statistics primer for an introduction to Bayesian statistics if you haven't already. Our methodology uses a minimally informative prior of ALPHA_PRIOR = 1 and BETA_PRIOR = 1.
Win probabilities
The win probability tells you how likely it is that a given variant has the highest conversion rate compared to all other variants in the experiment. It helps you determine whether the experiment shows a statistically significant real effect vs. simply random chance.
Let's say you're testing a new signup flow and have these results:
- Control: 100 signups from 1000 visitors (10% conversion)
- Test: 150 signups from 1000 visitors (15% conversion)
To calculate the win probabilities for the experiment, our methodology:
Models each variant's conversion rate using a Beta distribution:
- Control: Beta(100 + ALPHA_PRIOR, 900 + BETA_PRIOR)
- Test: Beta(150 + ALPHA_PRIOR, 850 + BETA_PRIOR)
Takes 10,000 random samples from each distribution.
Checks which variant had the higher conversion rate for each sample.
Calculates the final win probabilities:
- Control wins in 40 out of 10,000 samples = 0.4% probability
- Test wins in 9,960 out of 10,000 samples = 99.6% probability
These results tell us we can be 99.6% confident that the test variant performs better than the control.
Credible intervals
A credible interval tells you the range where the true conversion rate lies with 95% probability. Unlike traditional confidence intervals, credible intervals give you a direct probability statement about the conversion rate.
For example, if you have these results:
- Control: 100 signups from 1000 visitors (10% conversion)
- Test: 150 signups from 1000 visitors (15% conversion)
To calculate the credible intervals for the experiment, our methodology will:
Create a Beta distribution for each variant:
- Control: Beta(100 + ALPHA_PRIOR, 900 + BETA_PRIOR)
- Test: Beta(150 + ALPHA_PRIOR, 850 + BETA_PRIOR)
Find the 2.5th and 97.5% percentiles of each distribution
- Control: [8.3%, 12%] = "You can be 95% confident the true conversion rate is between 8.3% and 12.0%"
- Test: [12.9%, 17.3%] = "You can be 95% confident the true conversion rate is between 12.9% and 17.3%"
Since these intervals don't overlap, you can be quite confident that the test variant performs better than the control. The intervals will become narrower as you collect more data, reflecting your increasing certainty about the true conversion rates.