A sample size calculator will allow you to calculate the sample size you need when you enter the following information:īaseline conversion rate (current conversion rate of your control-Version A)ĭesired statistical significance (in CRO and UX, the accepted standard is 95%) Statisticians use a complex formula to calculate statistical significance, but you don’t have to worry about any of that. The reason for this is simple-a major difference in performance is less likely to be caused by chance, whereas a small difference could easily be the result of randomness. When all other variables remain constant, a higher effect size produces a higher confidence level. The effect size is, once again, 20% (since Version B shows a 20% increase in conversion). The effect size is 20% (since Version B shows a 20% decrease in conversions).Įxample #2: Version A converts at 10% and Version B converts at 12%. These are the steps to calculate it:Ĭalculate the increase/decrease: new number-original numberĮxample #1: Version A converts at 10% and Version B converts at 8%. What does 'effect size' mean?Įffect size (also known as upflit in case of an increase or downlift in case of a decrease) is the percent increase or decrease in conversions between Version A and Version B of an A/B test. This is often necessary when website traffic is low and it takes longer to build up a large sample size. Statistical significance helps you determine the level of risk you’re willing to accept, and you can balance the desire for accuracy with the resources you have available.įor instance, higher statistical significance requires a larger sample size (all things being equal), so if you’re willing to accept a greater risk that your results were caused by random chance, you can get away with running tests with a smaller sample size. In much the same way, randomness in A/B testing can produce results that don’t reflect reality. Run any experiment enough times and unlikely events (statistical anomalies) are pretty much guaranteed. Those odds may seem low, but if 5,000 people read these words and try that experiment, there’s a high probability at least one of them will get 10 ‘heads’ in a row (the odds are 99.24%). If you flip a coin 10 times in a row, there’s a 1 in 1,024 chance it will land on heads every time-due to nothing more than random chance. Why is statistical significance important?
0 kommentar(er)
