Confidence Level is the percentage of time that a statistical outcome would be sufficient if numerous random samples are taken.
Confidence level carries a great importance in interpreting A/B testing results.
Let's give a confidence level example to clarify things for you:
>>> A/B testing with two email variations is conducted, and the following results appear:
Conversion= Number of Clickthroughs / Number of Emails Sent
A: 0.02
B: 0.025
Standard Error= Square root of <Conversion Rate*(1-Conversion Rate)/Sample Size>
A: 0.00198
B: 0.0020
Significance= <Conversion Rate(Variation B) – Conversion Rate(Variation A)>/Square root of <(Standard ErrorStanard Error)(Variation A) + (Standard ErrorStanard Error) (Variation B)>
Z-Score: 1.77
Interpretation:
The probability corresponding to Z-Score of 1.77 is 0.96. This means the test is 96% confident that conversion derived in Variation B is truly higher than conversion in Variation A.
Generally in the case of email A/B testing, a confidence level of 95% or above is recommended.
