Test Result Confidence Calculator
Assess the reliability of your A/B test results with confidence and risk metrics. Also known as the Statisical Significance Calculator
Go to Audience Size & Test Duration Calculator|
Total Users
Total Users
Total Users represent the number of people, trials, or observations in each variation of your test. Larger sample sizes generally lead to more accurate and reliable results. However, small numbers of users can still provide useful insights, but with a higher level of uncertainty.
|
Conversions
Conversions
Conversions refer to the number of successful outcomes, such as purchases, sign-ups, or any other desired action, within each variation of your test. This is known as 'Successes' in traditional terminology.
|
Conversion Rate
Conversion Rate
Conversion Rate is the percentage of users who completed the desired action (Conversions) out of the total number of users (Total Users) for each variation. It's a key metric to measure the effectiveness of your test.
|
Relative Change
Relative Change
Relative Change is the percentage increase or decrease in the conversion rate of a variation compared to the control group. This gives a sense of the magnitude of the effect of your changes. It's important to look at this in conjunction with the 'Probability of Variation > Control' and the 'Probability to be best' to understand both the size and the reliability of the effect.
|
Probability of
Variation > Control
Probability of Variation > Control
This represents the probability that a variation's conversion rate is superior to the control group's. If this probability exceeds your chosen Statistical Significance, it means that the variation is statistically significantly better than the control. It's a way of judging whether a variation is likely to provide a genuine improvement.
|
Probability to be best
Probability to be best
This tells you the likelihood that a particular variation will outperform all other variations and the control group. If this percentage exceeds your chosen Statistical Significance, it means the variation is statistically significantly the best. However, if no variation's 'Probability to be best' surpasses the Confidence Interval, no clear winner is declared, even though one or more variations may be significantly better than the control.
|
Expected loss
Expected loss
Expected loss estimates the potential downside if a specific variation is chosen over the others, including the control. It's a way of understanding the potential risk associated with that choice. A lower number here is preferable, implying less risk involved with that selection.
|
||
|---|---|---|---|---|---|---|---|---|
|
Control |
--- |
--- |
--- |
--- |
--- |
|||
|
Variation 1 |
--- |
--- |
--- |
--- |
--- |
|||
Desired Direction
Desired Direction
Choose the direction you expect your test variant might influence conversion rates. This choice affects the calculation of 'Probability of Variation > Control', 'Probability to be best', and 'Expected loss'. It's important to remember that actual results might differ from expectations, and all results need to be interpreted in light of your chosen Confidence Interval.
Confidence Interval
Confidence Interval
Confidence Interval reflects our certainty about these estimates. A higher percentage means we're more certain, but also gives a wider estimated range. This is also the threshold for declaring a variation as 'better than control' or 'the best' - if a variation's probabilities exceed this interval, they are considered statistically significant. Although 90% is the recommended standard, it can be adjusted based on your test's specific requirements.
About the Test Result Confidence Calculator
Our Test Result Confidence Calculator allows you to assess the reliability and potential risks associated with your A/B test results using a Bayesian method. This tool simplifies complex statistical concepts into understandable metrics, allowing you to grasp the significance of your results and the risk of false positives. You can make data-driven decisions with increased confidence, regardless of your level of statistical knowledge.
How to use the Calculator
- Enter the Total Users (or sample size) for both the Control and Variation group/s. This is the total number of people who have taken part in each variant of your test.
- Record the Conversions (or successful outcomes) for both the Control and Variation group/s. These are the successful actions like sign-ups or purchases made by users in each variant of your test.
- Choose the Desired Direction of your expected result. Are you expecting the conversion rates to increase or decrease with your test variant?
- Select your Confidence Interval. This tells us how confident we are about our result. A higher percentage means more confidence, but also a wider range where the true result might lie. It is also used to declare significant results and determine the winning variation.
- The calculator then provides the Conversion Rates, Relative Change, Probability of Variation being greater than the Control, Probability to be Best, and Expected Loss for both groups.
- You'll see the results explained based on the Desired Direction and Confidence Interval. The tool helps interpret your A/B testing results.
The Lumery PTY Ltd ©
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.