Data Confidence
What is Data Confidence?
Data Confidence in the Crobox app measures the trustworthiness of the results from our experiments. This is an important metric that determines the likelihood our experiments are accurate and not caused by random chance. To test the data confidence yourselves, try the A/B Test Calculator.
If you’re looking at CTR uplift, the Data Confidence indicates whether you can trust the insights based on the amount of data collected. The more trustworthy this data, the more certain you can be about reusing the data or making assumptions about your shoppers’ behavior.
Our data bars go from red (not enough data to be trustworthy) to yellow (slightly trustworthy) to green (trustworthy). The more data that is collected, the higher the Data Confidence. So, if your results are currently red, you should give the campaign more time to collect data or adjust the timeframe in which you are viewing the data.
You can hover over the bars to understand the exact observed confidence.
We measure Data Confidence based on statistical significance using a null hypothesis test. This tests the difference between the performance of the Crobox group vs. Control group. In the Crobox group, users are exposed to our messages. In the Control group, users are exposed to invisible messages so that we can test the difference between the two.
The null hypothesis test determines whether or not the use of Crobox’s Dynamic Messages has any impact on your KPIs.
There are two metrics of importance when determining statistical significance: P Value and Power. The P Value in your performance tracking is how we measure statistical significance. Power is the probability that a test of significance will pick up on an effect that is present.
A P Value of 0.05 (5%) means you can say with 95% confidence that there is a difference between the Crobox group vs. Control group. We always make sure our experiments have at least 95% statistical significance.
Last modified 3mo ago