Below you will find an approach on how to better attribute and measure the impact of an A/B Testing program. If you find yourself asking any of the following questions, throughout your A/B Testing journey then you have come to the right place.
A/B Testing Program Overview
Below you will find an approach on how to better attribute and measure the impact of an A/B Testing program. If you find yourself asking any of the following questions, throughout your A/B Testing journey then you have come to the right place.
Problem Statements
What is the dollar value of an A/B Testing program?
How long should we attribute incrementality for “go-forward” winning experience running at 100%?
How can I make sure I am running meaningful tests?
We are running all these tests but I am not seeing it hit my bottom line.
WHAT
This document is designed to show the total incremental revenue gained/lost for all tests run. This approach uses revenue per session as the primary metric to determine incremental revenue.
HOW
This is a winning A/B test that ran at 50/50 for 35 days.
STEP 1 - CALCULATE REVENUE PER SESSION
Test RPS = (Test Sales: $2,181,976 ÷ Sessions: 62,754) = $34.77
STEP 2 - CALCULATE AVG SALES LIFT
NOTE: It’s important that this metric is at statistical significance( >95% confidence).
STEP 3 - CALCULATE THE COST OF RUNNING TEST
Cost of running test = ( Control Sales: $2,081,976 * Avg Sales Lift: 3.54% ) = $73,783
STEP 4 - CALCULATE MULTIPLIER FOR TRAFFIC SPLIT
Total Sessions = 62,000 + 62,754 = 124,754
Control Traffic Distribution = 62,000 ÷ 124,754 = 49.7%
Test Traffic Distribution = 62,754 ÷ 124,754 = 50.3%
STEP 5 - CALCULATE VALUE OF THE CHANGE IF ROLLED OUT 100%
STEP 6 - CALCULATE VALUE GAINED FROM TESTING PERIOD
STEP 7 - FORECAST INCREMENTAL REVENUE FOR EXPERIENCE AT 100%
Avg Sales Per Day = ($148,463 ÷ 35) = $4,241
Projection = ($4,241 * 120 days) = $509,018
Explained: We recommend a 120 day forecast due to the always changing ecosystem. customer behavior, seasonality, promotions, site re-designs, development updates, and further A/B Testing make it difficult to expect consistent incremental lift for more than 120 days. You may also choose to run a winning test at 90/10 or 95/5 so you ensure that the experience is not negatively impacting KPI’s over time.
STEP 8 - CALCULATE OVERVIEW PROGRAM VALUE
Winning test: Steps 1-7
Losing tests: Steps 1-3 negative number.
Cost: We found that it costs ~$8-10K per test after meetings, creative design, test plans, analytics, development, and QA. For accuracy, we recommend that each program does their own cost calculations.
SUMMARY
As we start running more tests we will want to show that the program is profitable. Using this process we can keep track of our win/loss ratio, our incrementality, goals and more. This can validate that we are running impactful tests with meaningful hypotheses.
Not all tests will have RPS as a primary metric however we feel it should still be reported in the same fashion. This attribution model is not meant for all A/B practices however, all A/B practices should have some sort of program overview in place.
Useful excel formulas
- Statistical Significance
NORMSDIST(ABS( test avg sales – control avg sales)÷√ ( test std sales^2 ÷ test sessions + control std sales ^2 ÷ control sessions )
- Confidence Interval
CONFIDENCE.NORM (alpha 0.05, standard dev, size)
Interested in Learning More?
Get your free A/B testing and optimization assessment now!