Wondering how Motiva message testing works? Here's a peek under the hood.
Our customers sometimes ask why Motiva Message Testing takes a few days to find a winner compared to other A/B testing that seems to find a winner in 4-12 hours.
Statistical Confidence
Our goal is to determine a winner with a high degree of statistical confidence, so you know with almost absolute certainty that one version is better than the others.
Unfortunately, most A/B testing setups are designed to choose the version that gets the most engagement in a limited period of time, even though that version probably isn't the best.
To gather enough data to determine statistical significance, Motiva requires a minimum number of contacts and our model runs for several days so you don't get fooled by random spikes or dips in engagement. You'll see a blue notification bar appear at the bottom of the Message Test configuration window that alerts you to the contact requirements. See below.
To better understand why we have these requirements, it’s helpful to understand some of the science behind it.
There’s a type of modeling used in email testing known as a Multi-Armed bandit, named for the classic gambling slot machines which inspired the model's design. Here's how it works:
Picture yourself standing on the casino floor in front of multiple one-armed bandit slot machines. You’ve heard that one of them pays out at a higher rate than the other slot machines, but you aren’t sure which one.
You decide to experiment with three of the slot machines. They only cost a penny each, so you put 150 pennies in each machine and record your results. You notice that one pays out between 4 and 7% of the time, while the other pays out at a rate between 6 and 9%.
As you put more groups of 150 pennies in each slot machine, you narrow it down and confirm that one has an average of 8% overall, and the other is only 5%. So you focus the rest of your playing on the one that pays out at a higher rate.
The Multi-Armed model is essentially the same thing, only on a larger scale. Let’s say you’re experimenting with four slot machines. As you play more, you can see which slots are paying out at higher frequencies. So you focus more and more attention on the higher paying slots until you find the one or two with the highest payout percentages.
Now, imagine that instead of 150 pennies in each test round, you have 1,500. At ten times the number of pennies, you’ll get your answer that much more quickly and clearly.
In terms of multivariate email testing, it’s similar. Instead of slot machines, we're testing emails and measuring based on engagement rates instead of payouts.
Motiva Message Testing builds small test groups of contacts for each email version and sends out tests every send hour of the day. We then calculate the average engagement rate for each version across these many tests to determine the true performance with a high degree of statistical confidence.
Contact Requirements
The more contacts you have, the more data we can gather from each test. If you have only the minimum number of contacts, then each test is easily swayed by one or two additional click-throughs and it may take us longer to find a winner.
But if we have more contacts to work with, then each test group can be larger and one or two extra clicks won’t make much of an impact.
With larger numbers of contacts, it’s much easier to recognize patterns, which allows us to deliver your most potent content to your audience.
Want to dig deeper? Click here to read more about how to interpret the results of your Message Testing step.
Looking for a hands-on demonstration of what Motiva AI can do for you? Schedule a Demo today!