A/B and Multivariate Testing

Explore A/B and multivariate testing in Insightly Marketing

Email marketing testing: Table stakes in growing your business

So, what exactly is A/B and multivariate testing? A/B and multivariate testing are used to validate how emails are performing; that is to say, which email template and design drives better user engagement. The goal is to consistently improve the user experience on an ongoing basis to drive incremental revenue. 

Why test? It’s table stakes for any company that wants to measure marketing performance, have a quantitative methodology for making creative decisions, consistently improve revenue, and beat their competition. Companies are expected to deliver great experiences to their customers and also provide creative updates to customers at a frequent clip. Customers can quickly tire of seeing the same content over and over again. Stale content can suppress performance. Ultimately, you want to help customers drive gains in conversion, and testing is the best way to do that in an ongoing, measurable way. 

A/B testing is available on all Insightly plans, and multivariate testing is available on Insightly’s Professional and Enterprise plans

What is A/B and multivariate testing?

A/B testing

It’s the least complex way to evaluate webpage or email design. A/B testing is a measurable way for customers to test key elements of the user experience and determine which experience drives a more effective response (usually measured as conversion) from the customer or prospect. 

It starts with a question or hypothesis to be tested: 

  • Will this button or link perform better at the middle of the email or the bottom?  
  • Will a customer see the button and click if the button is red or green?
  • Will a customer react positively if they read a list of features horizontally or vertically? 
  • Which video will a customer watch, and will it drive a response?
  • What performs better? Shorter copy, or longer? 

A/B testing is simple to control and manage since there are just two experiences that the team has to monitor. It’s also a good way to easily introduce testing to teams that might be skeptical about the value of testing, because you’ll get results faster due to the need for less traffic/smaller populations. 

Multivariate testing

Multivariate testing is a powerful way to help users hone in on the most effective use of creative elements to drive performance.  Not only do you get to determine winners and losers, but you also find out which components of the creative design are responsible for driving success and failure. For example, you might find out that header copy has absolutely no impact on performance; and that only button placement and body copy matter to response. 

Multivariate testing is similar to A/B testing, but it requires more traffic (higher populations) and provides the tester with more granular information about winners and losers. It allows the user to test more variables and it analyzes more information about which variable is driving better performance or conversion. Traffic is split between different creative options, and the multivariate test is conducted to measure the effectiveness of each design combination.  

Standard elements might be a subject line, header, body copy, footer, button, or form (Continue reading to find out what variables you can test in Insightly Marketing). The user sends discrete populations of users to every possible combination of these elements, and you’ll get an answer related to which elements are driving better responses. 

What variables can be tested with Insightly Marketing? 

  • Subject Lines: Different phrasing of subjects may get a higher open or click rate
  • From Names: Recipients might be more responsive to emails coming from a person’s name or from the name of your company or organization. The user can provide the From name and From email address to use for each combination.
  • Email Content: Users can create different versions of email content to see which version gets a better response. This variable can be used to test small content changes or completely different email templates.
  • Send Time: Users can use this variable to learn when recipients are most likely to open emails. Since this option tests specific days and times, users must send combinations to all recipients at once because the winning combination can’t be sent at a time that has already passed. The results of this test can be used to inform when to send or schedule future emails.

How to set up A/B and multivariate testing campaigns in Insightly

The user will choose the A/B Test Email option to send an email, then choose a single variable type— Subject line, From name, Content, or Send time—and create up to three different variations. No prospect receives more than one combination of the campaign. The version that a prospect receives is chosen at random and tracked solely to choose a winner, so users won’t be able to see which combination a subscriber received.

For multivariate tests – the user can choose up to three variables to test in these categories

  • Subject line
  • From name
  • Content
  • Send time

and create up to eight variations (Note: the user cannot choose all four variables because that would create more than 8 variations). Again, no prospect receives more than one combination of the email campaign. The specific combination variation that each prospect receives is chosen at random and tracked solely to choose a winner, so the user is not able to determine which combination each prospect receives.

How Insightly identifies the winning creative

Insightly will send the combinations to all email recipients at once if testing the Send time variable (but we recommend at least 5,000 recipients per combination because testing requires a valid statistically significant sample size). With other variables, Insightly will allow the customer to send test combinations to a percentage of email recipients and then send the winning combination to the remaining recipients.

Insightly will automatically choose the winning campaign to send to the remaining recipients after a set amount of time. The winner can be determined automatically by either the highest open or click rate.

Frequently Asked Questions

Why is A/B and Multivariate testing important for marketers?

The biggest challenge for customers is that they cannot grow incremental revenue without an ongoing testing strategy. Testing must be an integral part of any sophisticated marketing strategy, and customers need to be bought into creative/design and content/copy investment. Without testing, UX and creative decisions are made arbitrarily: assumptions, personal opinions and agendas, hierarchies, and political considerations are driving design. Customers need a data-driven approach to making improvements to their digital properties and the customer journey. They also need to understand what elements drive consumer behavior and, more specifically, conversions.

How do we build email tests?

A/B & Multivariate testing is located within the Emails object within Insightly Marketing. When you select ‘New Email,’ you will be presented with two new options in the drop-down for the type of testing you want. Here’s more information on how to get started.

Can I see more this process in action?

Yes, please review this video to learn more.