What is multivariate testing? When & why you should use it Marketing by GABRIEL SWAIN December 02, 2020 We recently published an article on A/B testing in marketing—also known as split testing—which allows you to experiment with two versions of the same marketing tactic. Now, we take a step further with multivariate testing (in the context of email campaigns). Multivariate testing allows you to experiment with granular variations of the same email and zero in on which variation is more effective. It’s best used to identify the right tactic for a smaller segment of your audience, not your entire contact database. The goal of multivariate testing is to improve audience engagement, conversion rates, and increase revenue generation through your digital marketing efforts. Let’s dig in, starting with the basics. What is multivariate testing? Multivariate email testing involves a complex and advanced approach to tactical email evaluation. You can test several combinations of the same email. With the ability to swap out email subject lines, email body copy, images, the sender “From” name or address, send time, and more, you have many options to play around with. This is where it diverges from a very similar approach, A/B testing. How is multivariate testing different from A/B testing? A/B testing experiments with two elements of an email. Multivariate testing experiments with up to four elements in an email, in multiple combinations, to determine the best performing combination. For example, Insightly’s multivariate testing feature—among its new marketing automation features—lets you compare four variables in up to eight versions of the same email. You can test Subject Line 1 with Email Body 2, Sender Name 3, and Send Time 4. That’s just one combination and you have eight to play with, providing deeper insight to drive future email campaign success. In some scenarios, A/B testing is all you need, while in others, multivariate testing is called for to generate deeper, actionable insight (more on that below). How do you run a multivariate test? Much in the same way A/B testing lets you select two options of the same email element; multivariate testing lets you choose multiple options. Ultimately, this is all done in your marketing automation solution and is quite easy to pull off. Set up is fairly straightforward. We’ll use Insightly Marketing to explain. When you select the multivariate testing option for an email campaign, system workflows guide you through the process of selecting which elements you want to test and let you select what each respective combination of elements looks like. Once this is set up, all combinations are sent out to a test group of your target audience. In the case of Insightly, the system automatically monitors the engagement levels of each email version sent to the test group. It then examines the resulting metrics, determines the “winning” combination, and routes that version to the rest of your target audience on the assumption that it will produce the best results. Because the process is automated, you simply select the combinations you want to test then sit back and wait for the results to come in. Your unified CRM with marketing automation does all the heavy lifting while you focus on strategy. What do you need to run your test? First and foremost, you need the right technology. If you use a unified CRM that includes both sales and marketing automation capabilities, all your data is stored under one roof and the process is exponentially easier. If not, you at least need a marketing automation solution that can run this kind of test. Without the right software, multivariate testing is virtually impossible. Additionally, you need to know what insight you aim to gain from your test and which problem or challenge you’re trying to solve. Below are some tips and questions to ask yourself to gain the insight necessary to run an effective test. Tips for effective multivariate testing Multivariate testing is used to solve a problem or address poor performance in a given area. When planning and setting up your test, it’s important to: Identify the problem you’re trying to solve or question you want to answer Define your hypothesis Decide which combinations you want to test Calculate the correct sample size to give you statistically accurate results Run a test on fake leads first to ensure every automated, triggered action is set up correctly Monitor the test as it progresses with real leads to ensure no unexpected hiccups occur Asking questions before you start A good way to get the most out of your multivariate email campaign testing is to ask yourself some preliminary questions. Below are a few good questions to ask yourself during the planning stage—these will help put you in mental testing mode. Does my audience prefer creative or short, direct subject lines? Does asking a question in the subject line result in more opens? Are recipients more likely to click a call-to-action button (or image) than hyperlinked text? Are they more likely to open an email from [Company Name] than an individual? What is the best time to send emails to my target audience to get higher open and click-through rates? Are concise bullet points more effective at driving click-throughs than text in paragraph format? I could go on and on, but you get the point. It’s important to approach multivariate testing with an inquisitive mind. Digital marketing must adapt to shifting customer expectations and testing new tactics is a great way to adapt. When should you use multivariate testing vs. A/B testing? A/B testing provides sweeping generalizations about the efficacy of one specific element of an email (or another marketing asset). This is high-level insight that tends to apply across the board and inform large shifts in your digital marketing strategy. On the other hand, multivariate testing provides more granular insight into a tactic. Use it when you need to optimize smaller elements with precision as they relate to a segment of your audience. In short, if you want insight into how effective a tactic is to your audience as a whole, A/B testing is the way to go. If you’re looking to discover which tactic best engages a specific, targeted segment of your audience and why, go with multivariate testing. Ready to run your own test? We strongly encourage you to do so. If you don’t constantly evolve your digital marketing practices and engage in experimental, disruptive marketing, you’re likely to be overtaken by the competition. Luckily, with marketing automation today, it’s easier than ever to engage in multivariate testing and A/B testing. After all, with this technology at your fingertips, you no longer need to rely on guesswork to find the most effective digital marketing tactics. It’s increasingly easy to use data instead of intuition in business decision making. Get the right tools, the right mindset, and see what your experimentation uncovers. Would you like to learn more about how multivariate testing could help you ramp up your digital marketing efforts? We’re happy to provide more details and answer your questions. Just request a free demo and we’ll provide the information you need. Request a demo Marketing Automation | Marketing KPIs | Measuring ROI GABRIEL SWAIN Gabriel is a content and digital marketing strategist, avid CRM blogger, and business growth consultant. Having spent years working with CRM providers, he has a deep understanding of users’ needs, challenges, and objectives. He uses that knowledge to provide growing businesses with educational content that allows them to maintain insight into emerging trends and retain a competitive edge.