A guide to A/B testing with Flows

Instead of basing decisions on gut feelings and instinct, A/B testing allows merchants to make data-driven decisions that can lead to the most successful outcomes—like higher revenue and average order value (AOV).  

With Flows, A/B testing can be done in three simple steps. Below, we’ll dive into how you can identify what type of tests to run and how to review the results. Plus, we’ll share some encouraging data from merchants already using Flows to A/B test.  

What is A/B testing?

A/B testing, also known as split testing, is a method of comparing two versions of a webpage or marketing campaign against each other to determine which one performs better. It involves showing two different versions—version A and version B—to similar audiences simultaneously, and then measuring which version yields better results based on a predefined metric.

For ecommerce merchants, there are some important things to know about A/B testing:

• You must have a control and variant group: the control group = existing version A. The variant group = modified version B.
• It’s crucial that subjects are randomly assigned to either the control or variant group to ensure unbiased results.
• A sufficient sample size (that should be statistically significant) is needed to draw reliable conclusions. This helps to determine if the observed differences are likely to be meaningful. 

Why is A/B testing important to business growth? 

A/B testing is pivotal in optimizing marketing efforts and driving business growth for a few reasons, including:

1. Data-driven decision making provides concrete, data-backed insights about what works and what doesn’t work. Businesses can move away from gut-feel decisions and base strategies on empirical evidence.

2. Testing different elements can refine and improve user experiences and ensure that they are more engaging and intuitive, and further drive LTV growth.

3. Marketers can use A/B testing to fine-tune strategies to maximize return on investment. Use A/B testing to identify high-performing campaigns and channels so that resources can be redirected where they have the most impact.

4. Test different messages to specific segments of the audience to be able to tailor and personalize messages to specific groups. This will increase relevance and resonance and drive higher conversion rates.

5. Markets and consumer behavior evolve. Test to adapt and stay competitive by continuously refining strategies based on real-time feedback. 

How to use Flows to conduct A/B testing

Recharge’s new tool, Flows, is designed specifically to help merchants enhance their customer journey. By giving you the ability to quickly test and iterate on strategies that will boost revenue and retention, Flows can help you increase customer lifetime value (LTV) for your online store. 

Flows is designed to simply and intuitively build comprehensive customer journeys. A/B testing is a core functionality of all Flows, including pre-built templates. It enables you to run A/B tests on different incentives and offers to determine which marketing efforts actually increase retention and AOV throughout the customer journey.

There are three steps to best design and structure your Flows, which we’ll detail below. 

1. Determine the goals of your test

The first step is to identify the goals of your test, which can be done using the following questions:

• What: What hypothesis would you like to test?
• When: What part of the customer journey would you like to test? When do you want the flow to trigger? 
• Who: Which customers do you want to target? Gain deeper insights by targeting users based on their characteristics, such as their recurring order number, AOV, LTV, location, or product usage.
• Time: How long do I want to run the experiment and how many customers do I want to participate?

2. Create an A/B test

The second step is to go to Flows and choose A/B test flow type.

 3. Review results

Finally, to determine the success of the incentive or offer, check out your branch results within the flow. Easily see how many customers entered the flow and how many accepted the offer and processed their next charge.

Merchants finding success with Flows

Let’s take a look at some examples of merchants who have utilized A/B testing within Flows, leading to increased orders, AOV, and revenue. Both merchants leveraged surprise and delight marketing by surprising customers with gifts, enhancing customer satisfaction and boosting key metrics. 

A/B testing a free gift incentive on the second order

For a period of a few weeks, a merchant ran an A/B test for all customers placing their second order. Upon order processing, there were two options:

• Path A: 50% of the customers were offered a free gift (valued at $15)
• Path B: 50% were not offered anything 

Of the orders that included the free gift, 100% of customers claimed the offer, leading to a 13.1% higher conversion rate than those customers who were not offered a gift. This generated 23.3% more revenue and 7.4% higher AOV. 

A/B testing a free gift incentive at the first order

Another merchant ran an A/B test for a couple weeks for all customers placing their first order. Upon processing the order, customers were offered:

• Path A: 50% of the customers were offered a free gift (valued at $33)
• Path B: 50% were not offered anything

In total, during that time period, 131 customers entered this test. Out of those offered the free gift, 100% claimed the offer, leading to a 4.5% order conversion rate and generating more revenue than those not offered anything. 

Utilizing A/B testing with Flows

Thanks to Flows, merchants have the opportunity to run A/B tests quickly and efficiently to determine which incentives and marketing efforts are working best. By running the right tests and analyzing results, merchants can work on increasing retention and AOV at various points throughout the customer journey.