A/B Test
Räftegård, Elin avatar
Written by Räftegård, Elin
Updated over a week ago

A/B Test

Test what content generates greater engagement among your Profiles. What color is better on the button? Where should I place the link? What image generates more clicks?

A/B testing compares different aspects of your Email on smaller groups. The winning version can then be sent to a wider audience. Set up A/B test after selecting Subscription to send to.

For Profile to be a part of an A/B test, the Profile must exist in the account before the Email is scheduled. Otherwise the Profile will not be included in the A/B test.


In this Article


How A/B testing works

You can do an A/B test on either Email design or Send time.

You prepare Email content versions in the Email editor. In the A/B test step, you set up the test: define the splits and add Email content versions.

The All Profiles in the sending will be randomly divided into groups, and receive the different content versions. When testing sender time, all Profiles receive the same content but at different times.

By studying the result of which group performed higher engagement (opens and clicks), you better understand your Profiles preferences.

Example:

Test which call to action (CTA) button color generates higher engagement: pink or cyan.

Note: If you are sending to less than 1000 Profiles, there is not enough data to actually conclude results with certainty. But of course, you could always split your Profiles into two groups to see how the sending results differ.


Prepare Email content versions

The different Emails versions for an A/B test is created in the same Email. The specific versions are then defined similarly to how you would segment an Email.

To create a version, you select Row(s) and connect them to a version.

Rows with no version connected to them, will be displayed in all Email versions.

Create a version

1. Create an Email in the Email editor, as you would for a regular sending. Include all content you are planning to send (for all versions).

2. Time to define versions - which Rows each version should include.
​Select what Rows belong to "version 1" by selecting Row in the structure panel to the left.

3. Open the A/B Test Versions option, in the Row settings on the right.

4. Define the selected Row as part of a version.

Add the version name (for example “version1”), and select it.

5. Define version 2 by selecting Row(s) you want to display in version 2.

In Row settings, create another version name (for example “version 2”) and select it.

6. Continue to connect a version to all Rows that belong to a certain version.

7. Use Preview to visualise what the different versions look like.

Rows with no version = displayed to all

Keep in mind that Rows that are not connected to a certain version, will be part of the general email and displayed to all recipients.

Settings

Adjust Email settings. If you have selected a bulk Email to use for the A/B test, these settings will no longer be editable once leaving the editor. Sender name, subject line and preheader can then be edited on the A/B test page.

8. When definition of versions is completed, go to the Next step.

9. Select Subscription and proceed to the A/B testing step.


Test Email design

Split type

Select Split type in the dropdown.

  • Split test without winner will send the different Splits to all groups of Profiles at the schedule time. You can study the results in the Email Reports to see what Split version performed better among your Profiles.

    This option is ideal when for test performed on a smaller amount of Profiles. To do a straightforward 50/50 test, for testing example what button color work better on your audience.

  • Split test with winner is best suited for a larger sending. This test type will initially send different Splits to a number of groups. After a certain amount of time (decided by you), the version performing the best will be sent out to the remaining group of Profiles. The winning criteria (for example opens or clicks) is also defined by you.

Number of splits

Select number of splits, number of groups you want in you test.

Click Calculate now to see how many Profiles match the setting.

Define splits

Use the percentage sliders to define sizes of the splits. To lock a value, click the padlock.

Set Email details for the different splits: Subject line, preheader and sender name.

Select Content version for each split.

In a A/B test without winner, click Next to proceed to scheduling sendout.

Schedule the sending of the A/B test. We recommend a minimum of 3 hours from the moment of the scheduling.

If performing a A/B test with winner, click the Define winner criteria button.

Clicking the Cancel split-sending button, will cancel the entire A/B test and the Email will be sent as a regular sending.

Define winning criteria

Select winner criteria in the dropdown.

Define when to consider it a tie. How small should the difference be in order to consider the versions to have performed equally?

Setting the majority condition to 5% would mean that its a tie if, the difference between the top two splits is less than 5% from each other.

If you prefer to use a simple majority select 0%.

Select version to send to the remaining Profiles, in case of a tie.

Decide number of hours to pass before sending out the winning version, after conditions are met.

To edit the split settings, click the Back to splits button. To proceed, click Next step.

Schedule send time.

On the Overview page, you can view details of the test, control send time, split sizes and preview the Emails.​

Click Activate to activate the A/B test.


Report

When performing an A/B test without winner, you study the results in the Email Report.

In the Report Overview for an A/B test with winner, the winning version is marked in the top graphs (based on the criteria you defined). This section presents the delivered and bounces Emails, in numbers and percentage.

In the Email breakdown you can study the recipients interaction with your Email: Open, clicks and unsubscribes. Unique opens and unique clicks counts the number of individual recipients interacting. A recipient opening the Email is registered as "Opens (unique) =1", and remains "1" regardless of how many the recipient opens the Email.

Click the information icon next to the title read explanations.

When performing an A/B test with winner, you find the winner details under the graphs.

image

Before the winner is picked, you can see the time scheduled for picking a winner.

Select Winner

With the option to manually select winner, you can chose what Split to send to the remaining Profiles. Click the Select winner button.

Select Split and send time in the drop-downs, for the sending a Split to the remaining Profiles.

Split Overview

Below follows a Split breakdown with details for the different Splits.

View Email

Under this tab you view the Email and toggle between desktop, mobile and text view.

Clicks tab

Under the Clicks tab you can see how many and which Profiles interacted with what links. Here you also find the link IDs.

Bounces tab

Under the Bounces tab you can study bounces. Click the View Profiles button to view what Profiles did not receive the Email and for what reason (bounce type).


Test Sender time

1. Select Split type: Sender time, in the dropdown.

2. Select number of groups you want to split the Profiles into.

3. Set the distribution of Profiles between the different groups with the sliders.

4. Schedule sending time for each group (expand with the green arrow on the right).

4. Expand the section with the arrow and schedule time for each Split 1.

5. For the remaining Split(s), schedule send time in relation to Split 1, number of minutes, hours, days or weeks after first sending.

6. Choose also whether to include new Profiles.

7. Click Next to proceed to the Details page, where you look over Subject line, Preheader and sender information.

Click Skip if you prefer to skip the splitting and proceed regularly with your sending.

8. On the Details page, enter Email details for split version 2.

9. Click Next.

10. Control the Split test set on the Overview page.

As this type of sending doesn't have a winning version, you have to study the results in the Email Report.


Clone or Cancel A/B test

To clone the A/B test, select the test on the Email start page and click Clone in the bottom bar. The whole A/B test activity is cloned. Adjust send time.

When cancelling an A/B test the sending will revert to a regular Email sending. Control the Email content as Rows with A/B test version added to them, will not be displayed to anyone.


Next step

Did this answer your question?