A/B Test
Test what content generates greater engagement among your Profiles. What color is better on the button? Where should I place the link? What image generates more clicks?
A/B testing compares different aspects of your Email on smaller groups. The winning version can then be sent to a wider audience. Set up A/B test after selecting Subscription to send to.
For Profile to be a part of an A/B test, the Profile must exist in the account before the Email is scheduled. Otherwise the Profile will not be included in the A/B test.
In this Article
How A/B testing works
You can do an A/B test on either Email design or Send time.
Prepare the Email content versions in the Email editor. In the A/B test step, you set up the test: define the splits and add Email content versions.
All Profiles in the sending will be randomly divided into groups, and receive the different content versions.
In the Report, study the result of which group performed higher engagement (opens and clicks), to better understand your Profiles preferences.
Example:
Test which call to action (CTA) button color generates higher engagement: pink or cyan.
Note: If you are sending to less than 1000 Profiles, there is not enough data to actually conclude results with certainty. But of course, you could always split your Profiles into two groups to see how the sending results differ.
Prepare Email content versions
The different Emails versions for an A/B test is created in the same Email.
The specific versions are defined similarly to how you would segment an Email.
To create a version, you select Row(s) and connect them to a version.
Rows with no version connected to them, will be displayed in all Email versions.
Create a version
1. Create an Email in the Email editor, as you would for a regular sending. Include all content you are planning to send (for all versions).
Time to define versions - which Rows each version should include.
Let's start with selecting what Rows belong to "version 1".
2. Select Row in the structure panel to the left.
3. Open the A/B Test Versions option, in the Row settings on the right.
4. Define the selected Row as part of a version.
Type in the version name (for example “version1”), and select it.
Now, time for version 2.
5. Select the Row(s) you want to display in version 2.
In Row settings, in the A/B Testing versions drop-down, type in another version name
(for example “version 2”) and select it.
6. Make sure Rows belonging to version 1 are connected to version 1 in the Row setting. Do the same with version 2.
7. Use Preview to visualise what the different versions look like.
Note. Rows with no version = displayed to all
Keep in mind that Rows that are not connected to any version, will be part of the general email and displayed to all recipients.
Settings
Adjust Email settings. If you have selected a bulk Email to use for the A/B test, these settings will no longer be editable once leaving the editor. Sender name, subject line and preheader can then be edited on the A/B test page.
8. When definition of versions is completed, go to the Next step.
9. Select Subscription and proceed to the A/B testing step.
Test Email design
A/B test type
Select test type in the dropdown.
A/B test without winner will send the different Splits to all groups of Profiles at the scheduled time. You can study the results in the Email Reports to see what Split version performed better among your Profiles.
This option is ideal when for test performed on a smaller amount of Profiles. To do a straightforward 50/50 test, for testing example what button color work better on your audience.
A/B test sending time is one of the test options in A/B test without winner.
A/B test with winner is best suited for a larger sending. This test type will initially send different Splits to a number of groups. After a certain amount of time (decided by you), the version performing the best will be sent out to the remaining group of Profiles. The winning criteria (for example opens or clicks) is also defined by you.
Number of splits
Select number of splits, number of groups you want in you test.
Click Calculate now to see how many Profiles match the setting.
Define splits sizes
Use the percentage sliders to define sizes of the splits. To lock a value, click the padlock.
Set Email details for the different splits: Subject line, preheader and sender name.
Select Content version for each split.
In an A/B test without winner, click Next to proceed.
Set day and time for the sending.
If performing a A/B test with winner, click the Define winner criteria button.
Clicking the Cancel split-sending button, will cancel the entire A/B test and the Email will be sent as a regular sending.
Define winning criteria
Select winner criteria in the dropdown.
Define when to consider it a tie. How small should the difference be in order to consider the versions to have performed equally?
Setting the majority condition to 5% would mean that its a tie if, the difference between the top two splits is less than 5% from each other.
If you prefer to use a simple majority select 0%.
Select version to send to the remaining Profiles, in case of a tie.
Decide number of hours to pass before sending out the winning version, after conditions are met.
To edit the split settings, click the Back to splits button. To proceed, click Next step.
Schedule send time.
On the Overview page, you can view details of the test, control send time, split sizes and preview the Emails.
Click Activate to activate the A/B test.
Test Sending time
To perform an A/B test on sending time, you do not need to create different versions of your email content. It is better to focus your testing, so you know why one split generated higher engagement than the other.
1. Select Split type: A/B test without winner, in the dropdown.
2. Select number of groups you want to split the Profiles into.
3. Set the distribution of Profiles between the different groups with the sliders.
4. Tick the Send splits at different times checkbox.
5. Fill out details for the different Splits; Subject text, Preheader text and Sender name. toggle between Splits with the tabs on the top.
If you do not have different content versions, simply disregard the Content dropdown.
6. Schedule the sending time for all the Splits.
As you have scheduled your sendings, you should disregard the scheduling step.
7. Click Next to proceed to the Overview page.
8. Control the Split test set on the Overview page.
As this type of sending doesn't have a winning version, you have to study the results in the Email Report.
Duplicate or Cancel A/B test
To duplicate the A/B test, select the test on the Email start page and click Duplicate in the bottom bar. The whole A/B test activity is duplicated. Adjust send time.
When cancelling an A/B test the sending will revert to a regular Email sending. Control the Email content as Rows with A/B test version added to them, will not be displayed to anyone.
Report
When performing an A/B test without winner, you study the results in the Email Report.
In the Report Overview for an A/B test with winner, the winning version is marked in the top graphs (based on the criteria you defined). This section presents the delivered and bounces Emails, in numbers and percentage.
In the Email breakdown you can study the recipients interaction with your Email: Open, clicks and unsubscribes. Unique opens and unique clicks counts the number of individual recipients interacting.
A recipient opening the Email is registered as "Opens (unique) =1", and remains "1" regardless of how many the recipient opens the Email. Click the information icon next to the title read explanations.
When performing an A/B test with winner, you find the winner details under the graphs.
Before the winner is picked, you can see the time scheduled for picking a winner.
Select Winner
With the option to manually select winner, you can chose what Split to send to the remaining Profiles. Click the Select winner button.
Select Split and send time in the drop-downs, for the sending a Split to the remaining Profiles.
Split Overview
Below follows a Split breakdown with details for the different Splits.
View Email
Under this tab you view the Email and toggle between desktop, mobile and text view.
Clicks tab
Under the Clicks tab you can see how many and which Profiles interacted with what links. Here you also find the link IDs.
Bounces tab
Under the Bounces tab you can study bounces. Click the View Profiles button to view what Profiles did not receive the Email and for what reason (bounce type).
Next step