Sarbacane Campaigns: how to perform an A/B test on an email campaign?

An A/B test allows for the evaluation of an email campaign's performance by sending two versions of an email to two representative samples of a recipient base. After comparing statistics, the more effective version of the two is sent to the rest of the contacts. The A/B test offers the possibility to compare different elements: the subject line, the sender field, or the entire message content.

How to create an A/B test campaign?

To create an A/B test email campaign, simply create an email campaign in Sarbacane Campaigns, then at the “Message” editing step, click on the “Split AB” section at the top right of the editor.

You will then be able to choose the elements you wish to test:

• The sender: allows you to compare two names and two sender addresses
• The subject: test two different subjects and find out which one is more convincing to your contact list
• The content: compare the performance of two completely different emails.

You have the possibility to test sender fields, subjects, and the entire content of your message simultaneously.

You can compare the two versions of each element of your email campaign via a tab system.

Setting up the A/B test

Once you have imported your recipient base, you can move on to setting up your A/B test.

Definition of Groups A and B

Here you can choose the proportion of your recipient base that will be used as a representative sample of your contacts, and thus the size of the two contact groups that will independently receive the two versions of your message. The recipients who will receive the tests will be chosen randomly.

For the sake of representativeness, it is advised to have at least 100 people per test group to avoid biasing the test.

To modify the proportion of contacts in the samples, move the slider to the left or right to increase or decrease this percentage.

Criteria for Evaluating the Winning Version

Check here the condition that will determine which version of your campaign will be defined as the winner at the end of the test:

Highest open rate: the message with the highest open rate will be considered the winner of the test.

Highest click-through rate: the message that has received the most clicks will be considered the winner.

Duration of the Evaluation Period

If you have chosen "Highest open rate" or "Highest click-through rate," you must choose the duration of the test using the selector. It is possible to set a number of hours or days after which the test will automatically be concluded and the statistics analyzed by Sarbacane to subsequently send the winning message to your entire base.

Mode of Sending the Winning Version

You can choose between two sending modes for your A/B test campaigns:

Automatically: Sarbacane Campaigns will take care of sending your two versions, analyzing the results, and sending the winning version to the rest of your list.

Manually: you choose the winner of the test yourself based on the behavioral statistics presented to you over time and decide when the A/B test should stop.

Test and Sending of an A/B Test

In an A/B Test, you get two different campaign summaries. You can switch between them by clicking on "Message A" and "Message B" to get details on either of these versions.

Similarly, you can also perform separate test sends to verify the correct layout of both of your messages.

By clicking on "Finalize campaign," you thus launch your A/B Test on the contact samples you have previously defined.

If you have chosen an automatic decision (highest open rate or highest click-through rate), you have nothing more to do, Sarbacane Campaigns will take care of sending your two versions, analyzing the results, and sending the final campaign.

If you have chosen manual triggering, you can check the performance of your two versions at any time from the Sarbacane Campaigns home screen or from the "Campaigns" menu by clicking on the statistics icon next to the name of your A/B test campaign. You can then trigger the sending manually whenever you see fit.