A/B Testing

Perform A/B tests on your audience to figure out which push and email configurations work better for you

A/B test helps you decide which push and email configurations work better for your audience. Create and send different versions of push or email messages to understand which one is more relevant and leads to more conversions. Specify different content, actions performed after the push is opened, segmentation or scheduling rules, and track the results!

Prerequisites

Before start, please make sure you've created Presets and Filters to use in your A/B Tests.

To start with A/B Testing, go to A/B Tests section of your Control Panel. Here, you'll see the list of already launched experiments.

Experiments list

To set up a new A/B Test, press + Create Experiment.

Create an experiment

Step 1. Enter the name of your experiment.

Type in the experiment name

Step 2. Select an app to carry out an experiment on.

Select an app

Step 3. Define the audience to which you'd like to send messages in the course of the experiment.

  • First, select a Filter to apply. The experiment will be held on only for those users who meet Filter's conditions.

  • Then, specify a segment size — the percentage of users meeting the Filter's conditions who will take part in your experiment.

To learn more about Filters and audience segmentation, please refer to the guide.

If you use a precompiled Filter, you’ll see the number of users complying with its conditions and the number of users in a segment specified.

Please be aware that a Filter saved in a preset does not apply to A/B Test segmentation. The experiments are only carried out on the audience you specify on step 3 of the experiment setting.

For example, if you set up the Name NOT SET Filter to be applied and the 50% segment size, half of the users with no value for the Name Tag will receive your A/B Testing messages, no matter what Filters are saved in the presets you send.

Define the experiment's audience

Step 4. Split the segment into groups. Each group will receive its own message.

Please note the A/B Tests are intended to compare the messages' content only. Scheduling and segmentation rules from the preset will not apply.

You can create up to 10 groups here.

Split the audience segment

Step 5. Specify content that will be sent to each of the groups.

  • First, select a content typePush preset or Email preset. The selected content type will be used for both groups.

  • To learn more about creating Presets, please refer to the guide.

Select a content type for both groups
  • Then, select a preset that will be sent to each of the groups.

The experiment will be held for all platforms enabled in all presets accumulatively. For example, if a preset for a Group A has the iOS platform enabled, and the preset for a Group B has the Android and Chrome platforms, the A/B experiment will include users with iOS, Android, or Chrome.

All that users will be split into groups according to the rate you set on step 4. In case a user's platform does not match the platform of preset specified for the user's testing group, that user will receive the preset with no platform-specific parameters.

Select a preset for each group

When everything's set up, press Send. The experiment will start immediately, that means all messages will be sent to all users taking part in the experiment with no delay.

Statistics

To explore the results of your A/B Test, go to A/B Tests section and press Statistics button next to the experiment.

Statistics for the experiment contains the summary of conditions applied and the number of users involved. For each of the groups, here you'll see the number of messages sent and opened, as well as the CTR. To see the detailed stats for each message, press the Statistics button placed in the corresponding table row.