A/B Testing

This guide explains how to set up and manage A/B tests for your Product Finder, allowing you to compare different versions and measure the impact of changes.

Create a Testing Variant

Follow these steps to create a new test variant for your Product Finder.

1

Create a New Variant

Navigate to the Tests tab in your Product Finder.

Click Create Variant to start setting up your A/B test.

2

Name Your Variant

Enter a descriptive name for the new variant. This name will appear in your test reports. For example, if you're testing the impact of a new question about weather conditions, you might name the variant “Weather Question”.

3

Choose the Base Variant

Select which existing Product Finder variant you want to base your new test on.

4

Set the Testing Weight

Define the weights for each variant. The weight determines the allocation of sessions between variants, with higher weights receiving a larger share of traffic. For example, if one variant is weighted 3 and another is weighted 10, the first variant will receive 23% of the sessions and the second will receive 77%.

5

Save Your Test

After saving, you will see both variants along with the test allocation percentage of each.

Edit a Testing Variant

Once your test variant is created, you can edit it just like a regular Product Finder. Here's how:

1

Select the Variant to Edit

In the top right corner, you'll see a dropdown showing which variant you are currently editing (accessible in the setup, flow, translations and products tab).

2

Edit the Variant

Make changes to the setup page, flow or translations. These changes will only affect the selected variant.

If you want to apply changes that affect the entire Finder (e.g., global settings or design), you’ll need to update each variant individually.

Key Testing Insights

Use A/B Tests for Quick Results For faster insights, use a 50/50 weight split between variants. This will give each variant an equal share of sessions for quicker comparisons.

Multivariate Tests These tests involve multiple variants but take longer to reach significant results due to fewer sessions per variant.

Editing Variants Any edits you make will apply only to the selected variant. To apply changes that affect the entire Finder, you must update each variant individually and publish your changes as a whole in the versions tab.

FAQ

What is A/B testing in the context of my Product Finder?

A/B testing allows you to compare two or more versions of your Product Finder to evaluate the impact of specific changes. You can test different variants of your Finder to see which performs better with users.

What is the difference between A/B testing and multivariate testing?

A/B Testing — Involves comparing two versions of your Product Finder. It's the fastest way to understand the impact of a change.

Multivariate Testing — Involves testing multiple variants simultaneously. It takes longer to reach significant results because fewer sessions are allocated to each variant.

How do I choose the weight for each variant?

The weight determines how sessions are allocated between variants. A higher weight means that variant will receive more traffic. For instance, if one variant is weighted 3 and another is weighted 10, the first variant will receive 23% of the sessions, and the second will receive 77%. For a balanced test, use a 50/50 weight split between variants.

Can I make changes to my Product Finder while a test is running?

Yes, you can edit the flow, setup, or translations for each variant during the test. However, any changes you make will only apply to the selected variant. If you need to make a global change (e.g., affecting all variants), you’ll need to update each variant individually.

How do I apply global changes across variants?

If you need to make a global change (e.g., a design update) that should apply to all variants, you will need to manually update each variant and publish the changes in the Versions Tab.

Can I use A/B testing to test changes on just part of my Product Finder?

Yes, you can create a variant to test specific changes, such as a new question, different styling or a different flow. By using A/B testing, you can measure the impact of those changes while keeping the rest of the Finder the same.

How can I track the performance of my variants?

After saving your test, you will see both variants in the Tests Tab with their respective test allocations. To track performance over time, refer to your Analytics dashboard, where you can compare key metrics for each variant by targeting the Product Finder Variant within filtering.