Deconstructing A/B Test

A practical guide to building custom A/B test frameworks without overhead

My boy and girl

Not a member? You can still check out this article through here.

When talking about A/B test, many organizations always think of the need for “certain” commercial tools, as if those tools are the only way to implement A/B test.

In fact, in the previous article, we talked about systems thinking, and the most important aspect of any system change is a shared vision, followed by a mental model, and the specific tools are at the surface level.

Therefore, the problem with the introduction of the A/B test is the mindset, not the tools.

This article aims to deconstruct the A/B test process and make the introduction of A/B test easier and more natural, without having to rely on a huge enterprise solution.

Breaking Down the A/B Test Workflow

An A/B test experiment consists of the following four core elements.

1. Setting up experiment conditions: Someone needs to design an experiment situation to verify the hypotheses, e.g. we want to test whether the new version of checkout process will increase the conversion rate.
2. Implementing experiment behaviors: The application implements different workflows based on group information.
3. Event logging: Events need to be sent out in different workflows and these events need to contain group information.
4. Post-analysis: Based on the event logs, the analysis is done to determine whether the initial hypothesis is correct or not, and whether the initial experiment conditions are appropriate or not.

Let’s describe an A/B test interaction with a complete time sequence.

1. Front-end receives group information from somewhere.
2. Front-end interacts with back-end with this group information.
3. Either front-end or back-end needs to do something based on the group information.
4. Throughout the process, the events with the group information are sent to the event collector based on the tracking points.
5. The event collector will store all…