MVP application for launching and managing A/B tests

1. Summary

We developed a tool to keep track of A/B tests, which centralized the processes of conducting them. As a result, the company got a more transparent and convenient way of testing hypotheses. With its help, we conducted numerous experiments with statistically significant changes in metrics and spent less time preparing for them.

2. About the Project

At Ewa, processes for optimizing A/B testing were started. Together with the iOS Team Lead and analysts, we kicked off the development of our own tool under the working name A/B Toolkit for creating and keeping track of A/B tests. The development was led by our iOS Team Lead, and I took part in changing the testing processes and designed the MVP.

3. Research and Analysis

At the initial stage of the project, we carried out a detailed analysis of the current A/B testing process. During the analysis, we managed to find out the main issues the team was facing, including the inefficiency of some methods and limited resources. Thanks to a thorough research, we also found ways to develop and improve the process, which would allow us to streamline testing and improve its efficiency in the future.

3.1. Current process

The experiment process involves many participants. Due to communication difficulties, there are problems with determining who is in charge of launching the experiment, which leads to conflicts between managers and developers.

3.1.2 No change history

Links between process participants (managers, data analysts, developers, quality assurance specialists) do not allow for centralized tracking of changes. This leads to problems with restoring deleted tests and makes it harder to go over test activity for past periods.

3.1.3 High probability of human error

The diversity of participants and components responsible for testing significantly increases the likelihood of errors when entering data and setting configurations. Launching tests often requires manual changes to JSON code, which creates additional risks, as not all employees are well-versed in this language. Errors in the JSON structure can bring tests to a halt and kick users out of tests. As a result, such errors can set the A/B testing process back by 2-3 weeks.

4. Solution

One of the main solutions was the introduction of A/B Toolkit and Git for centralized test management and change history storage in one place. This allowed us to automate the publication of local and cloud tests, as well as notifications of changes through Slack, which significantly reduced the likelihood of human error. Moving from manual editing of JSON files to using UI and data validation before uploading also reduced the number of potential failures. Additionally, Android support was increased and a gradual phasing out of Firebase was started, which improved compatibility and reduced the risk of errors due to platform differences. These measures made the A/B testing process more centralized, automated, and resilient to errors, making the work easier for all project participants.

5. Design of A/B Toolkit

We decided to start by developing a native application for macOS for MVP, as this was the most economical option thanks to the availability of a sufficient number of iOS developers in the company. Unfortunately, A/B Toolkit could not be used by colleagues who did not have macOS or iOS devices, but the presence of the first version helped debug the product and collect valuable feedback for further improvement.

5.1. Design v1

After agreeing, we broke down the process of introducing new methods into several stages. At the first stage, we introduced A/B Toolkit — an application for macOS that centralized the management of local and cloud tests. It automatically uploaded local tests to a new release, and cloud tests — to EWA API. The application sends notifications of all changes to a Slack channel. We also introduced a Git repository for storing action history. Now all changes are made through a graphical interface, instead of manually editing JSON files.