Deteriorating Test on Removing User Books

1. Summary

We removed the unused functionality of uploading user books without critical business impact and reduced the workload on the development team.

2. About the Task

In the «Library» section, besides the app’s books, users could upload their text files, which were converted into book format. After reviewing this functionality, I proposed testing how the removal of the file upload option would affect the metrics using a deteriorating test.

2.1. Problem

Supporting and debugging this functionality takes up significant development resources. After analyzing the usage of user books, we found that the conversion rate for uploads was 2.6% (2,000 users) for paying users and 4% (2,300 users) for non-paying users, out of those who visited the «Library» section. Moreover, only 12% (250 users) successfully uploaded a file at least once.

2.2. Hypothesis

If we remove the «My Books» button, as it does not affect subscription purchase/renewal and retention, and requires significant resources for support and debugging, the key metrics will not change, allowing for more efficient allocation of development resources.

To test this hypothesis, we launched an A/B test where:

1. Control Group (A) remained unchanged: the «My Books» button was retained.
2. Test Group (B): the «My Books» button was removed.

P.S.: The functionality itself was not removed in the test groups, only the entry point.

Target metrics:

1. count_days (average number of active days in the app)
2. Conversion to purchase (for new users)
3. count_days (average number of active days)
4. Additionally, we monitored complaints to Support.

3. Test Results

Based on the experiment results, we did not observe statistically significant changes in financial metrics or activity, nor differences in subscription cancellations. Complaints were minimal. As a result, we decided to remove the user books functionality.

Financial Metrics

No statistically significant changes in financial metrics

Activity Metrics

The average number of active days in the app did not change. Retention did not change significantly.

Book Metrics

The average number of started audiobooks dropped by −6%. However, this is likely a false positive, as three times more users are needed to accurately record a statistically significant change.

Complaints