DataSentics

Factorial A/B testing

Introduction

A/B testing or measuring the impact of proposed options such as e-commerce websites is essential to increase conversion rates and other core business metrics. Our solution allows you to perform dozens of simultaneous A/B tests, get more interpretable results than just statistical significance sooner and more straightforward.

Business case

The complications are because of the restrictions of existing tests which process a limited portion of users and give slow results.

Our solution provides better insight into the test results, showing how features interact with each other. It is designed for e-commerce companies with a large customer base and a high volume of tests.

Solution

In addition to traditional A/B testing, faster and more interpretable results than just statistical significance can be achieved by implementing our solution.
To calculate such metrics as the probability that the change will have a positive effect or what part of the revenue is risked, we need to estimate the posterior distribution of the metrics, which are calculated by Generalized Linear Models (GLM).
And besides, thanks to implementing the PySpark, which does not provide standard coefficient errors, we can process gigabytes of data. Therefore, we use bootstrapping to estimate distributions.

Benfits

  • Faster results
  • An "unlimited" number of concurrent tests
  • Each test runs at 100% attendance