A/B Test Optimization Issues

Author: D.Maydanyuk for the AI Rush 2017 conference.

The article below answers questions you should have found asking yourself once or often despite your proficiency level in AB testing. So, consider the expert’s answers and ask more here – we will happily try to answer.

What is A/B testing?


Basics of A/B test performance:


What can be tested?

Some issues of A/B testing

How to solve A/B issues?

Minimum detectable effect

We should change the way we do our A/B test


A/B testing as a task of reinforcement learning

  • Try to find a compromise between the uncertainty and the ability to apply the existing knowledge.
  • Continue considering that the traffic is homogeneous.
  • Use a “quick” strategy of sampling for I-iteration.