A/B Test Optimization Issues

Author: D.Maydanyuk for the AI Rush 2017 conference.

The article below answers questions you should have found asking yourself once or often despite your proficiency level in AB testing. So, consider the expert’s answers and ask more here – we will happily try to answer.

What is A/B testing?

Рисунок1

Basics of A/B test performance:

Рисунок6

What can be tested?

Some issues of A/B testing

How to solve A/B issues?

Minimum detectable effect

We should change the way we do our A/B test

Рисунок10

A/B testing as a task of reinforcement learning

  • Try to find a compromise between the uncertainty and the ability to apply the existing knowledge.
  • Continue considering that the traffic is homogeneous.
  • Use a “quick” strategy of sampling for I-iteration.