A/B Test Optimization Issues

Author: D.Maydanyuk for the AI Rush 2017 conference.

The article below answers questions you should have found asking yourself once or often despite your proficiency level in AB testing. So, consider the expert’s answers and ask more here – we will happily try to answer.

What is A/B testing?

Basics of A/B test performance:

What can be tested?

Some issues of A/B testing

How to solve A/B issues?

• Do not fix the sample size before the test start.

  Tweet This

• Be guided by the “minimum detectable effect”.

  Tweet This

• Utilize the sequential statistical analysis.

  Tweet This

• Use Bayesian instead of the frequentist interpretation of variation probability to win.

  Tweet This

• Do not wait until the end of the test, but simultaneously maximize the win (the conversion number or profit).

  Tweet This

Minimum detectable effect

We should change the way we do our A/B test

A/B testing as a task of reinforcement learning

• Try to find a compromise between the uncertainty and the ability to apply the existing knowledge.
• Continue considering that the traffic is homogeneous.
• Use a “quick” strategy of sampling for I-iteration.