Online Experimentation

Use an online experiment to measure how a model change affects the product. Define the hypothesis and metrics, compare a control with a variation, interpret the statistical evidence, and check whether short-term gains persist.

Assume that the first version of an ML system, v0.1, is already deployed. Before changing it, decide what improvement would mean. An advertising platform might measure ad engagement and overall revenue; a search-ranking system might assess how well it orders results on the search engine result page (SERP). The experiment should test a proposed change against those goals.

Hypothesis and metrics intuition

At any point in time, the team can have multiple hypotheses that need to be validated via experimentation.

Imagine, for instance, that the team designing an ad prediction system wants to test the hypothesis that increase in the neural network model depth (increase in hidden layers) or width (increase in activation units) will increase latency and capacity but will still have an overall positive effect on user engagement and net ad revenue.

Similarly, a team working on designing a search engine wants to test the hypothesis that the pointwise algorithm instead of the pairwise algorithm would positively impact search relevance.

Test the proposed v0.2 in a controlled experiment before relying on the hypothesis. Comparing it with the existing system lets you measure how the change affects user behavior.

Running an online experiment

An A/B test compares two versions of the system at the same time. The existing version is the control; the changed version is the variation. In the ML examples here, the change may be the prediction model behind the page or app, rather than the visible interface.

We can formulate the following two hypotheses for the A/B test:

• Null hypothesis, H0: The change has no effect on the metric being tested.

• Alternative hypothesis, H1: The change has an effect, with the direction specified by the test design.

Rejecting H0 does not by itself justify launching the variation. Check the direction and size of the effect and the product’s other requirements.

Compare the chosen metric between the control and variation. Before running the experiment, use a power analysis test to plan the sample size needed to detect the effect you care about. The example below divides the experimental traffic equally between the two versions.

Measuring results

As visitors are served with either the control or variation/test version of the app, and their engagement with each experience is measured and analyzed through statistical analysis testing. Note that unless the tests are statistically significant, we cannot back up the claims of one version winning over another.

Computing statistical significance

P-value is used to help determine the statistical significance of the results. In interpreting the p-value of a significance test, a significance level (alpha) must be specified.

Note: The significance level is a boundary for specifying a statistically significant finding when interpreting the p-value. A commonly used value for the significance level is 5% written as 0.05.

The result of a significance test is claimed to be “statistically significant” if the p-value is less than the significance level.

• p <= alpha: reject H0 under the stated test procedure.

• p > alpha: fail to reject H0; this does not establish that the effect is zero.

A p-value is calculated under the null model and its assumptions. It measures how incompatible the observed result is with that model; it is not the probability that the result happened by chance or that H0 is true. Here, 0.05 is a 5% significance threshold, not a 95% significance level. Statistical significance also does not measure the size or practical value of an effect. Primary reference: ASA statement on p-values.

Measuring long term effects

In some cases, we need to be more confident about the result of an A/B experiment when it is overly optimistic.

Back Testing

Let’s assume that variation improved the overall system performance by 5% when the expected gain was 2%. In the case of the ads prediction system, we can say that the rate of user engagement with the ad increased by 5% in variation (system B). This surprising change puts forth a question. Is the result overly optimistic? To confirm the hypothesis and be more confident about the results, we can perform a backtest. Now we change criteria, system A is the previous system B, and vice versa.

In this reverse comparison, ask whether returning to the earlier system reverses the observed gain. The source example describes a 5% gain followed by a 5% loss. State the denominator when discussing those percentages: equal relative percentages in opposite directions do not exactly cancel, because they use different starting values. A reversal can support the interpretation of the experiment, but it does not guarantee that the original conclusion is correct.

Long-running A/B tests

A short experiment may not show effects that develop with continued use. Choose a longer observation period when the product question concerns long-term behavior, such as whether users keep returning.

For example, suppose that for the ad prediction system, the revenue went up by 5% when we started showing more ads to users but this had no effect on user retention. Will users start leaving the platform if we show them significantly more ads over a longer period of time? To answer this question, we might want to have a long-running A/B experiment to understand the impact.

The long-running experiment, which measures long-term behaviors, can also be done via a backtest. We can launch the experiment based on initial positive results while continuing to run a long-running backtest to measure any potential long term effects. If we can notice any significant negative behavior, we can revert the changes from the launched experiment.

---