Online Experimentation

Move from offline model selection to a controlled feed-ranking experiment. The example keeps the control and treatment populations equal, calculates relative engagement gain, and weighs the result against uncertainty and operating cost.

Let’s look at the steps from training the model to deploying it.

Step 1: Training different models

Earlier, in the training data generation lesson, we discussed a method of splitting the training data for training and validation purposes. After the split, the training data is utilized to train, say, fifteen different models, each with a different combination of hyperparameters, features, and machine learning algorithms.