Metrics

Choose separate measures for user behavior and offline ranking quality. Work through precision and recall at a cutoff, average precision, and rating error, while distinguishing the source’s custom combined metric from standard F1.

In this lesson, you will look at different metrics that you can use to gauge the performance of the movie/show recommendation system.

Types of metrics

Like any other optimization problem, there are two types of metrics to measure the success of a movie/show recommendation system:

1. Online metrics:

   Online metrics are used to see the system’s performance through online evaluations on live data during an A/B test.

2. Offline metrics:

   Offline metrics are used in offline evaluations, which simulate the model’s performance in the production environment.

Train candidates and tune them with validation data from historical interactions. Reserve held-out test data for the chosen configuration’s final offline evaluation. A promising result can justify an online A/B test, where the model is evaluated against actual user outcomes.