Metrics

Evaluate search quality using both user behavior and judged relevance. This lesson compares click-through rate and session success with offline NDCG, and explains why a click alone cannot tell you whether a search succeeded.

Choose a metric that reflects the behavior you want the search system to improve. A model comparison can look favorable on the chosen score while missing the searcher’s goal, so explain what each metric counts as success and what it misses.

There are two types of metrics to evaluate the success of a search query:

1. Online metrics

2. Offline metrics

We refer to metrics that are computed as part of user interaction in a live system as Online metrics. Meanwhile, offline metrics use offline data to measure the quality of your search engine and don’t rely on getting direct feedback from the users of the system.

Online metrics

In an online setting, you can base the success of a search session on user actions. On a per-query level, you can define success as the user action of clicking on a result.

A simple click-based metric is click-through rate.

Click-through rate

The click-through rate measures the ratio of clicks to impressions.

Note: Click through rate = $\frac{Numberofclicks}{Numberofimpressions}$