Model Debugging and Testing
In this lesson8 sections
Model Debugging and Testing
Debug an ML system by separating model-quality problems from problems in the data and serving path. This lesson follows a first model into online evaluation, then uses failure cases to decide whether to change features, collect data, or investigate a particular component.
The discussion covers two phases:
Building the first version of the model and the ML system.
Iterative improvements on top of the first version as well as debugging issues in large scale ML systems.
Building model v1
For the first version, establish a baseline and a way to evaluate it:
We begin by identifying a business problem in the first phase and mapping it to a machine learning problem.
We then go onto explore the training data and machine learning techniques that will work best on this problem.
Then we train the model given the available data and features, play around with hyper-parameters.
Once the model has been set up and we have early offline metrics like accuracy, precision/recall, AUC, etc., we continue to play around with the various features and training data strategies to improve our offline metrics.
If there is already a heuristics or rule-based system in place, our objective from the offline model would be to perform at least as good as the current system, e.g., for ads prediction problem, we would want our ML model AUC to be better than the current rule-based ads prediction based on only historical engagement rate.
It’s important to get version 1 launched to the real system quickly rather than spending too much time trying to optimize it. For example, if our AUC is 0.7 and it’s better than the current system with AUC 0.68, it’s generally a better idea to take model online and then continue to iterate to improve the quality. The reason is primarily that model improvement is an iterative process and we want validation from real traffic and data along with offline validation. We will look at various ideas that can help in that iterative development in the following sections.
Deploying and debugging v1 model
Some ML-based systems only operate in an offline setting, for example, detect objects from a large set of images. But, most systems have an online component as well, for example, building a search ranking ML model will have to run online to service incoming queries and ranking the documents that match the query.
The first online results may fall short of the offline results. Investigate how the live data and feature computation differ from the development setup before deciding what to change in the model.
Change in feature distribution
The change in the feature distribution of training and evaluation set can negatively affect the model performance. Let’s consider an example of an Entity linking system that is trained using a readily available Wikipedia dataset. As we start using the system for real traffic, the traffic that we are now getting for finding entities is a mix of Wikipedia articles as well as research papers. Given the model wasn’t trained on that data, its feature distribution would be a lot different than what it was trained on. Hence it is not performing as well on the research articles entity detection.
Another scenario could be a significant change in incoming traffic because of seasonality. Let’s consider an example of a search system trained using data for the last 2 weeks of December, i.e., mostly holiday traffic. If we deploy this system in January, the queries that it will see will be vastly different than what it was trained on and hence not performing as well as we observed in our offline validation.
Feature logging issues
When the model is trained offline, there is an assumption that features of the model generated offline would exactly be the same when the model is taken online. However, this might not be true as the way we generated features for our online system might not exactly be the same. It’s a common practice to append features offline to our training data for offline training and then add them later to the online model serving part. So, if the model doesn’t perform as well as we anticipated online, it would be good to see if feature generation logic is the same for offline training as well as online serving part of model evaluation.
Suppose an ad-click model uses historical advertiser impressions as a feature. Training computes that feature over the last 7 days, but serving computes it over the last 30 days. The feature has the same name while representing a different quantity. Compare the definitions and computed values on both paths so that this discrepancy is visible before attributing the online result to model quality.
Overfitting
Overfitting happens when a model learns the intrinsic details in the training data to the extent that it negatively impacts the performance of the model on new or unseen data.
Strong training and validation results followed by weaker live results can prompt an overfitting investigation. Also check the distribution and feature-computation problems described above; the live drop alone does not distinguish them. Keep a held-out test set outside hyperparameter tuning and use it for the final model-quality assessment.
Another important part is to have a comprehensive and large test set to cover all possible scenarios in a fairly similar distribution to how we anticipate them in live traffic. For example, consider an image object prediction system whose test set only has large-sized objects - (covering 50% pixels or more of the image) for 90% samples and small-sized objects for 10% samples (covering 10% pixels or less). If the live traffic has the opposite distribution of large and small size objects, then the model might not perform well on the live set.
Under-fitting
One indication from training the model could be that the model is unable to learn complex feature interactions especially if we are using a simplistic model. So, this might indicate to us that using slightly higher-order features, introduce more feature interactions, or use a more complex /expensive model such as a neural network.
Iterative model improvement
Use feedback from the first version to select the next experiment. Review failures for evidence of overfitting, underfitting, missing features, or gaps in the training data. The aim is to connect a proposed change to an observed problem and then measure whether that change helps.
Note: Debugging in current context doesn’t mean to optimize overall architecture and system design that we have deeply discussed in this course like how to set up a problem, setting up architecture etc. It comprises of general methods that we use to observe issues in current model to continue to optimize it.
Missing important feature
Digging deeper into failures examples can identify missing features that can help us perform better in failures cases, e.g., consider a scenario where a movie actually liked by the user was ranked very low by our recommendation system. On debugging, we figure out that the user has previously watched two movies by the same actor, so adding a feature on previous ratings by the user for this movie actor can help our model perform better in this case.
Insufficient training examples
We may also find that we are lacking training examples in cases where the model isn’t performing well. We will cater to all possible scenarios where the model is not performing well and update the training data accordingly. For example, for the image segmentation problem, the segmentation model is not able to segment the image when there are multiple traffic lights. One way to look at that point will be to count the number of such training examples that we have in our data set with multiple traffic lights.
So we will add more training examples of multiple traffic lights and enable the model to learn to segment multiple traffic lights better. The following illustration shows the above concept by taking an example of a test image from cityscapes dataset.
Debugging large scale systems
In the case of debugging large scale systems with multiple components(or models), we need to see which part of the overall system is not working correctly. It could be done for one failure example or over a set of examples to see where the opportunity lies to improve the metrics.
The following are a few key steps to think about iterative model improvement for large scale end to end ML systems:
Identify the component
First identify the component responsible for the failures under review. The search-ranking case study uses a layered approach, so candidate selection and ranking can be investigated separately.
In order to see the cause of failure, we will look at each layers’ performance to understand the opportunity to significantly improve the quality of our search system. Let’s assume that our search system has two key components 1) Document selection 2) Ranking of selected documents. Document selection focus is to ensure that all the top relevant documents get selected for the query while Ranker then ensures that our rank order is correct based on the relevance of the top 100 documents.
Inspect a few hundred failures and identify where the desired result was lost. In the hypothetical example, 80% of failures occur because candidate selection never returns the ideal document, so investigate that stage first. A later ranker cannot promote a document it never receives. If the desired documents are usually selected but ranked too low, focus the investigation on ranking instead.
Improve the quality of component
Some of the model improvement methods that we have discussed above like adding more training data, features, modeling approach in case of overfitting and underfitting will still be the same once we identify the component that needs work, e.g., if we identify that the candidate selection layer needs improvement in our search, we will try to see missing features, add more training data or play around with ML model parameters or try a new model.
