Metrics

Evaluate segmentation at both the pixel and class levels, then examine how it affects the larger driving pipeline. The 100-pixel example shows why high overall accuracy can hide complete failure on smaller classes.

Component level metric

The component of the self-driving car system under discussion here is the semantic segmentation of objects in the input image. In order to look for a suitable metric to measure the performance of an image segmenter, the first notion that comes to mind is the pixel-wise accuracy. Using this metric, you can simply compare the ground truth segmentation with the model’s predictive segmentation at a pixel level. However, this might not be the best idea, e.g., consider a scenario where the driving scene image has a major class imbalance, i.e., it mostly consists of sky and road.

Note: For this example, assume one-hundred pixels (ground truth) in the driving scene input image and the annotated distribution of these pixels by a human expert is as follows: sky=45, road=35, building=10, roadside=10.

Assume all 45 sky pixels and all 35 road pixels are correct, while the twenty building and roadside pixels are mislabeled as sky or road. The predictions contain fifty sky and fifty road pixels, but only eighty labels are correct. Pixel accuracy is therefore 80%, despite missing both smaller classes completely.