How Would You Verify a Smiling-Face Detection Feature in Google Photos?

Imagine you are part of the Google Photos team, and your feature detects smiling faces in photos. How would you verify that the detection is correct? ```hint Define correct first Before choosing any test, decide what counts as a smile, who decides when people disagree, and whether you judge individual faces or whole photos. ``` ```hint Look beyond one number Think about where a single overall accuracy figure could look fine while the feature still fails for particular kinds of photos or people. ``` ### Clarifying Questions - What exactly does the feature output: a label per detected face, a label per photo, or a confidence score compared with a threshold? - How does the product use the result (search, suggestions, automatic edits), and which mistake is worse there: missing a smile or tagging a face that is not smiling? - Is there a face detector upstream of the smile classifier, and is verifying face detection in scope? - Are we verifying a new model before launch, a change to an existing one, or the feature as it runs in production? - What privacy constraints apply to looking at or labeling users' photos? ### What a Strong Answer Covers - A precise definition of correctness and a ground-truth labeling process that handles ambiguous faces. - Offline evaluation with metrics and an operating threshold tied to the product cost of each kind of error. - Coverage of hard cases and of population slices, with attention to fairness. - Software-level tests for the pipeline around the model, including regression checks. - Verification after launch: staged rollout, user signals and monitoring for drift. - Data handling that respects user privacy. ### Follow-up Questions - A new model version improves overall precision but gets worse on photos of young children. Do you ship it, and what would you need to know first? - How would you detect that the feature has degraded in production when you cannot look at users' photos? - In a group photo where some people smile and others do not, how should the feature behave, and how would you test that? - Labelers disagree on many borderline faces. How does that change your metrics and your launch criteria?