You are interviewing for a **Data Scientist** internship at Amazon. The interviewer asks you to walk through how you think about an NLP **classification** project — for example, classifying customer messages, search queries, or support tickets into categories — and then probes your understanding of model evaluation fundamentals. Work through the parts below. The goal is to demonstrate that you can explain core metrics clearly, reason about tradeoffs in terms of business cost, and connect evaluation choices to real product decisions. ### Constraints & Assumptions - The setting is a real-world classification system (binary or multi-class), not a clean academic benchmark. - Classes may be **imbalanced** — the category you care most about (e.g. a severe policy violation, fraud, an escalation) is often rare. - The model outputs **scores or probabilities**, and a decision threshold converts those into actions. - Predictions may feed an automated action (banning, blocking, routing) or a human-in-the-loop review queue. - Some answers (Part 1) must be understandable by a non-technical audience; others (Parts 2–5) expect precise definitions and formulas. ### Clarifying Questions to Ask - Is the task binary or multi-class, and how many categories are there? - How imbalanced are the classes, and which class matters most to the business? - What happens downstream with a prediction — an automated action, or routing to a human reviewer? - What are the relative costs of a false positive vs. a false negative in this product? - Is there a latency or cost budget that constrains the model choice? ### Part 1 — Explain a confusion matrix to high school students Explain what a confusion matrix is to a group of high school students. Avoid jargon; use a concrete, relatable example. ```hint Anchor on one example Pick a single binary yes/no scenario they already understand (spam-or-not, sick-or-healthy) and name the four cells in plain words before you ever say "true positive." ``` #### What This Part Should Cover - Genuinely accessible to non-experts: one concrete example, no formulas, plain-language naming of the four cells. - Conveys *why* the matrix matters — that it separates the *kinds* of mistakes, not just the count. - Lands the punchline that different mistakes carry different real-world costs. ### Part 2 — Define precision, recall, F1, and AUC Give the precise definition of **precision**, **recall**, **F1 score**, and **AUC**, including formulas where they exist. State in one sentence what each one tells you. ```hint Build on the four cells Every one of these is a ratio or summary of the TP/FP/FN/TN counts from Part 1 — except AUC, which is a threshold-free ranking measure. Think about which mistake each ratio puts in its denominator. ``` ```hint AUC interpretation ROC AUC has a clean one-sentence probabilistic reading that uses the words "randomly chosen positive" and "randomly chosen negative" — can you state it? Also think about when PR AUC is the better choice than ROC AUC. ``` #### What This Part Should Cover - Correct formulas for precision, recall, and F1, stated against the TP/FP/FN counts. - A one-sentence intuition for each metric (and which error sits in each denominator). - The probabilistic interpretation of ROC AUC and that it is threshold-free. - Awareness that PR AUC is the better summary under heavy class imbalance. ### Part 3 — Precision vs. recall: which to prioritize when Explain when you would prioritize **precision** over **recall**, and when you would prioritize **recall** over **precision**. Tie your reasoning to a decision criterion, not just examples. ```hint The deciding question Frame it as: which is more expensive here, a false positive or a false negative? Then connect that to where you set the decision **threshold** on the model's score — moving it trades one metric for the other. An $F_\beta$ score lets you bake the chosen tradeoff into a single number. ``` #### What This Part Should Cover - A single decision criterion (relative cost of FP vs. FN), not a list of memorized examples. - The threshold lever: raising/lowering it trades precision against recall. - Recognition that $F_\beta$ encodes a chosen asymmetry, and that 0.5 is not a sacred default. ### Part 4 — Evaluating a multi-class classification model How would you evaluate a multi-class classification model (one of $K$ categories)? Go beyond a single accuracy number. ```hint Don't trust one number A $K\times K$ confusion matrix plus **per-class** precision/recall/F1 reveals failures that overall accuracy hides — especially on rare classes. ``` ```hint Averaging matters Micro vs. macro vs. weighted averaging answer different questions. Decide which one surfaces rare-class performance, and consider calibration and subgroup slices too. ``` #### What This Part Should Cover - The $K\times K$ confusion matrix and per-class precision/recall/F1, not just aggregate accuracy. - A deliberate choice among micro / macro / weighted averaging, with a reason tied to rare classes. - Going further than point metrics: calibration when probabilities drive actions, and subgroup/slice evaluation. ### Part 5 — Cross-entropy loss What is **cross-entropy loss**, and why is it so commonly used for classification? Give the binary and multi-class forms. ```hint What it measures Think about what two things you are comparing when you write the formula — and consider how severely the loss behaves when the model is very confident but completely wrong. Why does that behavior, combined with differentiability, make it gradient-descent-friendly? ``` #### What This Part Should Cover - Correct binary and multi-class (softmax) forms of the loss. - An interpretation: it compares the predicted distribution to the true label distribution, equivalent to maximizing likelihood. - Why it is the default — sharp penalty for confident-but-wrong predictions plus clean differentiability/gradients. ### Part 6 — When human evaluation beats an automatic metric In what situations can **human evaluation** be better than using an automatic objective function or metric? Also note the costs and pitfalls of human evaluation. ```hint Where proxies break down Think about tasks where the automatic metric only measures *surface* similarity, not meaning (summarization, search relevance, generation). Then weigh human eval's own weaknesses (cost, rater inconsistency) and how you'd control for them. ``` #### What This Part Should Cover - Concrete tasks where the automatic metric is only a surface proxy for the quality that matters. - The failure modes of automatic metrics (Goodhart's law, noisy or outdated references, surface-only similarity). - Honest acknowledgment of human eval's own weaknesses and the controls that make a study rigorous (rubrics, multiple blind raters, inter-annotator agreement). ### Part 7 — Describing an NLP project you've done If asked to describe an NLP project you have done, what technical details and tradeoffs should you discuss? Lay out the structure of a strong answer. ```hint Walk the lifecycle Cover problem framing, data and labeling, baselines vs. complex models, the metric you chose *and why*, error analysis, and deployment/impact. Interviewers reward the candidate who justifies a *simpler* model and shows what they learned from mistakes. ``` #### What This Part Should Cover - A lifecycle structure: framing → data/labeling → baselines vs. complex models → metric choice → error analysis → deployment/impact. - Justification of each choice, including the maturity to defend a *simpler* model. - Genuine error analysis and an honest account of tradeoffs and what was learned from mistakes. ### What a Strong Answer Covers These dimensions span all parts — they are what the interviewer is listening for across the whole conversation, on top of the per-Part rubrics above. - **Audience calibration:** Part 1 is genuinely accessible to non-experts; Parts 2–5 are precise and formula-correct — the candidate adjusts register to the listener. - **Cost-based reasoning:** metric and threshold choices are justified by the cost of each error type, not by rote examples. - **Imbalance awareness:** recognizes that accuracy misleads on rare classes, and reaches for macro F1, PR AUC, or per-class recall throughout. - **Connection to decisions:** the best metric is the one that captures the product's cost of mistakes and feeds a real decision, not the most sophisticated one. ### Follow-up Questions - Your model has ROC AUC of 0.95 but business stakeholders complain it's "useless" on the rare class. What's likely going on, and what would you measure instead? - Predictions feed an automated action. The model's probabilities are used as confidence to decide whether to auto-action or send to human review. How do you check the probabilities are trustworthy? - Suppose your offline macro F1 improved after a model change but the online business metric got worse. How do you reconcile and diagnose that? - How would you design a human evaluation study so its results are reproducible and not just one annotator's opinion?