Choose a North Star Metric for a Malicious-Webpage Classifier

# Choose a North Star Metric for a Malicious-Webpage Classifier A classifier flags whether a webpage a user is browsing is malicious. Propose a north star metric for the classifier and explain how you would measure it. Include the guardrails needed to prevent a higher metric from masking harm to users. Clarify the unit of evaluation and how reliable labels would be obtained. ### What a Strong Answer Covers - A metric tied to protection from malicious browsing rather than simply flag volume. - A precise denominator and a distinction between page-level quality and user-exposure impact. - False-positive and user-friction guardrails. - Label sampling that can estimate missed malicious pages outside the flagged set. ```hint Count opportunities for harm A rare malicious page visited often may matter differently from many pages that no one visits. ``` ### Follow-up Questions - Why can classifier accuracy improve while malicious exposure gets worse? - What additional evidence is needed to claim that warnings prevented harm?