Build Classifier: Evaluate with AUROC for Imbalanced Data
Quick Overview
Evaluates classifier design for detecting dead links in imbalanced URL data with leakage-safe validation. Strong answers cover leakage-safe features, grouped validation, AUROC versus accuracy, PR metrics, thresholding, deployment, and monitoring.
Build Classifier: Evaluate with AUROC for Imbalanced Data
Company: Google
Role: Data Scientist
Category: Machine Learning
Difficulty: medium
Interview Round: Technical Screen
##### Scenario
Detecting dead links with a data set of 1,000 labeled URLs (good vs. bad).
##### Question
Walk through how you would build and evaluate a classifier. Which metric would you choose and why might AUROC be preferred over accuracy for imbalanced data?
##### Hints
Logistic regression plus class-imbalance-aware metrics.
Quick Answer: Evaluates classifier design for detecting dead links in imbalanced URL data with leakage-safe validation. Strong answers cover leakage-safe features, grouped validation, AUROC versus accuracy, PR metrics, thresholding, deployment, and monitoring.
Detecting Dead Links: Build and Evaluate a Classifier
You have a dataset of 1,000 URLs labeled as good, meaning alive, or bad, meaning dead. The classes are likely imbalanced, with far fewer dead links than good links.
Describe how you would build the classifier end to end, choose evaluation metrics for imbalanced data, and explain why AUROC may be preferred over accuracy.
Constraints & Assumptions
Treat this as binary classification with possible class imbalance.
Prevent leakage from features collected after the label time.
Consider correlation among URLs from the same domain.
Discuss deployment and monitoring, not only offline training.
Clarifying Questions to Ask Guidance
How were labels generated, and at what time?
What features are available before prediction time?
What is the cost of missing a dead link versus incorrectly flagging a live link?
Will the model run in batch or real time?
Part 1 - Build the Classifier
Describe data prep, features, model, validation, and deployment considerations.
What This Part Should Cover Guidance
Deduplicate and normalize URLs, inspect class balance, and split data without domain leakage.
Engineer features from URL structure, domain history, crawl metadata, anchor text, page metadata, and temporal signals if available.
Start with a logistic regression or tree-based baseline and compare models.
Address calibration, thresholding, retraining, and monitoring.
Part 2 - Metrics for Imbalanced Data
Which metrics would you choose and why?
What This Part Should Cover Guidance
Include precision, recall, F1, PR-AUC, AUROC, confusion matrix, calibration, and cost-based metrics.
Explain threshold-dependent and threshold-independent metrics.
Choose metrics based on business costs and class prevalence.
Report segment-level performance by domain type or URL category.
Part 3 - AUROC Versus Accuracy
Clarify why AUROC might be preferred over accuracy under imbalance.
What This Part Should Cover Guidance
Explain that accuracy can be high for a trivial majority-class classifier.
Define AUROC as ranking ability across thresholds.
Mention AUROC limitations under severe imbalance and why PR-AUC may be more informative for rare positives.
Connect metric choice to operating threshold.
Follow-up Questions Guidance
How would you choose the classification threshold?
What if URLs from the same domain appear in both train and test?
How would you monitor the model as websites change over time?