Explain overfitting, imbalance, undersampling, and attention heads

Quick Overview

This interview question evaluates core ML concepts, assumptions, math intuition, training/evaluation trade-offs, and practical failure modes in a realistic interview setting. A strong answer for Explain overfitting, imbalance, undersampling, and attention heads states assumptions, handles edge cases, explains trade-offs, and shows how to validate the result clearly.

Explain overfitting, imbalance, undersampling, and attention heads

Company: TikTok

Role: Machine Learning Engineer

Category: Machine Learning

Difficulty: hard

Interview Round: Technical Screen

How do you diagnose and mitigate overfitting in machine learning models? Discuss techniques such as regularization (L1/L 2), data augmentation, early stopping, dropout, architecture changes, cross-validation, and appropriate validation strategies; explain trade-offs for each. How do you handle class imbalance? Compare undersampling, oversampling (including SMOTE/variants), class weighting, focal loss, and thresholding; specify when each is appropriate and how to evaluate with metrics like precision/recall, PR-AUC, ROC-AUC, and calibration. Detail common undersampling strategies—random undersampling, Tomek links, Edited Nearest Neighbors (ENN), NearMiss variants, and cluster-centroid methods—and discuss their impact on bias/variance and minority class recall. What is an attention head? Explain queries, keys, and values; how multi-head attention splits representation space; what multiple heads capture; and how head count affects capacity, compute, and inductive bias.

Quick Answer: This interview question evaluates core ML concepts, assumptions, math intuition, training/evaluation trade-offs, and practical failure modes in a realistic interview setting. A strong answer for Explain overfitting, imbalance, undersampling, and attention heads states assumptions, handles edge cases, explains trade-offs, and shows how to validate the result clearly.

|Home/Machine Learning/TikTok
TikTok logo
TikTok
Aug 8, 2025, 12:00 AM
hardMachine Learning EngineerTechnical ScreenMachine Learning
3
0

Explain overfitting, imbalance, undersampling, and attention heads

Context

You are designing and evaluating production machine learning models, with emphasis on classification, reliability, and efficient architectures. Answer the following multi-part question.

Part 1 — Diagnose and Mitigate Overfitting

Explain how to diagnose overfitting and discuss mitigation techniques, including:

  • Regularization: L1, L2 (weight decay)
  • Data augmentation (and label-preserving transformations)
  • Early stopping and learning-rate schedules
  • Dropout and related stochastic regularizers
  • Architecture changes (capacity, inductive bias, normalization)
  • Cross-validation
  • Proper validation strategies (avoiding leakage; stratified/group/time-based splits) For each technique, explain trade-offs.

Part 2 — Handle Class Imbalance

Discuss approaches to class imbalance:

  • Undersampling vs. oversampling (random; SMOTE and variants)
  • Class weighting, focal loss, and threshold selection
  • When each method is appropriate and how to evaluate with precision/recall, PR-AUC, ROC-AUC, and calibration

Detail common undersampling strategies and their impact on bias/variance and minority-class recall:

  • Random undersampling
  • Tomek links
  • Edited Nearest Neighbors (ENN)
  • NearMiss variants
  • Cluster-centroid methods

Part 3 — Attention Heads in Transformers

Define an attention head and explain:

  • Queries, keys, values
  • How multi-head attention splits representation space
  • What multiple heads can capture
  • How head count affects capacity, compute, and inductive bias

Constraints & Assumptions

  • Preserve the scope, facts, inputs, and requested outputs from the prompt above.
  • If the prompt leaves a detail unspecified, state a reasonable assumption before relying on it.
  • Keep the answer interview-ready: concise enough to present, but concrete enough to implement or evaluate.

Clarifying Questions to Ask Guidance

  • Clarify the task, data shape, labels, constraints, and evaluation metric.
  • State assumptions behind the math or modeling technique you choose.
  • Connect theory to practical training, debugging, and deployment implications.

What a Strong Answer Covers Guidance

  • Correct definitions and formulas where the prompt requires them.
  • A practical explanation of how the method behaves on real data.
  • Trade-offs, failure modes, diagnostics, and mitigation strategies.
  • Evaluation choices that match the product or modeling objective.

Follow-up Questions Guidance

  • How would noisy labels, class imbalance, or distribution shift affect the answer?
  • What would you monitor after deployment?
  • Which baseline would you compare against first?
Loading comments...