Explain core ML concepts and design choices

Quick Overview

This question evaluates mastery of core machine learning fundamentals including probabilistic loss functions (cross-entropy and KL divergence), regularization and uncertainty techniques (dropout and MC dropout), normalization choices in large language models, optimizer behaviors (SGD, momentum, Adam), and policy optimization stability (PPO).

Explain core ML concepts and design choices

Company: Snapchat

Role: Machine Learning Engineer

Category: Machine Learning

Difficulty: hard

Interview Round: Onsite

Answer the following ML fundamentals: 1) Explain the relationship between cross-entropy and KL divergence, and derive when cross-entropy equals entropy plus KL. 2) Explain how dropout induces a training/testing distribution mismatch; describe practical fixes (e.g., inverted dropout scaling, Monte Carlo dropout) and when to prefer each. 3) Justify why large language models typically use LayerNorm instead of BatchNorm; discuss implications for sequence length, micro-batching, and stability. 4) Compare optimizers (SGD, SGD with momentum, Adam): update rules, convergence behavior, generalization trade-offs, and when you would choose each. 5) Explain why PPO introduces a KL-based constraint/penalty (or clipping) and how it stabilizes policy updates; discuss hyperparameter tuning and failure modes.

Quick Answer: This question evaluates mastery of core machine learning fundamentals including probabilistic loss functions (cross-entropy and KL divergence), regularization and uncertainty techniques (dropout and MC dropout), normalization choices in large language models, optimizer behaviors (SGD, momentum, Adam), and policy optimization stability (PPO).

|Home/Machine Learning/Snapchat
Snapchat logo
Snapchat
Sep 6, 2025, 12:00 AM
hardMachine Learning EngineerOnsiteMachine Learning
5
0

ML Fundamentals — Interview Questions

Instructions

Answer the following five ML fundamentals questions. Use precise definitions, equations, and concise justifications. If a derivation is requested, show the algebra clearly.

Questions

  1. Cross-entropy vs. KL divergence
  • Define cross-entropy H(p, q), entropy H(p), and KL divergence KL(p || q).
  • Derive the relationship H(p, q) = H(p) + KL(p || q) and explain its implications for classification loss.
  1. Dropout: training/testing distribution mismatch
  • Explain how dropout at training induces a distribution mismatch at test time if used naively.
  • Describe practical fixes, including inverted dropout scaling and Monte Carlo (MC) dropout.
  • State when you would prefer each approach in practice.
  1. Normalization in large language models (LLMs)
  • Justify why LLMs typically use LayerNorm instead of BatchNorm.
  • Discuss implications for sequence length, micro-batching/distributed training, and training stability (e.g., pre-LN vs. post-LN).
  1. Optimizers: SGD, SGD with momentum, Adam
  • Write the update rules for each (include bias-corrections for Adam).
  • Compare convergence behavior and generalization trade-offs.
  • Give concrete guidance on when to choose each.
  1. PPO (Proximal Policy Optimization)
  • Explain why PPO introduces a KL-based constraint/penalty or clipping and how this stabilizes policy updates.
  • Include the clipped surrogate objective and the KL-penalized objective.
  • Discuss key hyperparameters, tuning strategies (e.g., target KL, clip fraction), and common failure modes.
Loading comments...

Browse More Questions

More Machine Learning•More Snapchat•More Machine Learning Engineer•Snapchat Machine Learning Engineer•Snapchat Machine Learning•Machine Learning Engineer Machine Learning