Explain ML evaluation, sequence models, and optimizers

Scenario

An interviewer is deep-diving into an ML project you built (you can assume it is a supervised model unless specified otherwise). They want you to justify model choices, evaluation, and training decisions.

Part A — Evaluation design and metrics

1. Describe how you evaluate your model end-to-end (data split strategy, validation protocol, test usage).
2. Which metrics do you use and why (business/ML tradeoffs)?
3. Provide the mathematical definitions for the metrics you mention (e.g., accuracy, precision/recall, F1, ROC-AUC, PR-AUC, log loss, MSE/MAE, calibration metrics).
4. Propose a better evaluation workflow than a single holdout set (e.g., cross-validation, time-based split, stratification, repeated runs, confidence intervals).
5. If you can add human labels (or human evaluation), explain:
   • what you would label,
   • how you would ensure quality (guidelines, inter-annotator agreement),
   • how it improves the evaluation signal.
6. If you have no labels , what is the simplest way to estimate whether two model outputs/answers are similar ?

Part B — Transformers vs RNNs on long inputs

1. Compare Transformer and RNN/LSTM/GRU architectures.
2. For very long sequences , discuss the pros/cons of each (training stability, ability to capture long-range dependencies, compute/memory).
3. Explain why attention can capture long-range dependencies, and why vanilla RNNs often struggle.

Part C — Detecting distribution mismatch in images

You have two sets of images (Set A and Set B). How would you test whether they appear to come from the same underlying distribution?

Part D — Optimizers

Compare the practical differences and tradeoffs among SGD (with/without momentum), RMSProp, Adam, and AdamW. When would AdamW be preferable?