Machine Learning Fundamentals: Tree Models, Training, Evaluation, and Embeddings

# Machine Learning Fundamentals: Tree Models, Training, Evaluation, and Embeddings This is a concept-check round for an early-career ML engineer. The goal is not deep mathematical derivation but to test whether you can explain core machine learning ideas clearly and correctly, and reason about the trade-offs behind them. The interviewer will walk through several short topics: tree-based models, the training process, model evaluation, embeddings, and a few transformer basics. Treat each part as a 3-5 minute discussion where you explain the concept, why it works, and when you would or would not use it. ### Constraints & Assumptions - You are expected to communicate clearly to a technical interviewer, not to produce formal proofs. - Concrete examples and trade-offs matter more than memorized definitions. - Where a concept has well-known pitfalls (overfitting, leakage, metric misuse), you are expected to surface them unprompted. ### Clarifying Questions to Ask - Is the target audience a hands-on practitioner, or should I keep explanations at an intuitive level? - For evaluation, are we assuming a classification, regression, or ranking setting? The right metrics differ. - For the tree models, are we talking about a single decision tree, random forests, or gradient-boosted trees specifically? - Is there a particular domain (e.g., recommendations, tabular data, NLP) you want me to ground the examples in? ### Part 1 Explain how tree-based models work. Start with a single decision tree, then contrast bagging (random forests) with boosting (gradient-boosted trees). Why do ensembles outperform a single tree, and when would you prefer gradient boosting over a random forest? ```hint Where to start A single tree recursively splits the feature space to reduce an impurity measure (Gini / entropy for classification, variance / MSE for regression). Frame ensembles by what error they attack: bagging reduces **variance**, boosting reduces **bias**. ``` ```hint Bagging vs boosting Random forests train many de-correlated trees in parallel on bootstrap samples with feature subsampling, then average. Boosting trains trees **sequentially**, each fitting the residual error of the running ensemble. ``` #### What This Part Should Cover ```premium-lock What This Part Should Cover ``` ### Part 2 Walk through the model training process for a supervised model. Explain the role of the loss function, gradient descent, the train/validation/test split, regularization, and how you detect and prevent overfitting. ```hint Frame it as a loop Training = minimize a loss over parameters via (stochastic) gradient descent. The validation set is what tells you when to stop and how to tune hyperparameters; the test set is touched only once. ``` #### What This Part Should Cover ```premium-lock What This Part Should Cover ``` ### Part 3 How do you evaluate a model? Discuss metric selection, why accuracy can be misleading, and how class imbalance and threshold choice affect your conclusions. ```hint Pick the metric from the cost of errors Accuracy hides failure under imbalance (a 99%-negative dataset scores 99% by always predicting negative). Reach for precision/recall, F1, and threshold-independent ROC-AUC / PR-AUC, and tie the choice to the business cost of false positives vs false negatives. ``` #### What This Part Should Cover ```premium-lock What This Part Should Cover ``` ### Part 4 What is an embedding? Explain what an embedding represents, why we use them instead of raw IDs or one-hot vectors, how they are learned, and one place you have used or would use them. ```hint Anchor on the geometry An embedding maps a discrete entity (word, user, item) to a dense low-dimensional vector so that **similar entities land close together**, which one-hot vectors cannot express (every one-hot pair is equidistant). ``` #### What This Part Should Cover ```premium-lock What This Part Should Cover ``` ### Part 5 Cover a few transformer basics: what self-attention computes, why transformers replaced RNNs for sequence modeling, and the role of positional encoding. ```hint Self-attention in one line Each token builds a query, key, and value; attention weights come from query-key similarity (scaled dot-product, softmaxed), and the output is the weighted sum of values. This lets any token directly attend to any other, regardless of distance. ``` #### What This Part Should Cover ```premium-lock What This Part Should Cover ``` ### What a Strong Answer Covers ```premium-lock What a Strong Answer Covers ``` ### Follow-up Questions - For gradient-boosted trees (Part 1), what specifically does the learning rate control, and how does it interact with the number of trees? - In Part 2, you mentioned a train/validation/test split. How would you adapt this for time-series data where rows are not exchangeable? - For Part 3, when would you prefer PR-AUC over ROC-AUC, and why? - For the transformer in Part 5, what is the computational complexity of self-attention with respect to sequence length, and why is that a scaling concern?