Explain ML and statistical modeling
Quick Overview
This question evaluates mastery of machine learning and statistical modeling concepts including class-imbalance strategies, loss function behavior and design (MAE, MSE, Huber, asymmetric losses, quantile regression), adversarial objectives and GAN stability, sequence-model trade-offs (RNNs vs Transformers), PCA and orthogonal regression, measurement error in linear models, spectral methods and sparse PCA, bias-variance trade-offs, MLE consistency, spiked covariance estimation, testing skill versus luck from pairwise outcomes, expectation inequalities, and residualization in two-stage regression. It is commonly asked to probe theoretical foundations and practical implications across the Machine Learning and Statistics domains—covering linear algebra, probability, optimization, and model evaluation—and gauges both conceptual understanding (statistical principles and asymptotics) and practical application (loss selection, algorithmic behavior, and stability).
Discuss the following machine learning and statistics topics:
- In a supervised learning problem with severe class imbalance, what techniques would you use at the data, loss, model, and evaluation levels?
- Compare MAE, MSE, and Huber loss. When is each preferable?
- How would you design a loss function that penalizes overestimation more than underestimation, or vice versa?
- What is quantile regression, how is its objective defined, and when is it preferable to mean regression?
- Explain the adversarial objective in GAN training and common stability issues.
- Compare RNNs and Transformers for sequence modeling.
- Show why one-dimensional orthogonal regression is closely related to PCA.
- In linear regression, if the observed feature is X_obs = X + U where U is independent measurement noise, what happens to the estimated coefficients?
- How does the power method recover the top eigenvector? Why is sparse PCA much harder than ordinary PCA?
- Explain the bias-variance trade-off and how it appears across model classes.
- Give examples where maximum likelihood estimation is not consistent.
- Suppose x_i is distributed as N(0, I + beta v v^T) with ||v|| = 1. How would you estimate v, and what is the leading-order dependence of the estimation error on sample size n, dimension d, and signal strength beta?
- Given only win/loss outcomes among n players, how would you test whether the game is mostly luck versus skill?
- Is it always true that min over y of E_X[f(X, y)] is less than or equal to E_{X,Y}[f(X, Y)]? What changes if X and Y are independent?
- If predictors are strongly correlated, how can residualization or innovations be used in a two-stage regression pipeline?