Walmart Labs Data Scientist Interview Prep Guide

Your main focus is refreshing Statistics & Math and Machine Learning: you rated both 3/5, have no solved-question signal yet, and explicitly said you are rusty with Stats and ML concepts. Nothing is clearly a strong area from the signals, so SQL/Python, coding, analytics, and behavioral stay at normal review rather than being over-expanded. For Walmart Labs, the plan highlights retail demand, recommendations, pricing and promotions, inventory stockouts, fraud/anomaly detection, fulfillment ETA, and retail-scale model monitoring. With 1–2 weeks left, this is an emphasis-heavy plan: spend most practice time on explaining assumptions, metrics, validation, and business trade-offs clearly.

Onsite — 60 min

Machine Learning

Focus area — You said ML concepts feel rusty; validation leakage, bias-variance, and model selection are common onsite probes.

What's being tested

Candidates must demonstrate practical mastery of K-Fold Cross-Validation as a tool for model selection and honest performance estimation: when to use it, how to configure it (k, stratification, grouping), and how to avoid common statistical and data-leakage pitfalls. Interviewers probe your ability to trade off computational cost vs. estimator variance, to decide when nested procedures are required for hyperparameter tuning, and to communicate uncertainty in the chosen model’s expected production performance. Walmart Labs cares because reproducible, unbiased model selection prevents bad business decisions and expensive rollbacks.

Core knowledge

• K-Fold Cross-Validation: partition data into k roughly equal folds, train on k−1, evaluate on the holdout, repeat k times, aggregate metric (typically mean ± std). Common defaults: k=5 or 10.

• Bias–variance of CV estimate: smaller k (e.g., 5) gives higher bias but lower variance; larger k (e.g., LOOCV) lowers bias but raises variance and cost; 5–10 is a pragmatic tradeoff for N up to ~1e6 rows.

• Nested cross-validation: use an inner CV loop for hyperparameter tuning and an outer loop for unbiased model assessment; needed when hyperparameters are chosen based on CV performance to avoid optimistic selection bias.

• Data leakage: any preprocessing (feature scaling, imputation, feature selection, target encoding) must be computed inside each training fold and applied to validation fold only; otherwise the CV estimate is optimistically biased.

• Grouped / clustered CV: when observations share a key (user_id, session_id, product_id), use grouped CV (e.g., GroupKFold) so groups do not span train/validation, preventing leakage and inflated metrics.

• Time-series CV: for temporal data, use forward-chaining (rolling/horizon) CV that respects chronology; random k-folds break temporal dependency and produce misleading performance.

• Stratification: for imbalanced classification, use stratified K-fold to preserve label proportions across folds; when using both grouping and imbalance, prefer group-aware stratification or repeated sampling.

• Out-of-fold predictions: collect OOF predictions during CV to compute blended ensembles, stacking features, or calibration without leaking test data; OOF is key to honest meta-model training.

• Metric aggregation & dispersion: report mean CV score and a dispersion measure (std, 95% CI via t-distribution or bootstrap). For skewed metrics (e.g., precision@k) report median and IQR or bootstrap CIs.

• Multiple model comparisons: when comparing many models/hyperparameter sets, adjust for multiple comparisons (e.g., control false discovery or report effect sizes); repeated CV or nested CV reduces selection bias.

• Compute considerations: large datasets (N >> 1e6) may require subsampling, incremental/online validation, or reduced k; parallelize folds with careful resource controls (joblib, cluster jobs).

• When holdout test needed: always keep a final untouched test set for the last-stage evaluation if models will be released to production; nested CV can substitute when test set is not available, but external holdout remains best practice.

Worked example — Evaluate K-Fold Cross-Validation for Model Selection

First 30 seconds: clarify the problem scope — is the goal model selection (choose among families) or hyperparameter tuning within one family; ask about data shape (time-series, grouped, class balance), metric (AUC, RMSE, precision@k), and compute constraints. Skeleton answer pillars: (1) choose fold strategy (stratified/grouped/temporal), (2) decide nested vs. single CV for hyperparameter search, (3) choose k and aggregation/uncertainty reporting, (4) practical engineering (preprocessing-in-fold, parallelization, OOF predictions). One explicit tradeoff: using nested CV gives unbiased generalization estimates but multiplies compute cost by inner-loop search complexity — if compute is constrained, use a separate holdout test and single CV for tuning. Close by stating further work: "if I had more time, I’d run repeated nested CV to measure selection stability, examine OOF residuals by cohort, and validate performance on a temporally separated production-like test set."

A second angle — model selection for temporal / productionized systems

When the same concept is applied to time-series forecasting or models used in production with temporal drift, random k-folds fail. You’d frame the solution around rolling-origin CV (expanding/rolling windows), choose evaluation horizons matching business KPIs, and ensure hyperparameter tuning happens within the temporal fold to avoid peeking into the future. Additionally, report performance decay by training-window size and validate model retraining cadence; emphasize backtesting on chronologically later holdouts and stress-test on regime shifts to estimate maintenance cost.

Common pitfalls

Pitfall: doing preprocessing globally before CV.

Computing scalers, PCA, or target encodings on the full dataset leaks information from validation folds into training, inflating CV scores and producing models that underperform in production.

Pitfall: using standard k-fold on grouped or time-dependent data.

If records from the same user or later timestamps appear in both train and validation then CV optimism can be large; grouped CV or time-aware CV is required.

Pitfall: trusting a single CV mean without uncertainty or stability checks.

A small mean improvement (e.g., +0.5% AUC) across one CV run may be noise; report variance, run repeated CV, or test on an external holdout before claiming superiority.

Connections

This topic commonly leads to adjacent concepts interviewers may pivot to: experiment design / A/B testing (designing offline metrics that align with online metrics), feature engineering pipelines (how to version and compute features safely inside folds), and model monitoring / drift detection (post-deployment validation against CV expectations).

Further reading

• Practical Guide to Cross-Validation (Scikit-learn docs) — concise reference for GroupKFold, TimeSeriesSplit, and cross_val_score.

Practice questions

Focus area — This combines ML forecasting with business metrics, matching both your ML rustiness and Walmart campaign analytics.

What's being tested

Candidates must show they can turn historical conversion-rate signals into robust, business-actionable forecasts: define the metric, choose an appropriate statistical or ML model given exposure and sparsity, evaluate time-aware performance, and communicate uncertainty and tradeoffs. Interviewers probe model assumptions, handling of heteroskedasticity from varying impression counts, ways to pool information across campaigns, and how forecasts would be used to make decisions (budgeting, creative cadence, capacity).

Core knowledge

• Conversion rate formulation: conversions C over exposures/impressions N, so $p = C / N$. Treat raw counts as Binomial: $C \sim \text{Binomial}(N, p)$ whenever N is known and trials are independent.

• Rate modeling techniques: use a Binomial/Logistic GLM ($\text{logit}(p) = X\beta$) for direct rate modeling, or model counts with a Poisson/Negative Binomial with log(exposure) as an offset: $\log(\lambda) = X\beta + \log(N)$.

• Variance heteroskedasticity: variance of $p$ is $p(1−p)/N$; small N yields high variance. Weighting by N or modeling counts rather than rates avoids misleading fits.

• Link functions & transforms: apply logit: $\text{logit}(p) = \log(p/(1−p))$ for unconstrained prediction, or use Beta regression when modeling continuous rates in (0,1) with precision parameter $\phi$.

• Hierarchical / partial pooling: for many campaigns with limited history, use mixed-effects models or Bayesian hierarchical models to share signal across campaigns and avoid overfitting single-campaign noise.

• Time structure & seasonality: include lag features, rolling averages, weekly/holiday indicators, and consider state-space / ARIMA or `Prophet` for seasonality and trend components when behavior is temporal.

• Evaluation & CV: use time-series cross-validation (rolling-origin) and metrics robust to rate scale: MAE, RMSE on probabilities, calibration/Brier score, and coverage of prediction intervals; avoid random CV that leaks future info.

• Uncertainty quantification: provide prediction intervals (e.g., binomial CI, bootstrap, Bayesian posterior predictive intervals, or quantile regression); communicate heteroskedastic uncertainty driven by N.

• Cold-start / sparse campaigns: backfill with cohort-level averages, metadata-based similarity (channel, creative type), or use meta-learning / embedding features so new campaigns borrow strength.

• Feature engineering: include spend, bids, placement, creative attributes, prior-period clicks, device mix, price, inventory, and external signals (holidays). Interaction terms often matter (promotion × channel).

• Practical scale/tradeoffs: per-campaign Bayesian hierarchical models work well for thousands of groups; for tens of millions of rows, use regularized tree models like `LightGBM` or aggregated GLMs via `statsmodels`/`scikit-learn`. If N ~ 10M+ time series, pre-aggregate to daily or weekly per campaign.

• Model selection & business objective: define whether you optimize point forecasts (MAE) or tail risk (quantiles) or ranking (AUC for high/low converters). Choose objective aligned to downstream use (budget allocation vs. alerting).

• Bias from promotions & experiments: flagged promotions or ongoing A/B tests confound historical patterns; include experiment assignment as covariate or exclude those windows.

Worked example — Predict Next-Period Conversion Rate Using Historical Campaign Data

First 30s framing: ask the aggregation level (per campaign per day?), forecast horizon (next day, week), definition of conversions and exposures, available covariates (spend, creatives, promotions), and what decisions will use the forecast. Skeleton answer pillars: (1) exploratory baseline — compute weighted rolling mean and binomial CIs to establish naive accuracy; (2) statistical models — fit a Binomial GLM (logit link) or Poisson with exposure offset including lags, seasonality and promotion flags; (3) hierarchical pooling — if many low-volume campaigns, fit random intercepts by campaign to shrink noisy estimates; (4) ML enhancements — a gradient-boosted tree (e.g., `XGBoost`/`LightGBM`) on campaign features for nonlinearity, but with sample weighting by N. Flag one tradeoff explicitly: per-campaign models capture idiosyncratic trends but overfit when N is small; hierarchical models reduce variance but may underfit outlier campaigns. Close with deployment-readiness: backtest with rolling-origin CV, report prediction intervals and calibration; "if I had more time, I'd incorporate uplift/campaign-treatment indicators to separate organic conversion from campaign effect and tune hierarchical priors."

A second angle

Forecasting conversion for a brand-new campaign (no history) forces different choices: rely on meta-features (audience, creative type, channel) and cohort averages, plus a hierarchical model where the campaign-level effect is drawn from a group distribution estimated from historical campaigns. You might train a model to predict campaign-level baseline conversion from metadata (a regression or embedding + `LightGBM`) and then update quickly with Bayesian online updating as early impressions arrive. This variant emphasizes transfer learning and uncertainty (wide initial intervals) more than time-series smoothing.

Common pitfalls

Pitfall: Treating rates as homoskedastic and optimizing MSE on $p$ without accounting for exposure. This downweights high-volume campaigns and leads to overconfident predictions on low-N campaigns. Instead, model counts or weight by N.

Pitfall: Showing only point forecasts to stakeholders. Business users need prediction intervals — especially for low-volume campaigns where intervals can be orders of magnitude wider.

Pitfall: Using standard random CV. Time leakage yields over-optimistic error estimates; always use rolling-origin or blocked CV matching production forecasting cadence.

Connections

This topic connects directly to A/B testing / experimentation (you must separate organic baseline vs. treatment effects) and to uplift modeling (predict incremental conversions from a campaign). Interviewers may pivot to model monitoring: drift detection on conversion distributions or business-ROI evaluation.

Further reading

• [Bayesian Data Analysis — Gelman et al.] — authoritative treatment of hierarchical models and partial pooling for sparse groups.

• Prophet (Taylor & Letham) — pragmatic time-series trend/seasonality model often used for business forecasting.

Focus area — Added for Walmart relevance: connect predictions to availability, stockouts, safety stock, and operational decisions.

What's being tested

Interviewers probe your ability to turn sales and inventory signals into defensible analytics and ML models that reduce stockouts and optimize replenishment. They expect crisp metric design (service levels, fill rate), statistical reasoning about demand variability and lead time, causal thinking for policy changes, and practical model-evaluation choices that matter for business outcomes rather than pure ML metrics. Walmart cares because small improvements in stock availability meaningfully increase revenue and reduce customer churn; the interviewer wants to see you balance statistical rigor, business impact, and operational constraints.

Core knowledge

• Stockout vs backorder vs lost sales: define the observable in data—stockout event when on-hand inventory ≤ 0; backorder if order is accepted; lost sale if demand signal disappears. Distinguish each for correct metric attribution.

• Fill rate and service level: fill rate = fulfilled units / demanded units; cycle service level = P(no stockout during lead time). These are related but not identical; design experiments around the one that ties to revenue.

• Reorder point (R) and safety stock (SS): classical formula R = expected demand during lead time + SS, with SS = z * σ_d * sqrt(LT) assuming iid demand; specify distributional assumptions and z (target service percentile).

• Lead time variability: when lead time is stochastic, use E[demand]E[LT] and Var(demandLT) terms; approximate SS with delta method or simulate if independence fails.

• Forecasting errors & metrics: use MAE, RMSE, and MAPE but prefer scale-free metrics (sMAPE, MASE) when comparing SKUs of different volumes; calibrate error distribution tails—stockouts depend on tail behavior, not just mean error.

• Probabilistic forecasts: prefer full predictive distributions (quantiles) over point forecasts for safety-stock sizing; evaluate with CRPS or quantile loss because upper quantiles inform reorder decisions.

• Causal evaluation: for policy changes (faster replenishment), use randomized experiments or quasi-experimental methods like difference-in-differences and synthetic controls; check for inventory-level spillovers across SKUs/stores.

• Aggregation & hierarchy: forecast at SKU-store-day if you can; aggregate forecasts upward carefully—sum of medians != median of sums. Use top-down/bottom-up reconciliation (e.g., MinT) when required.

• Sparse low-volume SKUs: use hierarchical Bayesian shrinkage or Poisson/Negative Binomial demand models to borrow strength; naive ML will overfit noise on intermittent series.

• Anomaly detection for stockouts: monitor KPI shifts like sudden rise in stockout rate or fall in fill rate; use change-point tests (CUSUM) or control charts on rolling windows, and prioritize by monetary impact.

• Experiment metric design: choose primary metric tied to customer experience (e.g., % orders without substitution), power calculations must account for clustered traffic (store-SKU clusters) and overdispersion in counts.

• Model evaluation aligned to business: AUC or RMSE are secondary; evaluate using "lost revenue avoided," reduction in emergency shipments, or inventory turns improvement simulated in a simple inventory policy.

Worked example — "Estimate effect of faster replenishment policy on SKU-level stockouts"

Frame: ask clarifying questions—what is the policy (reduced lead time vs more frequent batches), rollout scope, available data (POs, on-hand inventory, sales, store-level prices/promotions), and whether stockouts lead to lost sales or backorders. Skeleton: (1) define primary metric (SKU-store 7-day stockout rate or lost-sales dollars), (2) choose evaluation design (prefer randomized A/B at store-SKU or rollout stagger enabling difference-in-differences), (3) control for demand shocks (promotions, seasonality) using covariates or fixed effects, (4) estimate treatment effect and heterogeneity, (5) translate effect into business value via revenue uplift or reduced emergency freight. Tradeoff: randomization gives unbiased causal effect but may be operationally costly; a staggered rollout with good pre-trends and store-SKU fixed effects trades some internal validity for feasibility. Close: report intent-to-treat and per-protocol estimates, run robustness checks (placebo windows), and say "if more time, I'd build a counterfactual simulator combining probabilistic forecasts with the new lead-time distribution to quantify long-run inventory economics."

A second angle — "Detect and prioritize SKU-store anomalous stockout spikes"

Here the problem is detection and triage rather than causal estimation. Build SKU-store daily time series of stockout indicator and use hierarchical smoothing to reduce noise. Apply change-point detection (e.g., rolling z-score on smoothed rate, CUSUM) with minimum-impact threshold in dollars to prioritize. Then perform quick root-cause regression: regress stockout spike magnitude on covariates (shipments missed, lead time increases, sudden demand surge, price changes) to surface likely causes. The core concept is the same—tie metrics to business impact and prefer aggregated, probabilistic signals—while constraints emphasize speed, interpretability, and prioritization rather than causal attribution.

Common pitfalls

Pitfall: Designing metrics that confuse availability with demand suppression.
Analysts often treat decreased sales during stockouts as lower demand; instead, mark missing sales as censored and avoid evaluating policy impact on naive sales counts without correcting for lost sales.

Pitfall: Using point-forecast error (RMSE) to size safety stock.
Because stockouts depend on tails, optimizing RMSE won't minimize lost sales; prefer quantile forecasts (e.g., 95th percentile) and evaluate with quantile loss or CRPS.

Pitfall: Ignoring clustering and spillovers in experiments.
Randomizing at order-line level while replenishment is at store-SKU or supplier level induces contamination; state cluster-level randomization and account for intra-cluster correlation in power calculations.

Connections

Interviewers may pivot to demand forecasting techniques (time-series, hierarchical models) or to supply chain optimization topics like lot-sizing and transportation cost tradeoffs; they may also ask about model monitoring and data drift for forecast degradation. Be ready to move from causal effect to simulation-based decisioning.

Further reading

• Forecasting: Principles and Practice (Hyndman & Athanasopoulos) — practical time-series forecasting methods and evaluation, strong on probabilistic forecasts.

• Forecasting at Scale (Taylor & Letham, Prophet) — introduces pragmatic forecasting with seasonality and changepoints, useful for business calendars.

• Silver, Pyke, Peterson — Inventory Management and Production Planning and Scheduling — classic treatment of safety stock, service levels, and inventory policy tradeoffs.

Practice questions

Focus area — Added for Walmart marketplace scale; practice anomaly framing, class imbalance, precision-recall, and alert quality.

What's being tested

Interviewers are probing your ability to detect, quantify, and mitigate fraud/anomalies in a retail marketplace using statistical reasoning, metric design, and model evaluation. Expect to justify metric choices under heavy class imbalance, reason about label delay/noise, design experiments that measure interventions’ causal impact, and trade off business cost versus customer friction. At Walmart scale they care about robust diagnostics, explainability, and tight decision thresholds that optimize expected loss, not just raw ML metrics.

Core knowledge

• Fraud types: Understand account takeovers, fake listings, payment fraud, return/refund abuse, and review manipulation; each implies different signals, latency, and remediation cost.

• Signal sources: Instrumentation includes order events, payment events, account creation/changes, device/IP telemetry, review activity, and manual review labels — treat these as upstream signals, not infra design.

• Labels & delay: Chargebacks and manual reviews are reliable but delayed; use positive-unlabeled (PU) learning, survival analysis, or delayed-label evaluation windows to avoid optimistic bias.

• Imbalanced metrics: Prefer precision@k, precision-recall AUC, and business-oriented expected-utility over ROC AUC; precision = TP/(TP+FP), recall = TP/(TP+FN), expected value = TPB - FPC.

• Supervised vs unsupervised: Use supervised (XGBoost, LightGBM) when labels exist; use Isolation Forest, LOF, or reconstruction autoencoders for unlabeled anomaly scoring; hybridize with rule-based heuristics.

• Feature engineering: Temporal aggregates, session statistics, velocity features, graph features (buyer-seller degree, edge churn), embeddings for sequences, and cross-feature interactions capture behavior patterns.

• Model validation: Use time-based cross-validation to prevent leakage; backtest with rolling windows and simulate interventions; calibrate scores with Platt or isotonic methods for threshold decisions.

• Cost-sensitive thresholds: Optimize threshold using business cost matrix (Cost_FP, Cost_FN) via expected-utility maximization or constrained optimization (maximize TP subject to FP ≤ budget).

• Multiple testing & alerting: For many concurrent anomalies, control false discoveries with Benjamini–Hochberg or use hierarchical alerting to prioritize impactful signals.

• Drift & monitoring: Monitor feature distributions, precision@k over time, and post-decision outcomes (reversal rates) to detect concept drift; set SLA for investigation latency.

• Explainability & compliance: Use SHAP for feature attribution, produce human-readable rules for blocking actions, and track appeal/override rates to refine models.

• Experimental design: For interventions (blocks, holds), run randomized rollouts or difference-in-differences when randomization isn't possible; monitor retention, conversion, and false-positive harm.

Worked example — “Detect anomalous spike in seller chargebacks”

Frame: ask what defines a chargeback spike (absolute vs relative), acceptable detection latency, business impact per chargeback, and whether rollout is global or cohorted. Skeleton: (1) define metric chargebacks_per_1000_orders and rolling baseline (7- and 28-day), (2) run statistical anomaly detectors (EWMA/CUSUM) and segment by seller cohorts, (3) drill down with cohort and graph features to distinguish correlated spikes (e.g., bot campaign) from isolated seller fraud, (4) if labels exist, train a classifier to predict future chargebacks. Tradeoff to flag: sensitivity vs operational load — set thresholds by expected cost (TPBenefit - FPCost). Close: if more time, build seller-graph propagation features, run an A/B test of a soft-hold intervention, and instrument downstream merchant appeals to measure false-positive harm.

A second angle — “Design a model to score new accounts for onboarding risk”

Same core ideas apply but constraints shift: new accounts lack history, so rely on device/IP signals, behavioral heuristics, and graph-based transductive features (shared payment instruments). Use XGBoost with strong regularization and calibrated outputs. Because false rejections hurt acquisition, optimize for high recall at low FP cost using cost-sensitive learning or low-friction mitigation (phone verification) instead of outright block. Validate with synthetic labels (simulated fraud) and monitor cohort conversion and lifetime-value to ensure onboarding friction doesn't reduce long-run revenue.

Common pitfalls

Pitfall: Over-relying on ROC AUC for imbalanced fraud problems. ROC can be misleading when positive prevalence is tiny; prefer precision-recall curves, precision@k, and business-utility metrics.

Pitfall: Ignoring label delay and leakage. Training on chargeback-confirmed events without accounting for delay leads to optimistic models; always define an evaluation window and consider PU or survival approaches.

Pitfall: Presenting only model metrics without business impact. Saying “AUC=0.92” is weaker than “At threshold X, we prevent $Y of expected loss while causing Z customer holds per day.”

Connections

Interviewers may pivot to graph analytics (fraud rings), causal inference/experimentation (measuring interventions), or model-monitoring and MLOps questions (drift detection, rollback policies). Familiarity with cross-functional tradeoffs (legal/compliance, operations) strengthens answers.

Further reading

• Chandola, Banerjee, Kumar — “Anomaly Detection: A Survey” (2009) — comprehensive algorithms and taxonomy useful for method selection.

• Bolton & Hand — “Statistical Fraud Detection: A Review” (2002) — classic on statistical approaches and evaluation pitfalls.

Practice questions

Focus area — Added for Walmart logistics relevance; review feature design, delay prediction, uncertainty, and business impact.

What's being tested

Interviewers are probing your ability to build, evaluate, and operationalize ETA (Estimated Time of Arrival) models for fulfillment — balancing predictive accuracy, calibrated uncertainty, and business impact. You'll be judged on metric design, label engineering (including censoring and late-arrival handling), segmentation/cohort evaluation, and how you trade off model complexity versus interpretability for downstream uses like SLAs and customer communications.

Core knowledge

• Label definition & censoring: define ETA as either time-to-event (seconds/minutes) or absolute timestamp; account for right-censoring when deliveries are in-flight at evaluation cutoff and for missing final-status events.

• Time-to-event / survival framing: use survival analysis or hazard models when censoring matters; survival function $S(t)=P(T>t)$ and hazard $h(t)=\frac{f(t)}{S(t)}$ let you produce full distributional ETAs.

• Distributional predictions & quantiles: prefer quantile regression or direct interval models to produce p50/p90/p95 estimates; optimize pinball loss for quantiles: $L_{q}(u)=\max(qu, (q-1)u)$ where $u=y-\hat{y}$.

• Point metrics vs business metrics: report MAE and RMSE for point accuracy, but prioritize p90/p95 on SLA cohorts and weighted errors for high-value orders; MAE = $\frac{1}{n}\sum|y_i-\hat{y}_i|$.

• Calibration & reliability: check coverage for predicted intervals (e.g., 95% CI covers ~95% of outcomes); use reliability diagrams and conformal prediction for distributional calibration.

• Feature design (signals): pull timestamps and status updates from Postgres/event logs and real-time signals like GPS, courier assignment, historical route durations, weather, and order size. Encode time-to-pickup, time-of-day, weekday, and distance-to-destination as core predictors.

• Model families & tradeoffs: tree ensembles (XGBoost, LightGBM) are strong baselines for tabular ETA; survival forests or gradient-boosted survival models handle censoring. Neural sequence models help for spatiotemporal streams but cost latency.

• Evaluation windows & leakage: split data by event time (not ingestion time) and avoid using signals that only exist post-label (e.g., final scan). Simulate production latency: only use features available at the prediction time.

• Error attribution & segmentation: calculate cohort-level metrics (by region, courier, order size) and delta decomposition — model error = bias + variance + irreducible noise — to prioritize feature collection or local models.

• Experimentation & causal inference: when ETA model changes affect downstream behavior (re-routing, customer cancellations), design A/B tests instrumenting treatment exposure; measure both prediction metrics and business KPIs (on-time rate, customer complaints).

• Operational concerns (DS scope): specify monitoring metrics: drift on feature distributions, model calibration, p95 SLA misses; propose retraining cadence and backfills, but leave pipeline plumbing to Data Engineering.

• Uncertainty communication: for UX, prefer range + confidence (e.g., ETA 45–60 min, p90=60) and explain tradeoff: tighter intervals increase risk of SLA misses.

Worked example — "Predict package ETA for last-mile delivery"

Frame: first ask clarifying Qs — what is the prediction timestamp (order placed, last-mile dispatch?), what SLA or business metric matters (p95 on same-day deliveries?), and what latency constraints exist for serving. Skeleton: (1) define label precisely (time from prediction timestamp to final delivery), handle right-censoring for undelivered events, (2) assemble features available at prediction time (courier status, route distance, real-time GPS if available, historical travel-times by route/time), (3) choose modeling approach — baseline LightGBM for point p50 with separate quantile models for p90/p95 or a survival-gradient-boosted model, (4) evaluation plan — time-based cross-validation, report MAE and p95 miss-rate by cohort, and (5) calibration step — isotonic regression or conformal intervals to ensure coverage. Tradeoff to flag: whether to serve a single global model (faster, less overfit) or per-region models (better local accuracy, more maintenance). Close with: if I had more time I'd instrument an online A/B test to measure behavioral impacts of narrower ETAs on cancellations and monitor p95 drift post-launch.

A second angle — "Detect systemic ETA drift causing p95 misses"

This reframes the problem from prediction to monitoring and root-cause. Start by defining the alerting metric: an increase in p95 ETA error or p95 SLA miss-rate across cohorts. Approach: segment by courier, time-of-day, and route; run significance tests on recent vs baseline error distributions (use bootstrapped CI or sequential testing). For causes, compare feature distribution shifts (e.g., average route speed down by X%), correlate external signals (traffic, weather), and run counterfactual checks: was a model change deployed? The DS role focuses on diagnostics and causal attribution, not fixing the streaming pipeline — propose remediation: retrain on recent data, change feature windows, or fall back to simpler robust baseline. Emphasize validating fixes with pre-post A/B tests to confirm reduced p95 misses.

Common pitfalls

Pitfall: leaking future signals into features — including post-dispatch scans or final courier-reported durations in training gives unrealistically low errors that fail in production.

Pitfall: reporting only point estimates (mean/p50) — stakeholders care about tail behavior; a model with good MAE but poor p95 will still break SLAs.

Pitfall: optimizing global metrics without cohort checks — an improvement in average error can hide regressions on high-value or rural cohorts; always report segmented KPIs and business-weighted objectives.

Connections

This area commonly pivots to adjacent problems: inventory provisioning (how ETA uncertainty affects safety stock), dynamic routing (real-time ETA feeds into route optimization), and experimentation infrastructure (sequential testing and measurement of downstream behavioral effects).

Further reading

• Klein & Moeschberger, Survival Analysis — practical survival methods and censoring essentials for time-to-event modeling.

• Vovk, Gammerman, Shafer — Algorithmic Learning in a Random World (Conformal Prediction) — useful for uncertainty-calibrated interval predictions.

Focus area — Added for production ML at Walmart scale; know drift, retraining triggers, metric monitoring, and incident triage.

What's being tested

Interviewers are checking your ability to operationalize model quality at retail scale: define meaningful drift, detect it statistically, prioritize alerts, and propose actionable mitigations that align with business metrics. They want to see statistical reasoning (power, false positives), signal selection (which features and KPIs to monitor), and root-cause diagnosis methods that a Data Scientist owns — not the low-level ingestion plumbing. Expect to justify thresholds, handle label delay, and demonstrate how monitoring ties back to downstream metrics like conversion or inventory turn.

Core knowledge

• Types of distributional shift: know covariate shift (P(x) changes), label shift (P(y) changes), and concept drift (P(y|x) changes); each implies different detection and remediation strategies.

• Univariate drift tests: use KS test, PSI (Population Stability Index), or shift in summary stats (means/quantiles); PSI ≈ Σ (Actual% - Expected%)*ln(Actual%/Expected%) with thresholds (PSI<0.1 small, 0.1–0.25 moderate).

• Multivariate detection: for joint changes use adversarial validation (train classifier to distinguish old vs new), Wasserstein distance, or embedding-space drift (distance in learned representation).

• Business KPIs first: monitor downstream metrics (DAU, conversion rate, revenue per user) and model metrics (AUC, Brier score, calibration) in parallel — sometimes KPI change is the only actionable signal.

• Label delay & proxy metrics: when labels are delayed (returns, conversions), rely on short-term proxies (clicks, add-to-cart), calibration and score-distribution shifts to surface issues before labels arrive.

• Significance, effect size, power: compute sample size with $n \approx \left(\frac{Z_{1-\alpha/2}+Z_{1-\beta}}{\Delta/\sigma}\right)^2$; report minimum detectable effect and expected false alarm rate.

• Seasonality and context: control for weekly/holiday/promotion cycles and geography; compare like-for-like windows (same weekday, promotion status) to avoid false positives from expected seasonality.

• Segmentation & cohort analysis: monitor at global and per-segment level (by SKU, store, geography, new vs returning users) to catch localized drift that averages out globally.

• Root-cause tools: use per-feature contribution changes (feature distributions, SHAP SHAP value drift), conditional checks (P(x_i|segment)), and adversarial feature importance to localize causes.

• Operational thresholds & alerting: define alert tiers — watch (small change, automated re-check), investigate (statistically significant with effect size), action (business impact). Expect tradeoffs: sensitivity vs alert fatigue.

• Retraining vs intervention: prefer targeted interventions (feature fixes, recalibration, business rule overrides) when concept drift is transient; schedule retrain/backtest when persistent drift exceeds business impact thresholds.

• Validation for retrain: ensure temporal cross-validation/backtest; use holdout that preserves time ordering and simulate deployment evaluation to avoid information leakage.

Worked example

Prompt (typical): "Design a monitoring plan for a store-level demand-forecasting model to detect feature drift and trigger investigation." First 30s you'd clarify: how frequently forecasts run, label availability latency (sales finalize after returns?), which downstream KPIs (stockouts, overstock cost) matter, and segmentation granularity (per-SKU per-store or aggregated). Organize the answer around three pillars: (1) Define drift signals — monitor key input features (promotions, price, historical sales), forecast residuals and calibration; (2) Detection algorithms & thresholds — univariate PSI/KS test per feature, adversarial validation for multivariate shift, and residual shift significance with power calculation for expected sample sizes; (3) Investigation & action flow — automatic cohort drilldowns (by store/sku/promo), feature-attribution via SHAP, and pre-defined mitigation (freeze forecasts for affected SKUs, notify supply chain). A concrete tradeoff to flag: increasing sensitivity (shorter windows) detects drift faster but raises false positives during sales/holiday spikes — mitigate by comparing to historical seasonal baselines and requiring persistence (e.g., 3 consecutive alerts). Close with: "If I had more time I'd prototype adversarial validation on embedding features and run offline backtests that simulate retrain frequency vs business impact (stockouts avoided per retrain)."

A second angle

Consider a ranking/recommendation model where labels are delayed and position bias exists. The same concept applies: monitor input distribution (user features, item catalog embeddings), score distribution, and downstream KPI (click-through and conversion). But constraints differ: you must correct for position bias (use propensity weighting or interleaving experiments) and evaluate top-k metrics (NDCG) rather than global AUC. Detection methods should emphasize top-of-funnel shifts (new popular items, inventory outs) and per-user segment drift (cold-start cohorts). Root-cause uses pairwise examination of recommended item features and recent catalog changes (new product launches, de-listings) that can cause sudden, high-impact drift.

Common pitfalls

Pitfall: Confusing seasonality with drift.
Many candidates trigger alarms on obvious holiday or promo cycles instead of comparing to matched historical windows; always normalize for expected cycles before declaring drift.

Pitfall: Ignoring label delay when rooting causes.
A tempting but wrong move is to conclude model degradation from immediate KPI dips without considering delayed conversions or returns; present a plan using proxies and delayed-label-aware diagnostics.

Pitfall: Promising infra or deployment fixes beyond DS scope.
Don't assume ownership of streaming ingestion, alert routing, or retrain pipelines — focus on the monitoring signals, statistical thresholds, and concrete investigative steps a Data Scientist would deliver.

Connections

Monitoring and drift work naturally lead to model lifecycle strategy (retrain cadence, canary evaluation) and to experimentation/causal analysis (did a marketing campaign cause the shift?). Interviewers may pivot to model governance (bias/fairness monitoring) or to feature-engineering practices (feature stores, feature lineage) to ask about traceability.

Further reading

• Hidden Technical Debt in Machine Learning Systems — Sculley et al., NIPS 2015 — explains monitoring and maintenance burdens in production ML.

• A Survey on Concept Drift Adaptation (Gama et al.) — comprehensive taxonomy of drift types and adaptation strategies.

Statistics & Math

Focus area — Your Stats rustiness makes observational causal inference, balance checks, overlap, and assumptions worth extra review.

What's being tested

Interviewers are testing whether you can use propensity score matching as part of a credible observational causal study, not just describe it mechanically. For an Amazon Data Scientist, this matters because many decisions involve non-random exposure: customers receive reminders, recommendations, coupons, ads, or seller interventions based on prior behavior, which creates selection bias. The interviewer is probing whether you can define treatment and outcome cleanly, identify confounders, estimate treatment effects under explicit assumptions, diagnose balance, and explain when matching is weaker than alternatives like difference-in-differences or randomized experiments. A strong answer sounds like a causal analysis plan, not a generic ML workflow.

Core knowledge

• Propensity score is the probability of receiving treatment conditional on observed covariates: $e(X)=P(T=1 \mid X)$. Matching on $e(X)$ compresses many covariates into one score, making treated and control units more comparable under the right assumptions.

• Treatment effect estimands must be named before modeling. ATT estimates impact on treated users, $E[Y(1)-Y(0)\mid T=1]$; ATE estimates impact over the full population. Reminder, coupon, or recommendation analyses often care about ATT because the business asks, “What was the effect on customers who actually received it?”

• Unconfoundedness means potential outcomes are independent of treatment after conditioning on observed covariates: $(Y(1),Y(0)) \perp T \mid X$. This is untestable and is the central weakness of matching; missing drivers like purchase intent, seasonality, or prior reminder eligibility can invalidate the estimate.

• Common support or overlap requires comparable treated and control units at similar propensity scores: $0 < P(T=1\mid X) < 1$. If high-propensity treated users have no comparable controls, trim them or report that the effect is only identified for the overlap population.

• Covariate selection should include pre-treatment variables that affect both treatment and outcome, such as prior purchases, browse frequency, tenure, Prime status, device type, geography, prior notification engagement, and baseline spend. Do not control for post-treatment mediators like clicks caused by the reminder.

• Matching methods include nearest-neighbor matching, caliper matching, exact matching on key strata, Mahalanobis distance within propensity calipers, and matching with or without replacement. A common rule is a caliper of $0.2$ standard deviations of the logit propensity score, though sensitivity checks matter more than blindly applying the rule.

• Balance diagnostics are more important than predictive accuracy. Use standardized mean difference: $SMD = \frac{\bar X_T-\bar X_C}{\sqrt{(s_T^2+s_C^2)/2}}$ and aim for absolute SMD below roughly 0.1 after matching. Also inspect propensity score overlap plots and variance ratios.

• Propensity model quality is not judged by AUC alone. A very high AUC can indicate treated and control groups are hard to compare, worsening overlap. Logistic regression is interpretable; XGBoost or random forests can capture nonlinear assignment, but still require balance checks.

• Effect estimation after matching usually compares matched outcomes: $\hat\tau_{ATT} = \frac{1}{N_T}\sum_{i:T_i=1}(Y_i-\sum_{j:T_j=0}w_{ij}Y_j)$. Use robust or bootstrap standard errors because matching induces dependence between observations.

• Weights versus matches are related but not identical. Inverse probability weighting uses weights like $T/e(X)+(1-T)/(1-e(X))$ for ATE, while matching forms comparable pairs or sets. Weighting can be efficient but unstable when scores are near 0 or 1.

• Time alignment is critical for DS causal work. Define covariates using only data before treatment exposure, define a clean treatment timestamp, and measure outcomes over a fixed post-period such as 7-day conversion, 14-day revenue, or weekly active users.

• Scale considerations matter in implementation but not as data engineering design. For millions of users, estimate scores in Python with sklearn or statsmodels, then use approximate nearest-neighbor libraries or stratified matching; for tens of millions, coarsened strata or weighting may be more practical than full pairwise matching.

Worked example

For Design causal study for reminder impact, a strong candidate would start by clarifying: “What reminder are we studying, who was eligible, when was it sent, and what primary outcome matters: purchase_conversion, GMV, repeat_visit, or retention?” They would declare that randomization is preferred, but if the reminder was already deployed non-randomly, they would frame this as an observational causal inference problem.

The answer skeleton should have four pillars. First, define treatment as receiving the reminder during a specific window and define the outcome over a fixed post-treatment horizon. Second, construct pre-treatment covariates capturing baseline customer behavior: prior sessions, purchases, category interest, reminder history, Prime status, tenure, and seasonality indicators. Third, estimate propensity scores and perform matching or weighting while checking common support and covariate balance. Fourth, estimate ATT and run robustness checks such as placebo outcomes, alternative calipers, trimming, and segment-level heterogeneity.

One tradeoff to flag is whether to use propensity score matching alone or combine it with difference-in-differences. If treated users were already trending differently before the reminder, simple matching on static covariates may not remove bias; matching users on baseline features and then applying DiD on pre/post outcomes can be stronger if parallel trends look plausible.

A good close would be: “If I had more time, I would compare this observational estimate against any historical holdout or staggered rollout, because a randomized or quasi-random design would be more credible than relying entirely on selection-on-observables.”

A second angle

For Explain Propensity Score Matching and Assess Covariate Balance, the emphasis shifts from study design to technical diagnostics. You should define the score, explain why matching reduces observed covariate imbalance, and immediately state the assumptions: unconfoundedness, overlap, and correct time ordering. The interviewer may push on how you know matching worked; the right answer is not “the propensity model has high accuracy,” but “post-match SMDs are small, score distributions overlap, and key business covariates are balanced.” You can also mention that if balance remains poor, you would revise covariates, use exact matching on critical dimensions like marketplace or customer segment, change calipers, trim non-overlap, or move to a different design.

Common pitfalls

Pitfall: Treating PSM like a magic bias-removal tool.

The tempting answer is “we match treated and untreated users, then compare outcomes, so we have causality.” That misses the key limitation: matching only adjusts for observed pre-treatment confounders. A better answer explicitly says what unobserved factors could remain, such as customer intent, concurrent promotions, or recommendation ranking changes.

Pitfall: Optimizing the propensity model for prediction instead of balance.

A candidate may emphasize AUC, feature importance, or complex models because that sounds rigorous. For causal matching, predictive performance is secondary; the interviewer wants to hear balance diagnostics, common support, SMD thresholds, and sensitivity checks.

Pitfall: Communicating only formulas without business framing.

It is not enough to write $e(X)=P(T=1\mid X)$ and list assumptions. Tie the method to a real decision: whether reminders incrementally increase purchases, whether the estimate applies only to reachable users, and whether the expected lift justifies expanding the program.

Connections

Interviewers often pivot from propensity score matching to difference-in-differences, synthetic control, instrumental variables, uplift modeling, or double machine learning. They may also ask how your observational estimate compares with an A/B test, especially around selection bias, interference, novelty effects, and metric guardrails.

Further reading

• Rosenbaum and Rubin, “The Central Role of the Propensity Score in Observational Studies for Causal Effects” — the foundational paper defining propensity scores and their balancing property.

• Imbens and Rubin, Causal Inference for Statistics, Social, and Biomedical Sciences — rigorous treatment of potential outcomes, matching, weighting, and identification assumptions.

• Austin, “An Introduction to Propensity Score Methods for Reducing the Effects of Confounding in Observational Studies” — practical guidance on matching choices, balance diagnostics, and applied pitfalls.

Focus area — Added because you explicitly said Stats concepts are rusty; rebuild confidence on distributions, confidence intervals, p-values, and assumptions.

What's being tested

Interviewers are probing your ability to reason about uncertainty: how to turn noisy samples into reliable decisions using probability and statistical inference. Expect to explain and compute hypothesis tests, confidence intervals, sample-size/power calculations, and to justify assumptions (randomization, independence, measurement). Walmart Labs cares because product and pricing choices depend on correctly measured effects (statistical and practical) and safe stopping/segmentation rules.

Core knowledge

• Central Limit Theorem (CLT): averages of i.i.d. draws converge to Normal for moderate n (rule of thumb n≥30), enabling z-tests; heavy tails or dependence break CLT assumptions.

• Standard error (SE) for a mean: $SE=\sigma/\sqrt{n}$; for difference of proportions $SE=\sqrt{\frac{p_1(1-p_1)}{n_1}+\frac{p_2(1-p_2)}{n_2}}$; use these to form z/t statistics.

• t-test vs z-test: use t-test when population σ is unknown and sample sizes are small; use Welch's t-test when variances are unequal (heteroscedasticity).

• Hypothesis testing vocabulary: Type I error (α), Type II error (β), statistical power = 1−β; p-value = probability of data (or more extreme) under null — not the probability the null is true.

• Sample-size / MDE formula (two-sample means): $n\approx\frac{(z_{1-\alpha/2}+z_{1-\beta})^2( \sigma_1^2+\sigma_2^2)}{\delta^2}$ where δ is minimum detectable effect (MDE); for proportions substitute Bernoulli variances.

• Multiple testing: Bonferroni controls family-wise error (α/m) — conservative; Benjamini-Hochberg controls false discovery rate (FDR) — less conservative for many metrics.

• Sequential monitoring and peeking: unadjusted repeated looks inflate Type I error; use alpha-spending (Pocock/O'Brien–Fleming) or sequential methods (e.g., always-valid p-values) or pre-specified group sequential plan.

• Nonparametric and resampling: use permutation tests for exchangeable data and bootstrap for CI when analytic SE is unreliable (small n, heavy tails, complex estimators).

• Rare events and rates: for low counts prefer exact binomial tests or Poisson/negative-binomial models; normal approximation fails when counts < ~10.

• Causal assumptions for observational inference: randomization, SUTVA (no interference), and ignorability/conditional exchangeability for regression/propensity adjustments.

• Effect size vs p-value: always report confidence interval and absolute/relative effect; small p with trivial effect is not actionable.

• Tip: wrap library names when you plan to run tests in code: use scipy.stats, statsmodels, or pingouin for standard routines.

Worked example — A/B test for conversion rate

Frame the problem: ask what the primary metric is (binary conversion?), unit of randomization (user, session), baseline conversion rate p0, desired MDE, α and power. First pillars: (1) pre-specify hypothesis H0: pA=pB vs H1: pA≠pB, (2) compute sample size using the proportions SE formula and MDE, (3) ensure randomization and instrumentation checks, (4) run test and compute z-statistic and two-sided CI, (5) report effect size and business interpretation. Explicit tradeoff: choosing a smaller α or detecting a smaller MDE increases required n (longer experiment), so negotiate business cost of time vs Type I risk. For stopping, either pre-specify a fixed-horizon analysis or adopt an alpha-spending plan; do not peek without adjustment. Close by saying: if more time, I'd add subgroup heterogeneity pre-specification, run permutation tests for robustness, and compute Bayesian posterior for intuitive credible intervals.

A second angle — regression adjustment in observational A/B-like comparison

If assignment isn't randomized, frame identification: state required conditional ignorability and list covariates to control (time, geography, prior behavior). Pillars: (1) model outcome with covariate adjustment or inverse-propensity weighting, (2) compute robust (sandwich) standard errors, (3) check overlap/balance and do sensitivity analysis for unobserved confounding. Tradeoffs: regression improves precision if model is correct, but model misspecification can bias estimates; propensity-score trimming mitigates extreme weights. This applies the same inference machinery (SE, CI, hypothesis tests) but emphasizes identification rather than randomization.

Common pitfalls

Pitfall: Interpreting a small p-value as the probability the null is false. This is wrong; instead report the effect estimate and confidence interval for practical relevance.

Pitfall: Peeking at results and stopping early without correction. Unadjusted repeated looks raise false positives; either fix sample size or use sequential corrections.

Pitfall: Reporting relative lifts only. A 10% lift on a 0.1% baseline is often meaningless; always show absolute differences and translate to business impact.

Connections

Interviewers can pivot from inference to causal inference (instrumental variables, difference-in-differences), uplift modelling (heterogeneous treatment effects), or to Bayesian A/B testing and always-valid inference techniques — be prepared to justify frequentist vs Bayesian choices.

Further reading

• [Imbens & Rubin, Causal Inference (2015) — textbook citation] — rigorous coverage of identification, randomization, and potential outcomes.

• [Efron & Tibshirani, An Introduction to the Bootstrap (1994) — book citation] — practical bootstrap methods for CIs and standard errors.

• [Wasserman, All of Statistics (2004) — textbook citation] — concise reference for hypothesis testing and asymptotics.

Practice questions

Analytics & Experimentation

Focus area — You flagged Stats rustiness; power, MDE, randomization, and metric interpretation are high-yield for Walmart experimentation.

What's being tested

Interviewers are probing the candidate’s ability to translate a business question into an experiment design with defensible statistical assumptions, compute a correct sample size, and surface operational constraints and diagnostics. They expect a Data Scientist to state clear assumptions (unit of randomization, primary metric, baseline, meaningful uplift), pick the right formula (binary vs continuous vs clustered), and explain tradeoffs like duration vs minimum detectable effect (`MDE`) and adjustments for peeking or multiple comparisons.

Core knowledge

• Definition of MDE: the smallest absolute effect size that matters to the business; sample size scales as $1/\text{MDE}^2$. Always state absolute vs relative uplift and tie to revenue or cost.

• Type I / Type II errors: `alpha` (false positive rate, commonly 0.05) and power ($1-\beta$, commonly 0.8); pick both before computing sample size and report per-arm and total sample.

• Binary metric formula: for equal allocation, per-arm sample for proportions:

  $$n = \frac{\left(Z_{1-\alpha/2}\sqrt{2\bar p(1-\bar p)} + Z_{1-\beta}\sqrt{p_1(1-p_1)+p_2(1-p_2)}\right)^2}{(p_1-p_2)^2}$$

  where $\bar p=(p_1+p_2)/2$.

• Continuous metric formula: per-arm sample:

  $$n = 2\left(\frac{Z_{1-\alpha/2}+Z_{1-\beta}}{\Delta/\sigma}\right)^2 = 2(Z_{1-\alpha/2}+Z_{1-\beta})^2\frac{\sigma^2}{\Delta^2}$$

  with $\sigma$ = standard deviation, $\Delta$ absolute difference.

• Cluster / group randomization: multiply by design effect `DE` = 1 + (`m` - 1) `ρ` where `m` = avg cluster size and `ρ` = intra-class correlation; cluster experiments often need orders of magnitude more clusters.

• Allocation ratio and unequal arms: sample size adjusts by factor (1 + 1/`k`) for treatment:control ratio `k`; unequal allocation increases total N for same power.

• Variance reduction techniques: stratification, blocking, and CUPED reduce $\sigma^2$ — directly lowering required N; demonstrate estimating pre-experiment covariance to compute expected reduction.

• Sequential monitoring / peeking: naive peeking inflates Type I error; use alpha-spending (O’Brien–Fleming, Pocock) or sequential tests, or pre-specify a fully Bayesian analysis to avoid corrections.

• Practical traffic/duration tradeoffs: compute required duration = required sample / expected daily eligible users; include seasonality and conversion latency when estimating real run time.

• Pre-experiment checks: confirm randomization via sample ratio mismatch (SRM) (binomial/chi-square test), validate metric instrumentation, and verify logged denominators (exposure counts) in `SQL`.

• Multiple metrics / multiple tests: control family-wise error with Bonferroni or control false discovery rate with Benjamini–Hochberg; report adjusted `p-value`s or pre-register primary metric.

• Realistic scaling numbers: small absolute lifts (0.1% on a 5% baseline) require millions of users; a 1 percentage point lift on a 10% baseline typically needs ~15k per arm (`alpha`=0.05, power=0.8).

Worked example — Calculate Sample Size for Effective A/B Test Design

First 30 seconds: clarify the randomization unit (user, session, cookie), the primary metric (binary conversion vs revenue-per-user), baseline rate or historical mean, the business-meaningful `MDE` (absolute), `alpha` and desired power, allocation ratio, and any clustering or stratification. Skeleton of a strong answer:

1. Estimate baseline and variance from recent telemetry and pick an absolute MDE tied to business (revenue or cost).

2. Choose the appropriate formula (binary vs continuous) and compute per-arm `n`; show the formula and plug numbers.

3. Adjust for clustering (`DE`) or for expected attrition and instrumentation loss by inflating `n`.

4. Plan monitoring: pre-register primary analysis, schedule SRM and instrumentation checks, and decide on alpha-spending if interim looks are needed.

A concrete tradeoff to flag: if traffic cannot support the required `n`, either extend duration, increase `MDE` (accept lower granularity), apply variance reduction (`CUPED`/stratification), or prioritize segments. Close with next steps: run a small pilot to validate variance estimates, and simulate power under plausible ranges; if more time, pre-specify secondary metrics, plan heterogeneity analysis, and draft the `SQL` checks and dashboards.

A second angle

Consider designing sample size when the metric is continuous revenue-per-user and results arrive with long-tailed distributions. Here you'd estimate `σ` either by log-transforming revenue or by Winsorizing extremes; compute `n` on transformed scale and translate back to business units. If randomization is at the store level or experiences interference (customers see multiple treatments), incorporate cluster `DE` and consider hierarchical models for analysis. When multiple treatments or dose levels are tested simultaneously, plan for multiplicity control (FDR or hierarchical testing) and recognize that correcting for many comparisons will meaningfully increase required overall sample or reduce per-comparison power.

Common pitfalls

Pitfall: Reporting a sample size without tying the MDE to business value. Interviewers want you to justify why the effect is worth detecting, not just show algebra.

Pitfall: Using historical average as baseline without checking recency or seasonality — a stale baseline underestimates variance and leads to underpowered tests.

Pitfall: Ignoring clustering or unit-of-analysis mismatch (randomizing users but analyzing sessions) — this inflates Type I error; always align randomization unit with analysis unit or adjust standard errors.

Connections

Interviewers often pivot to metric design (choosing guardrail and secondary metrics) and causal inference topics like SUTVA violations or interference. They may also ask about experiment analysis pipelines (pre-registration, dashboards, SRM checks) or advanced sequential/Bayesian testing strategies.

Further reading

• Practical Guide to Controlled Experiments on the Web — Kohavi et al. — concise industry practices and pitfalls for online A/B testing.

• CUPED: Controlled Experiments with Pre-Experiment Data — Deng et al. — variance reduction technique that often cuts required sample size.

General Prep — 24 min

Machine Learning

Focus area — Highly Walmart-specific and ML-heavy; review seasonality, promotions, hierarchy, stockouts, and forecast error trade-offs.

What's being tested

Interviewers are probing your ability to turn retailer signals into reliable, operational demand forecasts: problem framing, appropriate model choice, evaluation with business-aligned metrics, and communicating uncertainty and risk to downstream inventory and buying teams. Expect questions that test time-series reasoning (seasonality, promotions, stockouts), evaluation design (rolling-origin backtests, forecast horizons), and principled tradeoffs between accuracy, complexity, and maintainability.

Core knowledge

• Forecast horizon & cadence: Match horizon (e.g., 1–4 weeks vs 13 weeks) to supply-chain lead times; daily forecasts often require more noise-handling than weekly aggregation which stabilizes variance.

• Censoring / stockout bias: Sales data often show observed sales = min(true demand, inventory). Treat as censored demand; consider imputation, censorship models, or auxiliary `on_hand` signals rather than naively training on observed sales.

• Intermittent demand: For many `SKU`-store series with many zeros, use Croston or zero-inflated models, frequency/size decomposition, or global methods that pool information across items to avoid overfitting zeros.

• Model families & tradeoffs: Local ARIMA/ETS fit per series for explainability; global ML models (`LightGBM`, `XGBoost`, neural nets) scale better across millions of series and learn cross-series patterns, but need careful feature engineering and calibration.

• Feature engineering essentials: Lags, rolling means, price elasticity, promotion flags, day-of-week/holiday encodings, store-level seasonality, and inventory-level flags; lag windows typically 1–52 weeks and decay-weighted aggregates.

• Evaluation: rolling-origin & business metrics: Use rolling-origin cross-validation (walk-forward) and report multiple metrics: `MAE`/`RMSE` for scale, `MAPE` with caution (zeros), and RMSSE for comparability across items: $\text{RMSSE} = \sqrt{\frac{\frac{1}{h}\sum_{t=1}^h (y_t-\hat y_t)^2}{\frac{1}{T-1}\sum_{t=2}^T (y_t-y_{t-1})^2}}.$

• Probabilistic forecasts & loss functions: For inventory decisions, predict quantiles; train with pinball (quantile) loss: $L_\alpha(y,q)= (y-q)\cdot(\alpha - \mathbf{1}\{y<q\}).$ Also evaluate using `CRPS` when full distributions are available.

• Hierarchy & reconciliation: Forecasts must be coherent across `SKU`→store→region→category. Reconcile using bottom-up, top-down, or statistical methods like MinT (Minimum Trace) for optimal variance-adjusted reconciliation.

• Promotion & causal lift: Promotions change baseline demand and create displacement/cannibalization effects; for attribution, combine forecasting with uplift or causal methods (difference-in-differences, synthetic control) before embedding promotion-adjusted features.

• Cold-start / new-product strategies: Use attribute-driven forecasting (category/brand/price band), Bayesian hierarchical shrinkage, or analog-series matching. Always inflate uncertainty and consider experimentation for learning.

• Operational constraints & calibration: Penalize under-forecast vs over-forecast differently depending on stockout cost vs holding cost. Calibrate probabilistic forecasts using reliability diagrams and sharpness metrics before deployment.

• Scale & deployment considerations (DS lens): For millions of series, prefer global models with per-series embeddings, incremental retraining cadence aligned to data freshness, and robust backtesting that simulates production refresh and latency.

Worked example

Problem frame: "Forecast weekly demand for a `SKU` at store-level for the next 13 weeks given 3 years of POS sales, prices, promotions, holidays, and inventory snapshots." First 30s: clarify horizon, aggregation level (store vs region), whether inventory records contain stockouts, and business loss asymmetry (stockout vs overstock cost). Skeleton answer pillars: (1) Data preparation — flag stockout periods and impute censored demand; (2) Features — construct lags (1,2,4,13), rolling means, price elasticities, promotion encodings, holiday indicators; (3) Model — choose a global gradient-boosted tree (`LightGBM`) with quantile objectives to produce 10/50/90% forecasts; (4) Evaluation — rolling-origin backtest with RMSSE and pinball loss, and reconciliation if rolling up to region. Tradeoff to call out: per-`SKU` local ARIMA models are interpretable but won't share signal across sparse series, so I’d prefer a global model unless the business demands per-item explainability. Close: "If I had more time, I'd run a promotion-causal lift study to separate baseline vs promotional uplift and then retrain the forecast model on baseline-adjusted demand."

A second angle

Consider "forecasting demand for brand-new product launches." The same principles apply but with different emphasis: you cannot learn per-series lags, so rely on attribute-based features (category, price point, pack size), analogous-item time-series clusters, and Bayesian hierarchical models that borrow strength from category-level priors. Evaluation shifts from historic backtests to simulation and early-launch A/B tests; you must present wide prediction intervals and plan rapid learning (short retraining cycles) once early sales arrive. The interviewer will judge whether you can adapt pooling, uncertainty, and experiment-driven learning to a cold-start setting.

Common pitfalls

Pitfall: Using `MAPE` blindly for series with zeros or very low volume.

MAPE explodes on zeros and biases against low-volume items. Use `MAE`, `RMSE`, or RMSSE for comparisons, and report quantile coverage for uncertainty.

Pitfall: Ignoring censoring caused by stockouts.

Training on observed sales without correcting for stockouts underestimates true demand. Flag periods with zero sales and no inventory, impute or model censoring explicitly, and leverage `on_hand` signals where available.

Pitfall: Presenting point forecasts without business context.

Giving a single point estimate without quantifying downside risk (e.g., 90th percentile for safety stock) or stating cost asymmetry (stockout vs holding) loses credibility. Always link forecast outputs to a downstream decision metric.

Connections

Related pivots include inventory optimization (safety stock and reorder points derived from forecast uncertainty), price-elasticity modeling (integrate demand response into forecasts), and causal inference for promotion lift (if asked why promotions spike baseline demand).

Further reading

• Forecasting: Principles and Practice (Hyndman & Athanasopoulos) — practical time-series methods and evaluation guidance.

• M5 Forecasting Competition (Kaggle) — real-world retail forecasting patterns, hierarchical reconciliation, and probabilistic scoring techniques.

Focus area — Walmart Labs often values product discovery, search, and personalization; your ML refresh should include ranking basics.

What's being tested

Interviewers probe your ability to translate business goals into rigorous evaluation and experimentation for personalization and recommendation systems. They'll assess statistical reasoning (metric choice, bias correction, power), model-evaluation tradeoffs (offline proxies vs online impact), and how you diagnose metric regressions or heterogeneous effects across cohorts. Walmart Labs cares because recommender changes can shift revenue, conversion funnels, and long-term retention; the DS must measure and attribute those effects reliably.

Core knowledge

• candidate generation and ranking: two-stage pipeline; optimize recall at generation, then precision/utility at ranking. Rankers often use pointwise (regression), pairwise (RankNet), or listwise (LambdaMART) losses depending on goal.

• precision@k, recall@k, NDCG: use precision@k for top-k utility, recall@k for coverage, and NDCG where position discount matters. $\text{DCG}_k=\sum_{i=1}^k\frac{2^{rel_i}-1}{\log_2(i+1)},\ \text{NDCG}=\frac{\text{DCG}}{\text{IDCG}}.$

• Offline vs online alignment: offline metrics can be poor proxies for CTR or revenue due to exposure and position bias; measure offline-to-online correlation historically before trusting proxies.

• Logged-feedback (counterfactual) evaluation: use Inverse Propensity Scoring (IPS) and Doubly Robust estimators to evaluate new policies from logged data. IPS estimator: $\hat V_{IPS}=\frac{1}{N}\sum_{i=1}^N \frac{\mathbb{I}\{a_i=\pi(x_i)\}r_i}{p(a_i|x_i)}.$ Clip propensities to reduce variance.

• Explore–exploit and randomized exposure: randomized buckets or controlled exploration (epsilon-greedy, Thompson Sampling) are needed to collect unbiased learning signals; quantify business cost of exploration.

• A/B testing specifics for recommenders: randomize at user level; metric windows must consider delayed conversions; guardrail metrics (DAU, Conversion Rate, Average Order Value) and contamination checks (leakage, cross-user effects).

• Sequential testing and sample size: plan for minimum detectable effect (MDE), pre-specify primary metric and power (typ. 80%), use alpha-spending or corrected p-values for repeated looks (alpha-spending / Benjamini-Hochberg for FDR).

• Heterogeneous effects & segmentation: analyze treatment effects across cohorts (new vs returning, category affinity) using uplift models or stratified ATE; beware multiple comparisons.

• Position and selection bias remedies: log exposures and impressions; use randomized swaps, position-based models (PBM), or click models to de-bias observed clicks before training/ranking.

• Business-aware utility: incorporate downstream metrics (revenue, returns, margin) into evaluation. Optimize for long-term retention where applicable (multi-period metrics) not just immediate CTR.

• Interleaving & online ranking tests: for fast pairwise comparisons, use online interleaving; it measures preference with fewer samples but needs careful user-level aggregation.

• Failure modes & logging: always check impression logs, user_id consistency, and sample size per bucket; missing exposure data breaks causal estimates.

Worked example — "Design an offline evaluation metric for a recommender"

First 30s: ask clarifying questions — what's the business objective (clicks, revenue, retention), what data is available (impressions + clicks vs only transactions), and the acceptable offline/online proxy tradeoff. Skeleton answer pillars: 1) choose a primary metric aligned to business (e.g., NDCG@k if engagement with top results matters, or revenue-weighted precision@k if monetization is key), 2) correct for exposure bias using logged propensities or randomized impression data, 3) validate offline-to-online correlation with historical A/B pairs and compute calibration. A key tradeoff: variance of IPS estimators vs bias of naive metrics — you may prefer a mildly biased but low-variance proxy for rapid iteration, then use IPS/Doubly Robust for final evaluation. Close by proposing an online sanity check: small-scale randomized holdout or interleaving and a plan to monitor guardrails (CTR, DAU) and long-term retention.

A second angle — "Analyze an A/B test where recommendations changed DAU"

Framing changes: now the outcome is a product-level metric (DAU), not item-level clicks. Focus pillars: 1) ensure randomization at user-level and sufficient duration to capture habitual behavior changes; 2) attribute DAU changes to personalization by checking intermediate metrics (impressions, click-through, conversion funnel); 3) check for novelty/novel-adoption effects (short-term lift vs long-term decay). Special constraints: interference (you may influence other users via shared content) and user churn bias — use survival analysis for retention and define treatment exposure (how many recommendation impressions constitute "treated"). A useful tradeoff: shorter test gives speed but risks confusing short-term novelty with sustained retention; extend analysis with cohort-level longitudinal ATE.

Common pitfalls

Pitfall: Relying on naive click-rate comparisons from observational logs.
Using raw clicks without exposure or propensity correction confounds position and selection biases; reported lifts will be optimistic and non-causal.

Pitfall: Ignoring multiple comparisons when slicing by cohorts.
Running many subgroup tests inflates false positives; pre-register primary cohorts or use FDR control instead of reporting every significant slice.

Pitfall: Optimizing a proxy misaligned with business.
Maximizing precision@k can hurt long-term Average Order Value if recommendations push low-margin, high-click items — always include business-weighted metrics or offline simulators for downstream effects.

Connections

Interviewers may pivot to offline feature importance & explainability (how item/user features drive recommendations), bandit algorithms & reinforcement learning for online personalization, or causal inference topics (instrumental variables, difference-in-differences) when long-term effects and confounding are raised.

Further reading

• Unbiased Learning to Rank (Joachims et al.) — foundational on correcting position bias with propensity scoring.

• Doubly Robust Off-policy Evaluation (Schnabel et al.) — practical methods for low-variance counterfactual evaluation.

Statistics & Math

Focus area — This is both Walmart-specific and statistics-heavy: elasticity, lift, cannibalization, and margin trade-offs need clear explanations.

What's being tested

Interviewers probe your ability to translate sales and pricing data into actionable causal estimates and evaluations that drive merchandising and promotional decisions. They want to see competency in causal inference (separating price/promotional effects from confounders), appropriate metric design (revenue, margin, units, lift), and pragmatic model choices (panel regressions, hierarchical pooling, experiments vs. observational IVs). Walmart cares because small errors in elasticity or promotion-lift estimates scale to large revenue/margin impacts across millions of SKUs and stores.

Core knowledge

• Price elasticity definition and estimation: elasticity ε = %ΔQuantity / %ΔPrice; in log-log models ln(Q)=α+β ln(P)+Xγ, β is point elasticity. Understand interpretation, sign, and nonlinearity across ranges.

• Endogeneity of price: prices are often set in response to demand shocks (simultaneity). Naive OLS will be biased; plan for instrumentation or experiments to obtain causal effects.

• Instrumental variables (IV): valid instruments affect price but not demand directly (e.g., cost shocks, supplier promotions, competitor price changes). Check relevance (F-stat > 10) and exclusion restriction plausibility.

• Panel fixed-effects: use store-SKU-time fixed effects to absorb unobserved heterogeneity; difference-in-differences (DiD) for policy/promo rollouts. Beware serial correlation—use clustered standard errors at the store or SKU level.

• Promotions & cannibalization: model promotional flags, depth, and cross-price effects; cross-price elasticity matrix captures substitution between SKUs. Track cannibalization to avoid overestimating incremental units.

• Censoring & zero sales: use Tobit, zero-inflated, or Poisson pseudo-maximum-likelihood (PPML) estimators when many zeroes or censoring from stockouts exist; log models require handling zeros (e.g., ln(Q+1) or PPML).

• Hierarchical / Bayesian pooling: partial pooling across SKUs or stores stabilizes noisy elasticity estimates; hierarchical models borrow strength while allowing heterogeneity. Use for portfolios of thousands of SKUs.

• Experiment design for pricing: prefer randomized price or promo experiments when feasible; cluster-randomize by store/region to avoid interference, pre-stratify on baseline demand, and power for ratio metrics (revenue per store-day).

• Time-series causal methods: for single-unit interventions, use synthetic controls or Bayesian structural time-series (e.g., CausalImpact) to estimate counterfactual demand after a price change.

• Uplift modeling and targeting: build models predicting incremental response to a promotion (heterogeneous treatment effects) using causal trees/forests (e.g., uplift random forests); evaluate on holdout promotion groups, not standard predictive metrics.

• Evaluation metrics & business tradeoffs: report incremental revenue, incremental margin, units, and LTV; tie elasticity to profit optimization by solving max_{price} E[profit] = (P-C) * E[Q(P)].

• Multiple testing & sequential testing: correct for multiple SKUs/tests using Benjamini-Hochberg FDR or family-wise corrections; for sequential decisions use pre-specified stopping rules or sequential methods (alpha spending).

Worked example — "Estimate price elasticity for a product using observational sales & price history"

First 30s: ask clarifying questions—target elasticity granularity (SKU-store vs SKU-chain), horizon (short-term campaign vs long-term baseline), presence of promotions/stockouts, and available covariates (competitor prices, marketing, cost). Skeleton answer pillars: (1) Data prep: build a panel with daily store-SKU sales, price, promo flags, competitor price, holidays, and stockout indicators. (2) Identification strategy: start with a log-log fixed-effects panel ln(Q_{it}) = α_i + δ_t + β ln(P_{it}) + X_{it}γ, then test for endogeneity using Durbin-Wu-Hausman; if endogenous, propose instruments (supplier cost shock, regional wholesale price changes). (3) Estimation & validation: estimate IV-first-stage strength (F>10), IV second stage for β, cluster s.e. at store or region, and run placebo periods and pre-trend checks. Tradeoff to flag: IV reduces bias but increases variance — wide CIs matter for pricing decisions. Close with next steps: if more time, run a small randomized price experiment for high-variance SKUs or fit a hierarchical Bayesian model to pool across similar SKUs and produce shrinkage estimates with uncertainty.

A second angle — "Design an experiment to test a promotion across stores"

Here the framing shifts to randomized design and interference. You’d propose cluster randomization by store (or zip code) with stratification on baseline sales and store format. Define treatment unit (store-day) and guard band to limit spillover between nearby stores. Choose primary metric as incremental gross margin per store-week, compute power for store-level variance, and pre-register analysis window. Address interference/cannibalization by including geographic spill variables and measuring cross-store uplift. If stores are limited, consider a stepped-wedge rollout to gain power while eventually exposing all stores.

Common pitfalls

Pitfall: Ignoring price endogeneity and treating observed price drops as exogenous promotions.
This yields biased elasticity estimates; always test for endogeneity and prefer IVs or randomized experiments when possible.

Pitfall: Reporting point estimates without uncertainty or business interpretation.
Present elasticities with CIs, translate to expected revenue/margin impact for plausible price changes, and provide decision thresholds (e.g., change only if expected profit gain > operational cost).

Pitfall: Overfitting short promotional windows and extrapolating to long-run elasticities.
Short-term promotional lift often differs from long-run sensitivity; explicitly model time-varying elasticity or obtain long-horizon experiments before permanent price decisions.

Connections

Interviewers may pivot to assortment optimization (how price interacts with product assortment), demand forecasting (feeding elasticity into forecasts), or inventory optimization (price as a lever for stock-clearing). Be prepared to connect causal estimates to downstream optimization problems.

Further reading

• Wooldridge, Jeffrey M. "Econometric Analysis of Cross Section and Panel Data" — rigorous treatment of panel methods, IV, and fixed effects.

• Brodersen et al., "Inferring causal impact using Bayesian structural time-series" — practical method for single-unit intervention analysis (CausalImpact).