Core Probability And Statistical Inference Refresher

What's being tested

Interviewers are probing your ability to reason about uncertainty: how to turn noisy samples into reliable decisions using probability and statistical inference. Expect to explain and compute hypothesis tests, confidence intervals, sample-size/power calculations, and to justify assumptions (randomization, independence, measurement). Walmart Labs cares because product and pricing choices depend on correctly measured effects (statistical and practical) and safe stopping/segmentation rules.

Core knowledge

Central Limit Theorem (CLT): averages of i.i.d. draws converge to Normal for moderate n (rule of thumb n≥30), enabling z-tests; heavy tails or dependence break CLT assumptions.
Standard error (SE) for a mean: $SE=\sigma/\sqrt{n}$ ; for difference of proportions $SE=\sqrt{\frac{p_1(1-p_1)}{n_1}+\frac{p_2(1-p_2)}{n_2}}$ ; use these to form z/t statistics.
t-test vs z-test: use t-test when population σ is unknown and sample sizes are small; use Welch's t-test when variances are unequal (heteroscedasticity).
Hypothesis testing vocabulary: Type I error (α), Type II error (β), statistical power = 1−β; p-value = probability of data (or more extreme) under null — not the probability the null is true.
Sample-size / MDE formula (two-sample means): $n\approx\frac{(z_{1-\alpha/2}+z_{1-\beta})^2( \sigma_1^2+\sigma_2^2)}{\delta^2}$ where δ is minimum detectable effect (MDE); for proportions substitute Bernoulli variances.
Multiple testing: Bonferroni controls family-wise error (α/m) — conservative; Benjamini-Hochberg controls false discovery rate (FDR) — less conservative for many metrics.
Sequential monitoring and peeking: unadjusted repeated looks inflate Type I error; use alpha-spending (Pocock/O'Brien–Fleming) or sequential methods (e.g., always-valid p-values) or pre-specified group sequential plan.
Nonparametric and resampling: use permutation tests for exchangeable data and bootstrap for CI when analytic SE is unreliable (small n, heavy tails, complex estimators).
Rare events and rates: for low counts prefer exact binomial tests or Poisson/negative-binomial models; normal approximation fails when counts < ~10.
Causal assumptions for observational inference: randomization, SUTVA (no interference), and ignorability/conditional exchangeability for regression/propensity adjustments.
Effect size vs p-value: always report confidence interval and absolute/relative effect; small p with trivial effect is not actionable.
Tip: wrap library names when you plan to run tests in code: use scipy.stats, statsmodels, or pingouin for standard routines.

Worked example — A/B test for conversion rate

Frame the problem: ask what the primary metric is (binary conversion?), unit of randomization (user, session), baseline conversion rate p0, desired MDE, α and power. First pillars: (1) pre-specify hypothesis H0: pA=pB vs H1: pA≠pB, (2) compute sample size using the proportions SE formula and MDE, (3) ensure randomization and instrumentation checks, (4) run test and compute z-statistic and two-sided CI, (5) report effect size and business interpretation. Explicit tradeoff: choosing a smaller α or detecting a smaller MDE increases required n (longer experiment), so negotiate business cost of time vs Type I risk. For stopping, either pre-specify a fixed-horizon analysis or adopt an alpha-spending plan; do not peek without adjustment. Close by saying: if more time, I'd add subgroup heterogeneity pre-specification, run permutation tests for robustness, and compute Bayesian posterior for intuitive credible intervals.

A second angle — regression adjustment in observational A/B-like comparison

If assignment isn't randomized, frame identification: state required conditional ignorability and list covariates to control (time, geography, prior behavior). Pillars: (1) model outcome with covariate adjustment or inverse-propensity weighting, (2) compute robust (sandwich) standard errors, (3) check overlap/balance and do sensitivity analysis for unobserved confounding. Tradeoffs: regression improves precision if model is correct, but model misspecification can bias estimates; propensity-score trimming mitigates extreme weights. This applies the same inference machinery (SE, CI, hypothesis tests) but emphasizes identification rather than randomization.

Common pitfalls

Pitfall: Interpreting a small p-value as the probability the null is false. This is wrong; instead report the effect estimate and confidence interval for practical relevance.

Pitfall: Peeking at results and stopping early without correction. Unadjusted repeated looks raise false positives; either fix sample size or use sequential corrections.

Pitfall: Reporting relative lifts only. A 10% lift on a 0.1% baseline is often meaningless; always show absolute differences and translate to business impact.

Connections

Interviewers can pivot from inference to causal inference (instrumental variables, difference-in-differences), uplift modelling (heterogeneous treatment effects), or to Bayesian A/B testing and always-valid inference techniques — be prepared to justify frequentist vs Bayesian choices.