Statistical Inference For Proportions And Confidence Intervals

What's being tested

These prompts test a candidate's ability to do principled statistical inference for proportions: set up and run binomial-based hypothesis tests, compute confidence intervals, and reason about reach/expectation under random allocation. Interviewers check that you can choose appropriate approximations (normal/Poisson/exact), compute/interpret p-values, report confidence intervals, and assess practical significance and power for product decisions. At Meta, a Data Scientist must translate small-sample statistical answers into product-facing statements about lift, risk, and whether an experiment warrants rollout.

Core knowledge

• Binomial model: individual trials ~ Bernoulli(p); counts follow $X\sim\mathrm{Binomial}(n,p)$. Use when trials are independent and identical with fixed n and p. Edge case: dependence (e.g., repeated users) breaks this.

• Normal approximation / CLT: for large n, $\hat p\approx N(p,\;p(1-p)/n)$. Use when $np, n(1-p)\gtrsim 5$–10; otherwise use exact or adjusted intervals.

• Two-proportion z-test: test $H_0:p_1=p_2$ with pooled estimate $\hat p = (x_1+x_2)/(n_1+n_2)$; test statistic $z=\frac{\hat p_1-\hat p_2}{\sqrt{\hat p(1-\hat p)(1/n_1+1/n_2)}}$ and two-sided p-value $2(1-\Phi(|z|))$.

• Confidence intervals: common choices are Wald (naïve), Wilson (recommended), and Agresti–Coull; Wilson has better coverage for moderate/low n or p near 0/1. Wilson interval: $\frac{\hat p+z^2/(2n)\pm z\sqrt{\hat p(1-\hat p)/n+z^2/(4n^2)}}{1+z^2/n}.$

• Exact binomial test: compute tail probability from Binomial CDF when normal approx is invalid; use for small n or extreme p (e.g., conversions).

• Poisson approximation: when n large and p small ($\lambda=np$ modest), Binomial$(n,p)\approx\mathrm{Poisson}(\lambda)$; useful for expected impressions/reach calculations.

• Power and sample size: for target absolute lift $\Delta$, approximate required n per arm: $n\approx\frac{(z_{1-\alpha/2}+z_{1-\beta})^2(p_1(1-p_1)+p_2(1-p_2))}{\Delta^2}.$ Always express power for practical-effect sizes.

• Multiple comparisons & sequential testing: control family-wise error via Bonferroni or use FDR (Benjamini–Hochberg); for monitoring use alpha spending or sequential methods (e.g., O’Brien–Fleming) to avoid inflated Type I.

• Bayesian beta-binomial: conjugate prior Beta(a,b) gives posterior Beta(a+x,b+n-x). Use to compute credible intervals, posterior probability of uplift > 0, or small-sample regularization.

• Practical vs statistical significance: report absolute and relative lift, baseline rate, and expected incremental events; quantify uncertainty in both statistical and product-relevant units (e.g., incremental conversions per million impressions).

• Random allocation exposure modeling: expected impressions per user = $m\cdot r$ if $m$ total impressions and random over R users, or treat each user's impressions as Binomial/Poisson depending on replacement assumptions; probability of at least one exposure is $1-(1-p)^n$.

• Reporting standards: always state assumptions (independence, randomization, unit of analysis), show raw counts and intervals, and perform sanity checks (e.g., expected vs observed reach).

Worked example — Calculate Expected Impressions and Probability for Users

Start by clarifying assumptions: are impressions placed uniformly at random with replacement over users (independent draws) or without replacement? Define unit of analysis (user-level). Skeleton: (1) model impressions per user as Binomial(n=m, p=1/R) or Poisson($\lambda=m/R$) for large R, (2) compute expectation $E[\text{impressions}]=m/R$ and probability of at least one $1-(1-1/R)^m\approx1-e^{-m/R}$, (3) compute expected unique reach = $R\cdot(1-(1-1/R)^m)\approx R(1-e^{-m/R})$. Flag tradeoffs: if impressions are correlated by time or user caps, independence fails and Poisson overestimates variance. Close by stating what product decisions follow (e.g., estimated reach vs frequency caps) and what extra data you'd request (impression timestamps, dedup keys) if available.

A second angle — Determine Superiority of Model A Using Hypothesis Testing

Here the framing is a two-proportion comparison of model outcomes (click/convert rates). You'd check sample sizes to justify the normal approximation, compute pooled vs unpooled variance depending on H0 formulation, and report both p-value and a 95% confidence interval for the difference to show magnitude. Important tradeoff: statistical significance may be tiny absolute lift with negligible business value; always convert to expected incremental conversions per million impressions. If sequential model comparisons occur (many models), recommend multiplicity correction or Bayesian model comparison to avoid false discoveries.

Common pitfalls

Pitfall: treating tiny p-values as automatically important.
Confusing statistical significance with product impact is common. Always convert the uplift and CI into absolute events and expected business metric changes before recommending rollout.

Pitfall: blind use of the Wald interval or pooled z-test for small counts.
For small n or rare events (CTR << 1%), the normal approximation under-covers; use Wilson/Agresti–Coull or exact tests instead.

Pitfall: ignoring dependence and the correct unit of analysis.
Aggregating multiple impressions per user without accounting for intra-user correlation underestimates variance; use user-level metrics or clustered standard errors and state this assumption to the interviewer.

Connections

This topic often segues to sequential A/B testing, Bayesian decision rules (expected loss vs Type I/II tradeoffs), or metric design (defining conversion events and aggregation at the correct unit). Be prepared to move from inference to product impact and experiment monitoring.

Further reading

• Agresti, A. & Coull, B. (1998). "Approximate is Better than 'Exact' for Interval Estimation of Binomial Proportions" — explains Wilson/Agresti–Coull intervals and their superior coverage.

• Casella, G. & Berger, R. L. (2002). "Statistical Inference" — comprehensive theory for hypothesis testing, intervals, and decision-theoretic framing.