Privacy-Safe Measurement And Weighted Estimation At Google

What's being tested

This topic tests whether you can measure behavior when user-level data is incomplete, sampled, aggregated, or intentionally protected by privacy controls. You should connect privacy guarantees to estimand definition, sampling bias, weighting, uncertainty, and decision quality rather than treating privacy as a purely engineering constraint. Google cares because products and ads require reliable measurement across large populations while limiting exposure of sensitive individual information. The interviewer is probing whether you can explain what remains identifiable, quantify bias and variance, and recommend a method whose privacy–utility tradeoff matches the decision.

Core knowledge

• An estimand is the precise quantity being measured: for example, the population conversion rate, incremental conversions caused by an ad, or average treatment effect. Define the target population, time window, treatment, outcome, and unit before choosing a privacy-safe estimator.

• Inverse probability weighting (IPW) corrects unequal observation or treatment probabilities. If unit $i$ is observed with probability $\pi_i$, the Horvitz–Thompson estimator is $\hat{T}_{HT}=\sum_i R_i y_i/\pi_i$; unstable small $\pi_i$ values require trimming, stabilized weights, or a redesign.

• A Hájek estimator normalizes weighted totals: $\hat{\mu}_{H}=\frac{\sum_i R_iw_i y_i}{\sum_i R_iw_i}$. It is often less variable than Horvitz–Thompson for means, though normalization can introduce small finite-sample bias and must be included in inference.

• Post-stratification partitions users into cells such as geography, device, age band, or activity level, then weights each cell to known population totals. It reduces selection bias when outcomes are conditionally homogeneous, but sparse or sensitive cells may need merging or suppression.

• Calibration weighting chooses weights satisfying constraints such as $\sum_i w_i x_i = X_{\text{population}}$. Raking is practical for several margins, while entropy balancing or generalized regression can match richer moments; monitor extreme weights and nonconvergence.

• Effective sample size (ESS) reveals variance inflation from weights: $ESS=(\sum_i w_i)^2/\sum_i w_i^2$. A nominal sample of millions can have an ESS of only thousands when a few users dominate, so report both weighted point estimates and ESS.

• Differential privacy (DP) bounds the effect of any one person on released statistics. A mechanism satisfies $(\varepsilon,\delta)$-DP when neighboring datasets have output probabilities differing by at most $e^\varepsilon$, plus $\delta$ failure probability; smaller $\varepsilon$ generally means stronger privacy and more noise.

• For a count with sensitivity $1$, the Laplace mechanism adds noise proportional to $1/\varepsilon$; for bounded sums, sensitivity depends on the outcome range. Clipping contributions before noise addition is essential, but clipping creates bias that should be measured on realistic data.

• Composition consumes a privacy budget across repeated releases. Basic composition adds $\varepsilon$ values, while advanced composition or a privacy accountant can provide tighter bounds; repeated dashboards, slices, and experiments can consume substantially more budget than one headline metric.

• Thresholding and aggregation reduce disclosure risk but are not equivalent to DP. A minimum cohort size, suppression rule, or k-anonymity policy can still leak information through differencing or auxiliary data; assess neighboring releases and correlated metrics.

• Randomized response, local DP, and secure aggregation differ in where protection occurs. Local DP protects before collection but usually has worse utility; central DP gives stronger aggregate accuracy when a trusted curator can add noise; secure aggregation hides individual reports while enabling aggregate computation.

• Privacy-safe causal measurement needs both protected outcomes and valid counterfactuals. In an experiment, estimate an intention-to-treat effect first, preserve randomization, and apply privacy noise or weighting to treatment and control consistently; otherwise, differential noise can create artificial lift.

• For weighted estimates, uncertainty must include sampling and weighting variance, plus privacy noise. A useful approximation is $\operatorname{Var}(\hat{\theta}_{released})\approx\operatorname{Var}(\hat{\theta}_{sampling})+\operatorname{Var}(\eta_{privacy})$, with covariance terms considered when noise or weights are data-dependent.

• Small-cell risk is both a privacy and statistical problem. Suppressing a tiny cohort protects users but can change the estimand; replacing it with a broader cohort, hierarchical shrinkage, or a predeclared “not reported” category is preferable to silently dropping it.

• Validate with sensitivity analysis: compare unweighted, weighted, trimmed-weight, and privacy-noised results; vary clipping thresholds and privacy budgets; and report whether the product decision changes. Privacy protection should be evaluated alongside bias, interval width, subgroup coverage, and reproducibility.

Worked example

Representative question: “How would you estimate campaign conversions when user-level identifiers cannot be shared?”
I would first clarify the estimand—incremental conversions in the target population over a defined window—and ask whether randomized treatment assignment, aggregate cell counts, and population calibration totals are available. I would organize the answer around four pillars: preserve the experimental contrast, correct any sampling or exposure imbalance, apply a formally specified privacy mechanism, and quantify uncertainty and decision sensitivity. If treatment and control aggregates are available by non-sensitive strata, I would compute a weighted difference in means or rates, using stabilized weights and merging sparse cells before release. I would explicitly flag the tradeoff between finer segmentation, which improves bias correction, and larger privacy loss and variance from many released cells. I would add calibrated DP noise to bounded aggregates, track the total privacy budget across repeated queries, and report ESS, confidence intervals, and noise-induced uncertainty. I would close by saying that, with more time, I would run placebo and pre-period balance checks, compare several clipping thresholds, and test whether the launch decision is robust to plausible privacy budgets.

A second angle

Representative question: “How would you report a product metric for a rare user segment without exposing individuals?”
Here the primary challenge is not causal attribution but stable population estimation under small-cell risk. I would define whether the goal is a descriptive rate or a population-representative rate, then consider post-stratification, cell merging, shrinkage, suppression, and DP noise. A broad, predeclared segment may be more useful than a highly granular metric whose ESS is low and whose privacy noise dominates. I would communicate an interval or “insufficient evidence” result rather than presenting a noisy point estimate as precise.

Common pitfalls

Pitfall: Treating a weighted average as automatically unbiased. Weights correct only the selection mechanism they model; unmeasured differences, positivity violations, and extreme weights can leave substantial bias.

Pitfall: Saying “aggregate data is private.” Aggregates can leak through differencing, repeated queries, rare cohorts, or auxiliary information; explain the threat model and use DP, secure aggregation, suppression, or a combination appropriate to it.

Pitfall: Reporting only the privacy parameter. A strong answer also discusses clipping bias, sampling variance, ESS, composition across releases, subgroup utility, and whether the resulting uncertainty changes the product decision.

Connections

The interviewer may pivot to causal inference, randomized experiments, missing-data mechanisms, survey sampling, or differentially private hypothesis testing. Be ready to distinguish measurement error from selection bias and privacy noise from ordinary sampling noise.

Further reading

• “Calibrating Noise to Variance in Privacy-Preserving Data Analysis” — practical treatment of privacy noise and statistical accuracy.

• “The Algorithmic Foundations of Differential Privacy” — foundational definitions, mechanisms, and composition.

• “Survey Sampling” by Leslie Kish — classic grounding in weighting, stratification, and variance.