Google Fairness, Privacy, And Risk Governance

What's being tested

Interviewers are probing your ability to measure, reason about, and mitigate harms that arise from models and experiments while preserving statistical rigor. Expect questions that combine fairness metrics, privacy-utility tradeoffs, and risk monitoring with practical experimental and cohort-analysis thinking. Google cares because data scientists must quantify harms, propose defensible mitigations, and prioritize tradeoffs under uncertainty.

Core knowledge

Statistical parity / demographic parity — measure: $\Delta_{SP}=P(\hat{Y}=1|A=a)-P(\hat{Y}=1|A=b)$ ; also use the disparate impact ratio = ratio of those probabilities; both reveal allocation differences irrespective of true label.
Equalized odds & calibration — equalized odds requires equal TPR/FPR across groups; calibration requires $P(Y=1|score=s,A=a)\approx s$ for all groups; these goals can conflict and require tradeoffs.
Error-rate decomposition — report TPR, FPR, precision and overall accuracy by group; quantify tradeoffs via ROC/AUC stratified by group and by threshold.
Multiple comparisons & subgroup testing — use Benjamini–Hochberg FDR or Holm–Bonferroni for many subgroup tests; uncorrected p-values inflate false positives when searching for bias across many slices.
Sample size & power for proportions — for detecting a difference d between group positive rates: $n\approx \frac{(z_{1-\alpha/2}\sqrt{2\bar p(1-\bar p)}+z_{1-\beta}\sqrt{p_1(1-p_1)+p_2(1-p_2)})^2}{d^2}$ ; ensure adequate subgroup counts or report wide CIs.
Causal vs observational fairness — observational disparity doesn't always imply causal harm; use causal graphs or CATE/ITE estimates to assess whether a protected attribute causally affects decisions/outcomes.
Bias mitigation families — pre-processing (reweighting, IPW), in-processing (fairness-constrained objectives, Lagrangian), post-processing (threshold adjustments per group); evaluate utility loss and legal/operational constraints for each.
Differential privacy basics — epsilon (ε) measures privacy loss; sensitivity determines noise scale (Laplace/Gaussian). Translate privacy budgets into expected utility loss on your metric; treat ε as a negotiable axis.
Monitoring & risk governance metrics — define safety KPIs: subgroup uplift/regression, false positive surges, calibration drift; set alert thresholds (absolute and relative) and sample-minimum rules to avoid noisy triggers.
Heterogeneous treatment effects — use CATE estimation (e.g., causal forests) to detect harms concentrated in subgroups; then quantify expected utility loss and prevalence to prioritize fixes.
Intersections & small groups — handle intersectional slices carefully: aggregate small groups with hierarchical models or Bayesian shrinkage to avoid noisy estimates while still surfacing harms.
Actionability & stakeholder framing — for each detected disparity, quantify its operational impact (number affected, downstream cost/benefit) and propose concrete mitigations plus rollback criteria.

Worked example — Assessing fairness for a binary classifier across demographics

Frame first: ask which protected attribute(s) exist, whether labels are ground truth or proxy, and what operational cost of false positives/negatives is. Organize the answer into three pillars: measurement (choose metrics and ensure subgroup sample size), diagnosis (is disparity due to features, labels, or selection bias?), and remediation (select pre/in/post-processing and estimate utility tradeoff). Compute group-level TPR, FPR, statistical parity difference, and group-wise calibration curves; report confidence intervals and FDR-corrected p-values for multiple groups. A concrete tradeoff to flag: improving parity by equalizing thresholds may increase overall false positives, so quantify number of additional mistakes and business cost. Close by proposing monitoring (daily subgroup metrics with minimum-sample gating) and additional causal analysis: "if I had more time, I'd run targeted randomized experiments or use CATE estimators to separate label bias from model bias."

A second angle — Experiment shows overall lift but subgroup harm

Now suppose an A/B test increases overall engagement but reduces it for a particular demographic. Start by checking randomization balance across the protected attribute and pre-period trends; verify subgroup sample sizes and compute CIs. Frame response as: (1) Is the observed subgroup effect statistically and practically meaningful after multiplicity correction? (2) Is the effect transient (interaction with device/locale) or consistent across cohorts? (3) Propose mitigations: local rollback, targeted experiment variants, or feature changes, prioritizing groups by prevalence and harm severity. Emphasize causal checks (stratified ATE, interaction terms in regression), and propose uplift-modeling to identify which users benefit vs. are harmed.

Common pitfalls

Pitfall: Confusing observational disparity with causation — reporting a difference in outcomes and immediately ascribing it to the model without checking for label bias or confounding leads to wrong fixes. Always ask how labels were generated and test for selection effects.

Pitfall: Overcorrecting with noisy slices — adjusting model thresholds for tiny intersectional groups based on high-variance estimates can degrade overall utility; use hierarchical pooling or require minimum sample sizes and report uncertainty.

Pitfall: Ignoring multiple testing — searching across many slices and metrics without correction produces false positives; use FDR control and pre-register analyses where possible.

Connections

Interviewers may pivot to A/B testing design for heterogeneous effects, causal inference (instrumental variables, mediation), or model monitoring/ML observability (drift detection, alerts). Be prepared to tie fairness findings to operational metrics and rollback criteria.