Responsible AI, Fairness, Privacy, And Safety In Google ML

What's being tested

Interviewers probe whether you can operationalize responsible ML at the model-training and serving layer: measure and mitigate fairness gaps, apply privacy-preserving training, and harden models for basic safety/robustness — while balancing model utility, latency, and monitoring requirements. For a Machine Learning Engineer this means concrete systems-level tradeoffs: which algorithmic mitigations to integrate into training pipelines, how to instrument evaluation and continuous monitoring, and what guarantees those choices actually provide in deployment.

Core knowledge

Fairness metrics: know demographic parity $|P(\hat{Y}=1|A=0)-P(\hat{Y}=1|A=1)|$ , equalized odds (TPR/FPR gaps), and calibration per-group (reliability diagrams, Expected Calibration Error). Each implies different interventions.
Mitigation families: pre-processing (reweighing, resampling), in-processing (constrained optimization, adversarial debiasing), post-processing (threshold adjustment, calibration). Tradeoffs: utility loss, enforcement complexity, legal constraints.
Differential privacy (DP) basics: DP-SGD clips per-example gradients to norm $C$ , adds Gaussian noise with std $\sigma C$ , producing $(\varepsilon,\delta)$ privacy via accountant methods; higher $\sigma$ →better privacy, lower utility. Clip + noise formula: $\tilde{g}=\frac{1}{n}\left(\sum_{i}\text{clip}(g_i,C)\right)+\mathcal{N}(0,\sigma^2 C^2 I).$
Privacy budget & composition: account for epochs and batch sampling using a privacy accountant (Moments Accountant / RDP). Client-level DP (whole device) vs record-level DP have different costs in noise and utility.
Federated learning tradeoffs: reduces central raw data, but faces non-iid clients, intermittent connectivity, and amplified privacy cost per client. Combine with secure aggregation and DP for stronger guarantees.
Robustness & safety: basic defenses include adversarial training, input sanitization, and score-thresholding with human review for high-risk outputs. Certifiable robustness methods exist but are costly and often infeasible at scale.
Explainability for debugging: use feature importances, SHAP, Integrated Gradients to trace disparate impact sources, but be aware post-hoc explanations can be unstable across groups.
Monitoring & drift detection: instrument feature distributions with PSI, KL-divergence, MMD, and label-shift tests; monitor per-group metrics (TPR/FPR, calibration) and input-adversarial signatures. Set alerting thresholds and automated rollback triggers.
Evaluation rigor: stratify offline test sets by protected attributes and relevant slices; run counterfactual or synthetic perturbation tests; use uncertainty estimates (e.g., predictive entropy) when deploying fallback policies.
Operational constraints: privacy/robustness fixes often increase compute, inference latency, or model size; quantify cost per request and latency SLO impacts before selecting mitigation.
Documentation & artifacts: produce Model Card-style metadata (intended use, datasets, evaluation slices, known limitations) and log provenance for retraining/audits; these artifacts aid MLE handoffs without dictating product decisions.
Metrics-to-mitigation mapping: a high demographic parity gap suggests pre/post-processing; unequal TPRs often require in-processing constraints; calibration issues call for post-hoc calibration per-group.

Worked example — "Evaluate and mitigate bias in a loan-approval model"

Frame: ask what the sensitive attribute(s) are, the legal/operational constraints, the target population, acceptable fairness thresholds, and whether labels reflect historical bias. Skeleton: (1) measurement: compute selection rate, TPR/FPR, and calibration across groups plus confidence intervals; (2) diagnose: check label bias (proxy labels), sample balance, and feature correlation with the sensitive attribute; (3) mitigation plan: choose pre-, in-, or post-processing with explicit utility–fairness tradeoff quantification; (4) deployment: instrument per-group metrics and rollback. Explicit tradeoff: enforcing demographic parity may reduce calibration and overall accuracy; state that you’d optimize for the legally/ethically prioritized metric (e.g., minimize TPR gap subject to loss ≤ Δ). Close: propose an offline A/B or shadow deployment measuring real-world downstream impact, and iterate with improved data collection or causal analysis if disparities persist.

A second angle — "Train a privacy-preserving keyboard next-word model on mobile"

Framing differs: the dominant constraints are client compute, communication cost, and client-level privacy. MLE responsibility: integrate federated averaging, apply client-level DP with secure aggregation, tune per-client clipping and noise so utility remains acceptable, and design server updates to handle heterogeneous clients. Key differences from centralized DP: need to account for partial client participation (sampling rate impacts privacy accounting), and debug non-iid performance degradation across device demographics. Monitoring is on-device telemetry (with consent) and server-side aggregated metrics; implement lightweight per-slice evaluation to detect group performance regressions without collecting raw text.

Common pitfalls

Pitfall: Confusing fairness definitions and applying the wrong mitigation.
Picking demographic parity to fix label-bias problems will distort outcomes; ask whether labels reflect ground truth or historical discrimination before selecting remedies.

Pitfall: Presenting differential privacy as a binary guarantee.
Stating “we applied DP so users are safe” is misleading; report $(\varepsilon,\delta)$ , explain composition over training, and quantify expected utility loss for the chosen noise level.

Pitfall: Ignoring operational cost of mitigations.
A proposed adversarial or certifiably robust model that doubles inference latency without fallback plans will fail in production; always tie mitigation to latency/compute budgets and monitoring.

Connections

Interviewers may pivot to model monitoring & SRE questions (alerting, canary rollouts, rollback criteria) or to data-scientist-style causal diagnostics to root-cause bias (do-probing, counterfactuals). They may also ask about deployment pipelines (TFX, CI/CD) for integrating privacy/robustness steps.