Google-Scale Data Quality And Pipeline Reliability

What's being tested

Interviewers probe your ability to treat product metrics as signals: detect, diagnose, and prioritize real data-quality or reliability problems that affect downstream analysis and decisions. They want to see statistical reasoning (power, uncertainty, multiple-testing), pragmatic triage (fast checks vs deep forensics), and metric-design hygiene (definition, instrumentation, baseline behavior) — all from a Data Scientist’s accountability to produce trustworthy insights and experiment results.

Core knowledge

• Metric definition hygiene: A metric must have a clear numerator, denominator, inclusion/exclusion rules, and an owner. Ambiguity causes drift; always check the definition before debugging numbers.

• Signal vs. noise: Use standard error / confidence intervals for rates: $SE=\sqrt{\frac{p(1-p)}{n}}$ and relative change $\Delta = (p_{new}-p_{old})/p_{old}$. Small absolute changes may be statistically insignificant for small n.

• Power & minimum detectable effect (MDE): For proportions, $n\approx\frac{(Z_{1-\alpha/2}+Z_{1-\beta})^2 p(1-p)}{\Delta^2}$ — be explicit about $\alpha, \beta$ when interpreting an observed null or hit.

• Sample Ratio Mismatch (SRM): A different-than-expected treatment allocation often signals instrumentation or logging loss; compute expected vs observed counts with a $\chi^2$ test.

• Anomaly detection techniques: Use statistical process control like CUSUM and EWMA to detect small persistent shifts; use simple z-score or Poisson-rate tests for large sudden changes. Account for seasonality and auto-correlation.

• Multiple comparisons & FDR: When monitoring many metrics, apply Benjamini–Hochberg or stricter family-wise corrections (Bonferroni) to control false discoveries across dashboards.

• Instrumentation sanity checks: Validate by re-deriving a metric from raw events (clicks, impressions), cross-checking aggregated and event-level counts, and comparing different aggregation windows (daily, hourly) in BigQuery or logs.

• Segmented diagnosis: Always break down by device, country, user cohort, treatment, and time window; localizing the change often reveals causes (e.g., mobile-only, new SDK version).

• Alert thresholds & SLOs: Define SLIs for critical metrics (e.g., DAU, CTR) and set SLO/alerting bands based on historical volatility (e.g., mean ± 3σ) and business impact, not arbitrary percentages.

• Experiment integrity signals: Monitor holdout contamination, sample churn, and metric leakage; check pre-experiment balance, post-experiment behavior, and diversion in denominators.

• Quick forensic checklist: (1) Confirm metric definition, (2) Recompute from raw events, (3) Check upstream releases & experiments, (4) Segment to isolate, (5) Measure persistence vs one-off.

• Communication principle: Lead with business impact and confidence: quantify the size and uncertainty of the problem, propose a prioritized action, and state what additional data would change your recommendation.

Worked example — "Investigate a 20% drop in DAU"

First 30s framing questions: Which DAU definition (rolling 7-day vs calendar day)? When did the drop start (UTC vs local day boundary)? Any recent releases, experiments, or marketing changes? Are ingestion or reporting pipelines known to have issues? Skeleton of answer: (1) validate the metric by recomputing from raw daily active event logs for the affected day(s); (2) segment by platform, country, and app version to localize; (3) run SRM-like checks across cohorts and check for sudden user ID changes or spikes in anonymous sessions; (4) apply anomaly tests (z-score, CUSUM) to establish persistence. One tradeoff to flag: do a quick check using aggregated BigQuery queries and dashboards to decide if an urgent rollback/alert is needed, versus a full forensic replay which can take hours. Close: "If I had more time I'd replay raw event ingestion and cross-check third-party telemetry (ads, auth) and build a short-lived alert that trips if the drop persists or widens."

A second angle — "A/B shows a large lift in CTR but product metric purchase_rate unchanged"

Same core concept applies but constraints differ: here you must evaluate whether the CTR lift is real, driven by selective segments, or produced by instrumented duplication. Start by checking that the CTR and purchase_rate share a consistent denominator (same users/events). Run causal checks: is the traffic allocation balanced (SRM)? Segment the lift — is it confined to non-buying segments (e.g., new users, one country)? Consider sequential-peeking: was the experiment stopped early — adjust inference accordingly. Finally, explore whether metric cross-instrumentation produced artificial impressions (duplicates) that inflate CTR without affecting downstream conversion. Emphasize experiment validity (randomization, sample size, stopping rules) as a data-quality lens.

Common pitfalls

Pitfall: Mistaking seasonality for outage — comparing today to yesterday instead of the same weekday last week leads to false alarms; always compare to appropriate historical baselines.

Pitfall: Blaming the pipeline without evidence — asserting an ETL bug when a rollout or experiment explains the change undermines trust; present the minimal reproducible checks first (raw-event recompute).

Pitfall: Over-correcting with Bonferroni when monitoring many metrics — excessively strict corrections mask real incidents; prefer FDR control when the cost of some false positives is low.

Connections

This area naturally pivots to experiment design (power, stopping rules), causal inference (confounding and contamination), and ML model monitoring (drift detection and model SLOs). Interviewers may ask you to propose product actions or trade the depth of forensic work against time-to-alert.

Further reading

• Trustworthy Online Controlled Experiments (Kohavi et al.) — practical patterns for experiment integrity and metric validation.

• Benjamini & Hochberg (1995), "Controlling the False Discovery Rate" — why FDR beats naive multiple-testing corrections for monitoring.