Airbnb Trust, Safety, And Review Integrity Analytics

What's being tested

Interviewers are probing your ability to measure, detect, and attribute changes in trust, safety, and review integrity using rigorous statistical and causal tools. Expect to demonstrate experiment design, metric construction, anomaly diagnosis from a metrics lens, segmentation for heterogeneous effects, and model-evaluation thinking for classifiers that flag bad reviews/actors. Airbnb cares because small measurement or attribution mistakes can either miss fraud or wrongly penalize honest users, so the focus is on defensible, production-ready analytic reasoning.

Core knowledge

Metric hygiene: define primary/guardrail metrics (e.g., fraud_rate, review_submission_rate, appeal_rate), unit-of-analysis (booking, host, reviewer), and computation window; mismatched units produce Simpson’s paradox.
Randomization design: know cluster vs individual randomization tradeoffs; for host-level interventions prefer cluster assignment to avoid contamination and to satisfy SUTVA assumptions.
Sample size & power: for proportions use $n=\frac{(Z_{1-\alpha/2}+Z_{1-\beta})^2(p_1(1-p_1)+p_2(1-p_2))}{(p_1-p_2)^2}$ ; for continuous metrics use variance-based formula. Quantify minimum detectable effect (MDE) relative to baseline.
Sequential & ramping tests: apply alpha-spending/Bayesian sequential tests when doing business-need early ramps; pre-specify stopping rules to avoid inflated Type I error.
Multiple comparisons: control family-wise error or false discovery rate (Benjamini–Hochberg) when scanning many host cohorts, segments, or metrics.
Causal inference for observational signals: use difference-in-differences, propensity score matching, or synthetic controls to attribute shifts when randomization is impossible; check parallel trends and perform placebo tests.
Anomaly detection from metrics: combine time-series decomposition (trend, seasonality, residual), robust statistics (median/MAD), and thresholding by z-score or EWMA; confirm anomalies with secondary signals (content change, account events).
Classifier evaluation: for review-fraud models prioritize precision@k, calibration, and precision-recall over raw accuracy due to class imbalance; use ROC AUC but interpret with prevalence.
Labeling & feedback loops: beware of label bias (only flagged items reviewed) and feedback loops where model enforcement changes the distribution; use exploration buckets to gather unbiased labels.
Heterogeneous effects & uplift: test for differential impact across new_user vs experienced_user, geography, or host size; use interaction terms or uplift modeling to target policy.
Operationalization signals: tie analytic signals to sources like Postgres event logs, Snowflake aggregations, Looker dashboards; treat these as queryable signals, not infrastructure work.
Explainability & escalation: produce interpretable derivations (feature importances, counterfactual examples) to justify enforcement actions and satisfy product/legal stakeholders.

Worked example — "Design an experiment to test a new review verification flow requiring ID for first-time reviewers"

First 30s clarifying questions: define the objective metric (reduce fraudulent reviews vs maintain review volume), success criteria (relative reduction in fraud_rate and acceptable drop in review_submission_rate), unit-of-randomization (reviewer vs booking), and expected effect size. Skeleton answer pillars: (1) randomization plan — cluster on reviewer id with equal allocation and stratify by country; (2) metrics — primary: fraud_rate (flagged by manual review), guardrails: review_submission_rate, NPS; (3) sample size & rollout — compute MDE and plan ramp with pre-specified stopping; (4) analysis — intent-to-treat (ITT) estimate and complier average causal effect (CACE) to adjust for noncompliance; (5) monitoring & safety — near-term manual review of false positives and rollback thresholds. Key tradeoff to flag: stricter verification reduces fraud but increases friction and biases against new users; quantify tradeoff with cost per prevented fraudulent review. Close by saying if given more time you'd pilot in a small set of low-risk markets, instrument behavioral funnels, and gather labeled outcomes to train/refine automated detectors.

A second angle — "Investigate a sudden spike in 5-star reviews for a cohort of hosts"

Here you cannot rely on randomization, so apply the same causal rigor differently: first assemble multiple signals (timestamped ratings, review text embeddings, booking counts, payout changes). Use difference-in-differences comparing affected hosts to matched controls (match on historical rating trend, booking volume, geography). Run placebo tests on pre-spike windows to validate parallel trends. Augment with classifier outputs on review-text authenticity and look for covariates such as sudden referral traffic or promotional campaigns. Report sensitivity analyses (Rosenbaum bounds) to quantify how strong an unobserved confounder would need to be to explain the effect. Conclude with operational next steps: temporary manual audit of flagged reviews and retraining of models using new labels if fraud patterns emerge.

Common pitfalls

Pitfall: Defining the wrong unit-of-analysis — reporting effects at host level when randomization was at reviewer level leads to incorrect standard errors and false confidence.

Pitfall: Treating a drop in fraud_rate as success without checking guardrails — a lower fraud metric caused by fewer submitted reviews is a bad outcome.

Pitfall: Overclaiming causality from observational correlations — saying "UI X caused fraud to drop" without pre/post controls, placebo checks, or matching will undermine credibility.

Connections

Interviews often pivot to fraud-detection modeling (precision/recall tradeoffs for enforcement), experiment platform design (sequential testing and feature flags), or legal/compliance considerations for escalations and user appeals. Be ready to move from metrics to classifier evaluation and back.