Describe resolving revenue–UX metric conflict

# Model Answer (STAR) — Monetization vs. UX on a UGC Gaming/Social Platform ## Situation We were asked to increase ad revenue ahead of a major quarter while protecting core engagement. Product proposed increasing ad load on the home feed and adding a single interstitial during world-to-world teleports. The risk was degrading short-session users, new users, and teens—key cohorts for long-term retention. ## Task As the lead data scientist for ads, I owned the decision framework: define success metrics and guardrails, design the experiment, implement a staged rollout with sequential monitoring, and recommend go/no-go and targeting. ## Action ### 1) Metrics in tension (with baselines and targets) - Monetization metrics - Ad revenue per session (ARPS): baseline $0.21; target +6–8%. - Ad impressions per session: baseline 5.2; proposal +10–15%. - Fill rate and RPM (revenue per mille impressions) monitored for budget/quality shifts. - UX/safety metrics - Session length: baseline 14.2 minutes; acceptable change ≥ −2.0%. - Bounce rate (<60s): baseline 17.5%; acceptable change ≤ +0.5pp. - D1 retention: 41.8%; acceptable change ≥ −0.3pp. - D7 retention: 13.2%; acceptable change ≥ −0.2pp. - User complaints per 10k sessions: baseline 8.1; acceptable change ≤ +10%. - App crash rate: baseline 0.33%; acceptable change ≤ +0.03pp. Definitions - ARPS = total ad revenue / total sessions. - Dd retention = P(user active on day d | installed/active on day 0). - 30-day Ad LTV approximation: LTV_30 = Σ_{d=1}^{30} Ret_d × ARPDAU_d. ### 2) Guardrails and rationale - Bounce rate: +0.5pp limit. Justification: historically, +0.5pp translates to −0.15pp D7 and −1–2% LTV over 90 days. - Session length: −2% limit to preserve creator economy and recommendation quality. - D1/D7 retention: −0.3pp/−0.2pp respectively; these are material at our scale and correlate strongly with LTV. - Complaints: +10% max to avoid user trust erosion and moderation load spikes. - Crash rate: +0.03pp cap to maintain technical reliability. - Policy: Exclude under-13 users entirely from the new interstitial format (compliance and brand safety). ### 3) Decision structure - Stakeholders - Monetization PM (proposal owner) - Core UX PM (session metrics) - Trust & Safety (policy/complaints) - Ads Ops and Brand Safety (creative QA) - Data Engineering and Experimentation Platform (instrumentation) - Finance (forecasting) - Alignment and ownership - Decision owner: GM, Engagement & Monetization. - RACI: Monetization PM (Responsible), GM (Accountable), DS/Trust & Safety (Consulted), Eng/Ads Ops (Informed). - Artifacts: 6-pager with pre-reads, risk register, and pre-registered guardrails/stopping rules. - Timeline - Week 0–1: Define metrics/guardrails, power analysis, spec. - Week 2–3: Instrumentation and dark launch. - Week 4–6: Staged ramp and sequential monitoring. - Week 7: Decision and rollout plan. ### 4) Experiment and analysis - Design - Arms: Control; A) +15% feed ad load; B) +15% feed ad load + 1 teleport interstitial (shown once after 60s; frequency cap: 1 per 10 minutes; total cap: 6 ad exposures/session). - Randomization: user-level, stratified by platform (mobile/desktop), geography, and account age. CUPED using prior 14-day session metrics to reduce variance. - Power/variance - MDE for ARPS: 3%; for bounce and D7: 0.15pp. Required N ≈ 1.5–2.0M users per arm for 80–90% power. - Ramp plan and sequential monitoring - 1% dark launch (instrumentation only) → 5% → 10% → 25% → 50%, with 24–48h holds. - Group-sequential boundaries (O’Brien–Fleming) for harm on guardrails; mSPRT for ARPS uplift. - Risk mitigation - Creative QA and blocklists enabled; policy filters for sensitive segments; exclude under-13s from interstitial. - Kill-switch per cohort (new users, teens, specific geos). - Safety holdout (1% permanent) for backtesting. - Breach playbook (pre-registered) - If bounce +0.5pp in any protected cohort (new users: account_age < 7 days; teens): immediately disable interstitial for that cohort and continue monitoring feed-only variant. - If D7 projected −0.2pp: pause ramp; require re-tuning of frequency caps and creative mix. ### 5) What happened during the test - At 10% ramp (24h), cohort-specific breach - New users saw bounce +0.9pp in Arm B (interstitial). Action: executed playbook—disabled interstitial for account_age < 7 days; maintained feed-only Arm A for them. Within 48h, bounce delta normalized to +0.2pp. - Primary results at 50% ramp (days 7–14) - Arm A (feed +15%): ARPS +4.1% (95% CI: +3.4, +4.8), session length −0.6%, bounce +0.1pp, D7 −0.05pp (ns). - Arm B (feed + interstitial, excluding new users): ARPS +8.3% (CI: +7.5, +9.1), session length −1.3%, bounce +0.3pp, D7 −0.15pp (borderline but within −0.2pp guardrail). Complaints +6% (within cap); crashes unchanged. - LTV signal - Using retention × ARPDAU, LTV_30 uplift estimated at +2.2% for Arm A, +3.0% for Arm B among eligible users (account_age ≥ 7 days, 13+). ## Result ### Final decision - Roll out Arm B to eligible users (13+, account_age ≥ 7 days) with caps: max 1 interstitial per session, 6 total ad exposures/session; dynamic throttling for short sessions (<5 min) to protect UX. - Roll out Arm A to new users and short-session users; no interstitial for those cohorts. ### Quantified impact (two horizons) - Near-term (first 30 days post-rollout) - Monetization: blended ARPS +5.9% across all eligible users; incremental revenue +$3.8M vs. control forecast (scale-dependent; normalized by traffic). - UX: session length −1.1%; bounce +0.2pp; D7 −0.08pp. All within guardrails. - Medium-term (90 days) - Monetization: blended ARPS +5.1% (slight regression due to creative fatigue mitigated by rotation), LTV_90 +2.6% on eligible cohorts. - UX: D7 stabilized at −0.05pp; complaints returned to +3% with better creative QA; no crash impact. Business takeaway: The hybrid strategy captured most of the revenue (+5–6% ARPS) while protecting long-term engagement via cohort-based eligibility and frequency controls. ## Mistake and what I’d change - Mistake: I initially monitored guardrails only at the aggregate and by “new vs. existing,” which masked a larger bounce increase in 13–17-year-old short-session users within the “existing” bucket. We caught it at 10% ramp but could have flagged earlier. - Fix next time: - Pre-define segment-level guardrails for protected cohorts (teens, short-session users) with hierarchical monitoring and alerting. - Require “no material harm” per key cohort before ramping beyond 10%. - Add a pre-experiment observational backtest to tune interstitial timing for short sessions. ## Why this approach generalizes (guardrails and validation) - Guardrail-first design forces clarity on acceptable UX risk. - Cohort-targeted rollout preserves value while avoiding harm to sensitive users. - Sequential testing with pre-registered stop rules prevents p-hacking and limits downside. - CUPED and stratification reduce variance and speed decisions without sacrificing rigor. Formulas used - ARPS = Revenue / Sessions. - ΔD7 = D7_treatment − D7_control (pp). - LTV_30 = Σ_{d=1}^{30} Ret_d × ARPDAU_d. Common pitfalls to avoid - Simpson’s paradox across cohorts; always monitor segments. - Ad novelty and creative fatigue; plan rotation and re-measure at 60–90 days. - Interference: avoid cross-over by user-level randomization and frequency caps. - Seasonality/budget shifts: use holdouts and RPM/Fill diagnostics to separate demand shocks from UX effects.