Handle PM Ambiguity and Drive Outcomes

## Part A — Leadership Under Ambiguity (Teaching Approach + Example STAR Story) Framework to use: - Start by reframing the ambiguous goal into a testable hypothesis and a single north-star metric, with guardrails. - Use structured prioritization (Must/Should/Could) to negotiate scope under time pressure. - Make alignment visible: brief written PRD-lite/1-pager, decision log, RACI, and a short cadence of updates. Example STAR story (as a Data Scientist): - Situation: A PM proposed a broad "revamp the home experience to improve engagement" with no clear success definition, and wanted engineering to start immediately to hit a launch event. - Task: I needed to convert a vague objective into measurable outcomes, right-size scope, and ensure engineering and design had crisp targets and timelines. - Actions: 1) Defined measurable success and guardrails: - North-star: 7-day retained DAU uplift (+2% absolute) within 4 weeks. - Primary levers: Session starts per DAU (+5%), search-to-engagement rate (+3 pp). - Guardrails: Crash rate ≤ baseline +0.2 pp; p95 latency ≤ baseline +50 ms; revenue conversion ≥ baseline. - Wrote a 1-pager: hypothesis, metrics, segments, and how we’d read the experiment (success thresholds, stopping rules). 2) Negotiated scope with Must/Should/Could: - Must: Ship simplified feed modules and improved CTA copy (highest expected impact, low risk). - Should: Add personalized sorting (behind a flag) if engineering completes Must by mid-sprint. - Could: Complex recommendations v2 (deferred behind a separate flag). - Trade-off: Dropped a visual animation that added latency with uncertain value. 3) Aligned stakeholders: - Set up a feature flag rollout plan (10% → 25% → 50% → 100%). - Created a Looker dashboard tracking the north-star and guardrails by platform and cohort. - Daily 15-minute standup with Design/Eng/PM; twice-weekly summary to leadership with a decision log. - Results: - Achieved +2.4% 7-day retained DAU and +6% session starts per DAU at 50% rollout, with guardrails in bounds. Completed rollout in week 3. - The written metrics contract prevented scope creep and made the final go/no-go decision straightforward. Principles illustrated: - Make the success definition explicit before building. - Separate hypothesis from implementation: ship smallest testable package first. - Use flags and staged rollout to learn safely under time pressure. --- ## Part B — 48-Hour Incident Plan (20% DAU Drop After UI Change) Assumptions: - DAU is defined as unique users with at least one qualifying activity in a 24-hour window. - We have access to: product analytics (e.g., Amplitude/Looker), data warehouse (e.g., BigQuery/Presto/Hive), feature flag logs, server logs, monitoring (e.g., Datadog/Sentry), mobile release data, and crash/latency metrics. - Feature was released behind a flag or via a new app version. High-level objectives: - Verify the drop is real (not telemetry/ETL). - Bound blast radius (which platforms/versions/regions/cohorts). - Decide quickly: targeted rollback vs. forward fix behind a controlled holdout. - Maintain continuous, concise communication. ### 0–2 Hours: Triage and Truth Establishment 1) Freeze changes and assemble a war-room (PM, Eng, Data, QA, Support). 2) Snapshot key metrics vs. 7/14/28-day baselines: - DAU by platform (iOS/Android/Web), app version, region, new vs. returning, and feature-flag exposure. - Session starts per DAU, search starts, key funnel steps, conversion rate to core action (e.g., save/call/book), time-to-first-action. - Crash rate, ANR, p95 latency, HTTP 4xx/5xx, event ingestion lag, ETL row counts. - Rollout/flag exposure by cohort; build versions released in last 48 hours. - External confounders: outages (CDN, auth), marketing sends, seasonality, holidays. Sources: analytics dashboards, warehouse SQL, monitoring dashboards, feature flag system, release notes. 3) Fast instrumentation/data-quality checks (rule-out telemetry issues): - Compare DAU from server-side auth/session logs vs. client analytics events. If client-only drops but server DAU is stable, suspect instrumentation. - Check event schema/version changes: did the event name or property change? Any spike in null user_id or anonymous traffic mismatch? - Validate event volume timing: ingestion lag or late-arriving events (check partitions vs. wall time). - Spot-check end-to-end: use a test device to perform the action; confirm events appear in real-time stream and warehouse. - Recompute DAU from raw server logs for a random 1% sample to confirm counts. - Check deduplication or user-ID merge jobs for failures. Decision gate A (after 2 hours): - If server-side DAU is normal but client DAU is down → treat as telemetry incident; do NOT revert UI yet. Prioritize fixing analytics and continue monitoring server-side guardrails. - If both server and client DAU are down and guardrails (crash/latency) worsened → prepare rollback path while diagnosing scope. ### 2–6 Hours: Scope Blast Radius and Choose Initial Mitigation 4) Segment analysis to localize impact: - By platform, app version, region, new vs. returning users, and flag exposure. - Funnel deltas: where is the drop? Session starts? Entry to key action? Post-click conversion? - Check if drop correlates with a specific UI variant or navigation path. 5) Define explicit rollback vs. forward-fix thresholds: - Immediate rollback if ANY of the following hold for ≥2 consecutive hours: - Server-side DAU drop ≥15% across ≥2 platforms, or ≥20% on the top platform. - Crash rate up by ≥0.5 pp absolute or p95 latency up by ≥100 ms versus 7-day baseline. - Key conversion rate (to core action) down ≥10% with p < 0.01. - Targeted rollback (platform/version/region) if impact is localized and non-telemetry. - Forward-fix allowed if impact is localized, guardrails within +/− tolerance, and fix ETA < 24 hours with holdout in place. 6) Minimal viable control design (if not reverting fully): - Turn the UI change into a 50/50 user-level flag split immediately (or 10/90 if risk is high), stratified by platform and new/returning status. - Ensure consistent assignment (sticky bucketing) to avoid contamination. - Establish guardrail monitoring in real time for both groups. Communication (first update): - Send a brief status to leadership: what changed, current impact, suspected cause, immediate mitigations, decision thresholds, and next update time. - Notify Support with a short macro: who’s affected, known workarounds, and ETA for next update. ### 6–12 Hours: Decide on Rollback vs. Forward Fix 7) Statistical read of the emergency holdout: - Primary outcome: session starts per user and entry to core funnel (faster signal than DAU). - Use proportion difference tests or Bayesian A/B with sequential monitoring; predefine stopping rules. - Example sample sizing for a proportion metric: n_per_group ≈ 16 × p × (1 − p) / MDE^2. - If baseline session-start rate p = 0.30 and MDE = 0.02 (2 pp), then n ≈ 16 × 0.21 / 0.0004 ≈ 8,400 users per group. - With high traffic, you’ll reach this quickly; otherwise, focus on higher-frequency proxies than DAU. 8) Decision gate B (by hour ~12): - Rollback fully if treatment group shows ≥5% drop on leading indicators with p < 0.01, consistent across top platforms/cohorts, and no telemetry artifacts. - Otherwise, pursue targeted rollback or forward-fix with the holdout maintained until the next release. ### 12–24 Hours: Implement and Validate Fix or Rollback 9) If rolling back: - Execute rollbacks by flag (instant) or hotfix (app release) as needed. - Verify metrics recover in both server and client views within 2–4 hours. 10) If forward-fixing: - Implement and ship the smallest corrective changes (e.g., restore prior navigation affordance, increase affordance contrast, fix broken entry point). - Keep 10–20% holdout of the original experience for continued validation. - Monitor guardrails every hour. 11) Update communications: - Leadership: current metrics vs thresholds, decision taken, ETA for full recovery, next checkpoint. - Support: refreshed macro with latest guidance. ### 24–48 Hours: Stabilize, Confirm Causality, and Prepare Postmortem 12) Causality validation (beyond the emergency holdout): - If flags were not available pre-incident, use difference-in-differences with an unaffected platform/region as control. - Cross-check with pre-post on the same cohort while adjusting for day-of-week and seasonality. - Sanity check: recovery after rollback should mirror the initial decline if the UI change was causal. 13) Finalize mitigation: - If metrics are back within 2% of baseline for ≥12 hours and guardrails normal, proceed to remove emergency measures. 14) Postmortem plan (start drafting by hour 36, publish by day 5): - Timeline: what changed, when detected, who responded, and decisions made. - Impact quantification: user-minutes lost, DAU delta, conversion delta, revenue impact. Example: impact_hours × affected_users × average_sessions. - Root cause analysis (5 Whys): telemetry, rollout strategy, usability regression, or performance issues. - Preventative actions: - Launch gates: require experiment or 10/25/50/100% staged rollout with kill switch. - Guardrail alerts: 3-sigma anomaly detection on DAU, session starts, and core funnel; server-side truth alerts separate from client. - Data quality checks: schema versioning, contract tests in CI, backfill validation, user-ID merge tests. - Observability: crash/latency alerts and dashboards by version/flag. - Change management: lightweight RFC with risk assessment and clear rollback playbook. - Experiment hygiene: sticky bucketing, pre-registered success metrics and thresholds. --- ## Quick Reference — Thresholds and Formulas - Rollback now if: server-side DAU −15%+ across ≥2 platforms or −20% on lead platform; or crash rate +0.5 pp; or key conversion −10% (p < 0.01). - Forward fix allowed if: impact localized, guardrails okay, fix ETA < 24h, and holdout active. - Sample size for proportion metrics: n_per_group ≈ 16 × p × (1 − p) / MDE^2. - Validate telemetry by reconciling server-side actives vs client events, and checking ingestion lag, schema changes, and null/duplicate IDs. This plan balances speed and rigor: confirm the drop is real, bound the blast radius, use explicit thresholds for decisions, validate causality via holdouts or quasi-experiments, communicate crisply, and institutionalize learnings to prevent recurrence.