Deliver an elevator pitch and impact example

# 1) 60-second Elevator Pitch - I’m a data scientist with 7+ years in consumer growth and marketplace/notifications. I’ve run 200+ online experiments across products reaching 100M+ MAU, shipping features that moved DAU and revenue at scale. - My superpower is turning ambiguity into decision-ready experiments: crisp problem framing, clean metrics, and pre-registered analyses that stakeholders trust. - I partner closely with engineering and PMs, and I’m known for fast, reliable reads (CUPED/stratification) and telling a causal story that drives roadmap choices. Tip: Practice a 3-sentence version: role + scale, superpower + one quantified impact, collaboration style. # 2) Experimentation Case Study: Send-Time Personalization for Push Notifications Scenario: We wanted to grow high-quality sessions by sending each user notifications at their best time-of-day. A) Problem Framing - Observation: Notification open rates were flat, and weekly opt-out (“mute/unsubscribe”) rates were creeping up by +0.05 p.p./week. - Goal: Increase notification-driven session starts without harming user experience. - Decision: Build a per-user send-time model versus fixed times; test via A/B. B) Hypothesis - H1: Personalizing send-time will increase notification-driven session starts per user-week by ≥2% relative. - H2 (guardrail): Opt-out rate will not worsen by more than +0.10 percentage points. C) Unit of Randomization - User-level randomization (1:1). Rationale: Treatment is delivered at the user level; minimal network interference; avoids contamination. - Stratified by: app platform (iOS/Android), region (US/ROW), and engagement tier (low/med/high) to balance covariates and improve power. D) Metrics - Primary: Notification-driven session starts per user per week. - Attribution: session within 10 minutes of a received push (last-touch). - Key secondary: Notification open-through rate (OTR). - Guardrails: - Opt-out/mute rate (weekly, p.p.). - Negative feedback rate on notifications (p.p.). - Battery impact (avg CPU/network per active user). - Experiment collision rate (overlapping tests), crash rate. E) Sample Size, Power, Duration - Design: Two-sided test, α = 0.05, power = 0.80. - Metric type: Approximate primary as continuous (sessions per user-week) with historical σ ≈ 0.90 and mean ≈ 0.80. - Minimum Detectable Effect (MDE): +2% relative on mean = δ = 0.016 sessions/user-week. - Formula (two-sample t-test approximation): n_per_group ≈ 2 × (Z_{1-α/2} + Z_{1-β})^2 × σ^2 / δ^2 With Z_{1-α/2} = 1.96, Z_{1-β} = 0.84: n_per_group ≈ 2 × (2.8)^2 × (0.9)^2 / (0.016)^2 ≈ 61,000 users per group per full week. - CUPED variance reduction (25% observed historically) effectively reduces required n to ~46k per group. - Duration: 14 days to cover two weekly cycles and weekend effects; 10% → 50% → 100% ramp within the experiment while maintaining 1:1 assignment. F) Pre-registration / Analysis Plan - Assignment: User-level ITT (intention-to-treat). - Invariants check: Balance on key covariates (platform/region/engagement) and pre-period outcomes. - Variance reduction: CUPED using prior-week sessions (X): Y_adj = Y − θ (X − E[X]), where θ = Cov(Y, X)/Var(X). - Estimator: Difference-in-means with cluster-robust SEs at the user level; stratification fixed effects. - Multiple metrics: Control family-wise error by pre-specifying primary and interpreting guardrails descriptively unless breached. - Early looks: O’Brien–Fleming alpha-spending for optional stopping (checks at day 7 and 14). - Exclusions: Known push-denied users; catastrophic log gaps; retain all others in ITT. G) Execution Challenges - Capacity limits: Coordinated with infra to stagger send windows; used feature flags to rate-limit. - Time zones/daylight savings: Derived local send windows from device time; validated with synthetic tests. - Event attribution: Implemented 10-minute last-touch rule and de-duplicated bursts. - Experiment collisions: Registered and filtered users in high-conflict cohorts (other notif tests); monitored collision rate. - Novelty effects: Tracked effect decay over the 2-week window; planned post-ramp holdout. H) Results (illustrative but internally consistent) - Primary: +2.6% sessions/user-week (ITT), 95% CI [+1.8%, +3.4%], p < 0.001. - Secondary: +5.1% OTR, 95% CI [+3.9%, +6.3%]. - Guardrails: - Opt-out rate: −0.08 p.p. (improvement), 95% CI [−0.12, −0.04]. - Negative feedback: +0.01 p.p., n.s. - Battery: +0.2% CPU per active user, within SLO. - Heterogeneity (pre-specified): Larger effects for “low engagement” users (+4.3%) and evening-preferring clusters; iOS > Android. - Business impact: At 50M eligible weekly users, +2.6% translates to ~1.3M incremental weekly sessions, with improved opt-out—approved for 100% rollout. I) Causal Story (Why It Worked) - Mechanism: Aligning send-time with user availability increases salience and reduces interruption cost. Higher last-touch probability leads to more opens and near-immediate sessions. - Evidence: Lift concentrated where model confidence was high and during predicted peak times; no increase in negative feedback—suggests higher relevance rather than over-sending. J) Trade-offs Considered - Volume vs. quality: We held message volume constant to isolate timing; next step is jointly optimizing volume and timing. - Fairness: Guarded against systematically deprioritizing certain time zones or work schedules; monitored subgroup effects. - Platform complexity: Additional scheduling complexity vs. measurable lift; validated reliability under infra constraints. K) What I’d Do Differently - Long-run effects: Staggered geo rollouts with dark-holdout to measure persistence and novelty decay. - Modeling: Contextual bandits for joint timing + content; incorporate cost-aware policies (battery, channel fatigue). - Quality outcomes: Add downstream guardrails (session depth, well-being proxies) to avoid optimizing only last-touch. - Interference checks: Small cluster-randomized holdout by household/device family to confirm negligible spillovers. Teaching notes: The key is crisp pre-specification, a defensible primary metric that maps to business value, realistic power math, and a clean causal narrative. Guardrails should reflect user trust and system health. # 3) Why Meta? Product Surface + Fit - Motivation: I’m excited by Meta’s scale, rapid experimentation culture, and the chance to balance growth with integrity and long-term user value. - Product surface: Instagram Reels notifications and discovery. It’s a high-leverage surface connecting creators and viewers where timing, ranking, and user well-being all matter. - Fit: My strengths in experimental design (powering large-scale AA/A/Bs, CUPED, stratification), causal inference, and metric design map directly to optimizing alert relevance, watch-time quality, and opt-out/negative feedback guardrails. I’m comfortable partnering with engineering to build reliable experimentation plumbing and with PMs to define MDEs that matter. - Impact plan: Start by auditing metrics and invariants, ship a fast E2E timing/content test with pre-registered guardrails, then scale via adaptive policies and heterogeneity-aware insights for creators and cohorts. Checklist you can adapt: - State the business problem in one sentence; name the lever (e.g., timing). - Hypothesis with a numeric MDE that matters. - Unit of randomization and interference rationale. - Primary metric and 2–4 guardrails tied to user trust/system health. - Power math with assumptions and a duration plan. - Pre-registration: ITT, variance reduction, multiple-testing approach. - Execution risks and mitigations. - Results with CI/p-values and p.p. changes on guardrails. - Causal story, trade-offs, and a concrete “do next.” - Close with a specific team/surface and how your skills drive impact there.