Communicate trade-offs and influence launch

# Overview This is a classic person‑level value vs. unit‑level cannibalization problem with interference across accounts belonging to the same person. The decision requires: (a) reframing metrics at the person level, (b) estimating longer‑term value and churn risk, (c) mitigating cannibalization, and (d) roll‑out and monitoring with clear guardrails. --- ## 1) Framing the trade‑off for PM and partners - Define the units clearly: - Person‑level (all accounts a person uses): total time/engagement, revenue proxy, retention. - Account‑level: usage per account, creator/supply health, fairness across a person’s accounts. - Summarize the core trade‑off: - Benefit: Notifications drive incremental engagement/actions on notified accounts. - Cost: Cannibalization—usage shifts away from non‑notified sibling accounts for the same person; potential reduction in account diversity (number of distinct accounts used per week). - Key questions to align on: 1) Is person‑level total value positive (net engagement/revenue uplift after cannibalization)? 2) Is the reduction in account‑level diversity acceptable given ecosystem goals (e.g., supporting multiple identities, creators, or business pages)? 3) Do we see early signals of fatigue (mutes/unsubscribes) or churn risk linked to higher notification pressure? - Simple decision grid: - If Δ person‑level value > 0 and diversity drop within guardrails → consider launch with mitigations. - If Δ person‑level value ≤ 0 or diversity drop breaches guardrails → iterate on targeting/frequency before launch. Define two diagnostic metrics: - Net incremental per‑person value (NIPV): incremental sessions, time, or revenue per person across all their accounts. - Cannibalization ratio (CR): absolute loss on non‑notified accounts divided by gain on notified accounts. CR close to 1 implies mostly re‑allocation; CR < 0.5 with positive NIPV implies healthy net growth. Small example: - Notified account: +3 sessions/person/week; Other accounts: −2; Net = +1 → CR = 2/3. If +1 session converts to meaningful revenue or retention lift, this may be acceptable; if not, mitigate. --- ## 2) Estimating longer‑term user value and churn risk Goal: move beyond short‑term clicks to person‑level CLV and retention, accounting for interference. A) Ensure correct experimental design - Prefer cluster randomization at the person level (all accounts for a person assigned together) to avoid cross‑account spillovers violating SUTVA. - If initial test was account‑level randomized, re‑run with person‑level randomization or use within‑person reweighting/IV approaches to bound effects. B) Person‑level CLV modeling - Define CLV proxy if direct revenue is unavailable: use ad‑impression value or time‑based proxy. - Incremental CLV: - ΔCLV = Σ_t [ (ΔE[t] × v) × γ^t ] - ΔE[t]: incremental engagement (e.g., minutes, sessions) per person at horizon t - v: value per unit engagement (e.g., revenue/minute) - γ: discount factor per period (e.g., weekly 0.98–0.995) - Measure ΔE[t] via long‑horizon experiments (4–8+ weeks) or retention‑curve extrapolation (see below). C) Retention and churn risk - Fit survival/retention models at the person level: - Hazard(t | treatment, notif_volume, diversity, controls) - Check whether higher notification volume or reduced account diversity raises hazard. - Construct leading indicators: - Muting/unsubscribe rate, notification opt‑out, complaint reports. - Drop in “distinct accounts used per week” and “switches per session.” D) Account‑level activity diversity - Metrics per person per week: - Distinct accounts active (breadth) - Entropy of account activity share: H = −Σ p_i log p_i (higher = more diversified) - Gini of time across accounts (fairness/fragmentation) - Set acceptable deltas (e.g., ≤1% drop in breadth; ≤3% drop in entropy) based on historical variance and business goals. E) Decomposition of effects - Decompose net uplift into: - Direct effect on notified accounts - Indirect spillover on other accounts - Net person‑level effect = Direct + Indirect - Report by segment: number of accounts per person, primary vs. secondary accounts, recency of use, market. F) Long‑term extrapolation guardrails - Use cohort‑level difference‑in‑differences to estimate persistence/decay of ΔE[t]. - Sanity check with a long‑term holdout (e.g., 5–10% of people) for 8–12 weeks to verify model projections. --- ## 3) Mitigations and phased rollout with guardrails Mitigation levers - Targeting across a person’s accounts: - Preferentially notify accounts that are under‑engaged or at risk (e.g., non‑preferred or lagging accounts) to rebalance usage. - Rotate which account gets a notification per day to preserve diversity. - Skip notifications if a person recently engaged that account organically (no need to cannibalize). - Frequency and pressure control: - Per‑person frequency caps (e.g., ≤N notif/day, ≤K notif/week across all accounts) - Diminishing‑returns logic: estimated marginal value per additional notification must exceed a cost threshold that includes cannibalization risk. - Eligibility rules: - Pause for users showing fatigue signals (recent mutes/unsubscribes) - Cool‑off after low‑quality sessions (short bounces after notif‑driven opens) - Time‑of‑day and recency constraints to avoid clustering. - Content quality and intent: - Prioritize notifications with high predicted satisfaction (not just CTR), e.g., long session time, saves, positive feedback. - Avoid overlapping topics across a person’s accounts in short windows. Phased rollout plan - Phase 0: Re‑test with person‑level randomization and new mitigations on 1–2% of eligible people. - Phase 1: Expand to 10–20% if guardrails met for 2 consecutive weeks. - Phase 2: 50% rollout with a persistent 10% holdout for ongoing measurement. - Full: 100% only after long‑term holdout confirms no retention/diversity harm. Example guardrail thresholds (tune using historical variance and MDE) - Person‑level net value: NIPV ≥ +0.5% (or ≥ X revenue/minute per 1,000 notifications) - Account diversity: distinct accounts/week Δ ≥ −1.0%; entropy Δ ≥ −3.0% - Retention: WAU retention Δ ≥ −0.2% overall; ≥ −0.5% in multi‑account heavy segment - Fatigue: notif mutes Δ ≤ +10 bps; unsubscribes Δ ≤ +5 bps - Quality: bounce rate after notif‑open Δ ≤ +0.3pp; complaint rate no increase - Supply‑side: no negative impact to non‑notified account creator metrics beyond −0.5% If any guardrail breaches for 7 consecutive days (or a sequential test flags), auto‑pause or revert to previous phase. --- ## 4) Post‑launch monitoring plan Foundational instrumentation - Person‑level identity linkage across accounts, treatment exposure logs, and session attribution (notif‑driven vs organic). - Per‑person rollups (daily/weekly) to avoid misinterpreting account‑level gains as person‑level wins. Dashboards and alerts (segmented by multi‑account intensity) - Core outcomes: - Person‑level DAU/WAU/MAU, sessions/person, time/person, revenue proxy - NIPV per 1,000 notifications; CR (cannibalization ratio) - Diversity and switching: - Distinct accounts active per week; entropy/Gini of account time - Cross‑account switch rate per session; dwell time by account type - Fatigue and satisfaction: - Mutes, unsubscribes, complaint reports, “mark as spam,” app settings changes - Post‑open session quality: bounce rate, depth, repeat opens next day - Retention: - 1d/7d/28d retention changes; survival curves; hazard over notif pressure - System health: - Delivery latency, failure rate, burstiness; overlap/duplicate content alerts Ongoing experimentation and guardrails - Maintain a 5–10% person‑level geo or user holdout for continuous causal read. - Weekly sequential tests (e.g., SPRT/CUSUM) to detect drifts in guardrail metrics; BH‑corrected where multiple tests. - Drift detection in model scores (notif ranking) and feedback loops. Investigation playbooks - If diversity drops: tighten per‑person caps, increase rotation across accounts, raise quality thresholds for notifications to dominant accounts. - If fatigue increases: lengthen cool‑offs, reduce nighttime sends, suppress lower‑value topics. - If retention risk rises in a segment: roll back there first; run targeted A/Bs with stricter caps. --- ## Summary decision logic - Launch if: person‑level net value is positive and statistically reliable; diversity and retention guardrails are respected; fatigue stable; mitigations in place. - Iterate if: gains are primarily re‑allocation (high CR) or diversity drops beyond thresholds—apply balancing, frequency caps, and eligibility. - Continue measuring long‑term with persistent holdouts and explicit person‑level metrics to ensure sustained value without degrading the multi‑account ecosystem.