Decide under adverse signals and conflicts

## Overview and Assumptions - We are evaluating a new notifications/messaging feature intended to increase meaningful engagement. - Department-level metrics (DL): CTR, send volume, open rate. - Company-level metrics (CL): retention, time-on-site, complaints/unsubscribes (negative), long-term connections. - Mixed signals: some guardrails worsening (unsubscribe complaints ↑), while mission outcomes are positive but lagging. - Goal: Pre-commit objective criteria, decide a cautious rollout, and align partners. --- ## 1) Pre-commitment Process: Success and Stop-Loss with Explicit Trade-offs 1. Define the Objective and Metric Hierarchy - Primary Objective: Maximize long-term user value subject to safety/brand constraints. - Metric hierarchy (lexicographic priority): 1) Hard guardrails (must-pass): legal/compliance/privacy, complaint/unsubscribe thresholds, crash rates, deliverability reputation. 2) Company-level outcomes: retention (D7/D28), time-on-site/session depth, meaningful social interactions. 3) Department-level metrics: CTR, send volume, open rate. - Rationale: Company-level outcomes dominate; DL metrics are inputs, not ends. 2. Choose Decision Framework - Option A: Lexicographic (recommended). Pass all guardrails; then require CL metrics to be neutral or positive; only then consider DL improvements. - Option B: Weighted OEC (Overall Evaluation Criterion) when trade-offs must be quantified: OEC = w1*(ΔRetention) + w2*(ΔTime-on-site) − w3*(ΔComplaints) + w4*(ΔMeaningful Interactions) + w5*(ΔCTR) with w3 ≫ w5 to encode brand/safety priority. Pre-commit weights from historical LTV studies. 3. Quantify Trade-offs Using LTV/Economic Terms - Estimate value of outcomes and cost of harms: Expected Net Impact per user = LTV_gain_from_retention − Cost_of_churn/complaints − Reputational/Deliverability risk penalty + Short-term engagement value - Example (illustrative numbers): - +0.20 pp D28 retention ⇒ +$0.30 LTV - +0.8% time-on-site ⇒ +$0.05 LTV - +0.05 pp unsubscribe complaints ⇒ −$0.40 LTV - +5% CTR ⇒ +$0.03 LTV (only if not cannibalizing) Net = 0.30 + 0.05 − 0.40 + 0.03 = −$0.02 ⇒ Do not launch unless we mitigate complaints. 4. Pre-define Success, Neutral, and Stop-Loss Criteria - Hard Guardrails (stop-loss if any breached): - Complaints/Unsubscribes: Δ ≥ +10 bps over baseline for ≥12 consecutive hours or ≥2x SEM above baseline. - Deliverability/Reputation: bounce/spam trap rate ≥ X threshold; blocklist incidents any ⇒ rollback. - Latency/Errors: p95 latency > Y ms or error rate > Z% for 2 ramps ⇒ hold. - Company-Level Outcomes (must be non-negative within CI): - D7 retention: Δ ≥ 0 or 95% CI excludes −MDE_neg (e.g., −0.05 pp). If lagging, use validated proxy models (see below). - Time-on-site/session depth: Δ ≥ 0 or neutral within alpha-spending plan. - Department-Level Targets (nice-to-have, cannot override guardrails): - CTR: Δ ≥ +3% (95% CI > 0) OR CTR × quality (post-click engagement ≥ baseline). - Send Volume: capped; no increase if quality drops (quality = downstream engagement − complaints). 5. Handle Lagging Outcomes with Proxies and Models - Pre-register proxy metrics with validation: - Short-term proxy for retention: 7-day return intent model; serial correlation of session depth; meaningful interactions per active day. - Backtest: R² and calibration on prior launches; pre-commit acceptance bounds (e.g., proxy predicts ≥0 effect with 90% PI). 6. Power, MDE, and Monitoring Plan - Compute sample sizes for guardrails and CL outcomes; ensure each ramp has enough exposure to detect harmful deviations. - Sequential monitoring with alpha spending (e.g., Pocock/O’Brien-Fleming) to avoid p-hacking. - Heterogeneity checks pre-committed (new vs. long-tenure users, high-frequency vs. low-frequency, region). 7. Rollout and Rate-Limit Caps (pre-committed) - Per-user rate limits (e.g., ≤1 new notification/day; quiet hours; frequency cap by channel). - Content quality filters (only top decile relevance score during early ramps). - Automatic backoff if complaint rate in last N sends exceeds threshold. 8. Governance and Decision Rights - DRI (Directly Responsible Individual) named; approvers list. - Single source of truth doc with thresholds, formulas, dashboards, and runbooks. --- ## 2) Decision Memo Structure for Launch Under Adverse Leading Indicators 1. Title and Summary (1–2 paragraphs) - Decision asked: Proceed with phased rollout to X% or hold. - Current evidence: CTR +5–7%; complaints +6–8 bps; proxies suggest neutral-to-positive retention; long-term outcomes lagging. - Recommendation: e.g., Proceed to 5% with tightened guardrails and new mitigations; do not exceed 10% unless complaint rate ≤ +3 bps. 2. Context and Goals - Problem, target users, expected value, risks. Metric hierarchy and OEC definition. 3. Experimental Evidence to Date - A/A checks passed; data quality verified. - Results by metric tier: guardrails, CL, DL with CIs and MDEs. - Heterogeneity: any segments at risk. - Externalities: deliverability, saturation, cannibalization. 4. Modeled/Proxy Evidence for Lagging Metrics - Retention proxy model performance, backtests, prediction intervals for current ramp. - Sensitivity analysis (best/base/worst case) and LTV translation. 5. Risk Assessment and Mitigations - Risks: complaints, deliverability, legal, privacy, infra load, user trust. - Mitigations: frequency caps, content relevance floor, quiet hours, onboarding education, improved unsubscribe UX, rate-limited backoff, geo/user cohort scoping. 6. Phased Rollout Plan - Proposed ramp: 1% (24–48h) → 5% (48–72h) → 10% (72h) → 25% (1 week) → 50% (1 week) → 100%. - Entry criteria for each ramp: all guardrails within bounds; CL proxies ≥ 0; no high-risk segment regressions. - Exit/hold criteria: any hard guardrail breach; CL outcomes negative beyond pre-committed MDE_neg; infra/ops alerts. 7. Contingency Rollback Triggers (pre-registered) - Auto-rollback if: - Complaints +≥10 bps for ≥12h window (or 2x SEM); - Deliverability blocklist or spam trap spike > threshold; - D7 return intent proxy < −0.3σ from baseline for 24h; - Repeated p95 latency breaches across 2 checks. - Manual rollback authority and on-call rotation defined; action time ≤ 60 minutes. 8. Decision Log and Approvals - DRI, Data Science, Eng Lead, PM, Policy/Legal (if applicable), Support. - Timestamped sign-offs; dissenting opinions captured. 9. Next Steps and Owner Matrix - Owners for mitigations, dashboard updates, rollout, and comms. Appendices: Detailed metrics, model validation, dashboards, and runbooks. --- ## 3) Communication Plan to Secure Cross-Functional Alignment 1. Upfront Alignment on Principles - Share the metric hierarchy and OEC; emphasize that company-level outcomes and safety guardrails override local optimizations. - Conduct a 30-minute pre-mortem: enumerate plausible failure modes and map to mitigations and triggers. 2. Clear Docs and Artifacts - Living spec/PRD section with: goals, experimentation plan, thresholds, rollout schedule, dashboards, runbook. - One dashboard per tier: Guardrails (always top), Company-level, Department-level; red/amber/green statuses tied to thresholds. 3. Cadence and Decision Rituals - Daily 15-minute stand-up during ramps; a single Slack/Teams channel for alerts. - Checkpoints at each ramp gate to review entry/exit criteria; minutes recorded in the decision log. 4. Roles and Decision Rights (RACI) - DRI = PM or DS; Approvers = DS Lead, Eng Lead; Consulted = Policy/Legal/Support; Informed = broader org. - Clarify that any guardrail breach auto-triggers hold/rollback without committee delay. 5. Engineering Partnering - Translate thresholds into SLOs and automated monitors (alerts, feature flags, circuit breakers). - Provide event schemas and data quality checks; define on-call rotations and rollback playbook. 6. Handling Conflicting Metrics in Meetings - Start with guardrail status; then CL outcomes; then DL metrics. - Use pre-committed thresholds to avoid opinion battles; show sensitivity and LTV translation. - If adverse leading indicators persist, propose concrete mitigations (e.g., raise relevance threshold, lower send cap) and a re-test plan. 7. Stakeholder Confidence - Publish a one-page weekly update: ramp status, any incidents, actions taken, forecast vs. actual. - Invite dissent: document and address concerns explicitly; adjust mitigations if new risks surface. --- ## Pitfalls and Guardrails - Do not let CTR or send volume override complaints/unsubscribe signals. - Beware heterogeneity: protect vulnerable segments (e.g., new users) with stricter caps. - Avoid peeking without alpha control; pre-commit analysis plan. - Validate proxies; do not over-rely on uncalibrated models for retention. - Ensure privacy and policy reviews are complete; treat these as hard blockers. ## Minimal Example of Pre-Commit Table (illustrative) - Hard stop-loss: Complaints Δ ≥ +10 bps/12h; Deliverability blocklist any; p95 latency > 2x SLO. - Must-pass CL: D7 return intent proxy ≥ 0; Time-on-site Δ ≥ 0 (CI includes 0 at worst). - DL goals: CTR Δ ≥ +3% with downstream engagement ≥ 0. - Ramp gates: 1%→5% only if all above satisfied; hold otherwise, apply mitigations, re-measure 24–48h. This approach makes the trade-offs explicit, encodes them in pre-committed thresholds, and operationalizes a safe, transparent decision process that partners can execute and trust.