Lead XFN decision under tight timeline

# Overview This is a 72-hour, high-stakes decision under imperfect data. The strategy is: align fast on a single decision question and thresholds, triage data for decision-quality insights (not perfection), deliver a persuasion-first narrative, and pre-commit to post-launch guardrails and rollback. Below is a step-by-step plan you can execute with one analytics and one engineering IC. ## 1) First 6 Hours: Clarify Goals and Alignment Objective: Lock the decision question, must-have metrics, guardrails, and get written sign-off. - 0:00–0:30 Kickoff with PM, Ops, Sales, Eng Lead - Frame a single decision question: "Should we expand the new quoting workflow to [target scope] now, or gate expansion by cohort, given speed, quality, and volume trade-offs?" - Enumerate options upfront for decision clarity: - A: Expand to 100% of traffic. - B: Partial expansion (e.g., 50% or specific categories/geos). - C: Hold and instrument/mitigate gaps. - 0:30–1:30 Define must-have metrics (primary and guardrails) - Speed-to-ship (PM’s north-star) - Median time-to-first-quote (TTFQ) - % of requests with ≥1 quote within 30 minutes - % of requests with ≥3 quotes within 24 hours - Pro quality (Ops’ north-star) - Hire quality proxy: 7-day hire rate and post-hire refund/dispute rate - Customer CSAT/NPS for completed jobs (if lagging, use support contact rate per 100 jobs) - Pro-side quality proxy: rehire rate or average rating of hired pros - Near-term volume (Sales’ north-star) - Quotes sent per request - Hire conversion within 7 days - GMV/revenue per request - Guardrails (must not breach) - Support tickets per 100 requests (+X% max delta) - Fraud/spam flags per 1,000 quotes (+Y% max delta) - Pro churn risk proxy: 7-day returning pro rate (−Z% max delta) - 1:30–2:15 Set explicit decision thresholds and confidence level - Example thresholds (to be agreed): - TTFQ improvement ≥ 10% with 90% CI lower bound ≥ 5% - 7-day hire rate delta ≥ −0.5 percentage points (90% CI lower bound ≥ −1.0pp) - Guardrails within pre-set bounds (e.g., support tickets ≤ +10%, spam ≤ +5%) - Confidence: Use 90% CIs given time constraints; complement with sensitivity/bounds. - 2:15–3:45 Create and circulate a one-page Decision Brief for sign-off - Includes: decision question, options A/B/C, primary metrics, guardrails, thresholds, confidence level, and data limitations. - Send for written ack in Slack/email; timebox responses to 2 hours. - 3:45–6:00 Data quick-scan + finalize scope - Confirm the decision cohort (e.g., all categories vs. top 5 by volume; exclude long-tail if data is too sparse). - Lock the observation window (e.g., last 14–28 days) and unit of analysis (request-level for TTFQ and conversion). Document assumptions. Deliverables (by hour 6) - Signed 1-pager with metrics, thresholds, and options. - Finalized cohorts, timeframe, unit of analysis, and confidence level. ## 2) Plan, Delegation, SLAs, Risk Register, and Cadence - Team structure and SLAs - Analytics IC (A): Owns metric definitions, queries, QA, CIs, deep-dives; SLA: first-cut metrics by hour 18; refined by hour 36; final by hour 60. - Engineering IC (E): Owns instrumentation patches/backfills, schema clarification, query performance and extract; SLA: schema map by hour 10; missing-field workaround by hour 24; final extract stability by hour 36. - Lead (You): Decision framing, stakeholder mgmt, narrative, pre-wire, deck; SLA: deck skeleton by hour 24; pre-wire by hours 36–60. - Work breakdown (high level) - A: Build canonical request→quote→hire dataset; compute speed/quality/volume metrics; run stratified cuts; compute CIs; write metric notes. - E: Snowflake schema discovery; fix or backfill missing experiment fields; build views/materialized tables; provide performance-optimized datasets. - You: Decision brief; risk register; RACI; stakeholder pre-wire; storyline; exec summary; final recommendation. - Risk register (top items with mitigations) - Missing experiment assignment for a subset of events → Mitigate via joining to assignment tables, cookie/user/device stitching, or impute via first treatment exposure; mark confidence as medium. - Stale dashboards → Ignore; rebuild from raw with validated logic. - Seasonal/category mix shifts → Use stratification and difference-in-differences vs. unaffected cohorts. - Sparse lagging outcomes (e.g., post-hire ratings) → Use leading proxies (support rate) with bounds; plan 2-week validation. - Time/bandwidth → Maintain a cut list of non-critical analyses; escalate only if it changes the decision. - Daily checkpoint cadence (72-hour timeline) - Day 1 (hours 6–24): 15-min morning stand-up; 30-min EOD review; async mid-day updates in channel. - Day 2 (hours 24–48): Same cadence; add 30–60 min pre-wire sessions. - Day 3 (hours 48–72): Dry run; exec pre-read; final polish. - RACI snapshot (bulleted by workstream) - Metric definitions and SQL: R = A; A = You; C = PM, Ops; I = Sales, E - Instrumentation/backfills: R = E; A = E; C = You, PM; I = Ops, Sales - Decision brief and thresholds: R = You; A = You; C = PM, Ops, Sales; I = Eng - Analysis and CIs: R = A; A = You; C = PM; I = Ops, Sales, E - Deck + pre-wire: R = You; A = You; C = PM, Ops, Sales; I = Eng - Go/no-go meeting: R = You; A = VP; C = PM, Ops, Sales; I = Eng, A ## 3) Data Triage Under Ambiguity - Schema discovery (Snowflake) - Inventory relevant objects: - information_schema.tables / columns and SHOW commands - Pattern search: table_name ILIKE '%quote%', '%request%', '%hire%', '%experiment%' - Example: - SELECT table_schema, table_name, row_count, last_altered FROM information_schema.tables WHERE table_schema IN ('PROD','ANALYTICS') AND table_name ILIKE '%quote%'; - Freshness and completeness checks - Freshness: SELECT DATEDIFF('hour', MAX(event_time), CURRENT_TIMESTAMP) AS hours_since_last FROM schema.events_quotes; - Completeness: NULL rates per critical fields (request_id, pro_id, variant, timestamp). - Uniqueness: COUNT(*) vs COUNT(DISTINCT request_id, pro_id, event_time) for duplicate detection. - Referential integrity: Anti-joins for orphan events. - Build a canonical fact set - requests (request_id, created_at, category, geo) - quotes (request_id, pro_id, quoted_at, quote_metadata) - assignments/experiments (entity_id, variant, start_at) - hires (request_id, hire_at, amount, outcome) - Join keys: request_id primary; handle device/user stitching cautiously. - Reconcile conflicting sources (hierarchy of truth) - Event logs as primary for timestamps; transactional tables for outcomes (hire, amount, refunds). - If experiment logs are missing variant for X% rows: - Join to user/assignment table by request or session; if still missing, infer by exposure window or exclude from variant-level cuts but include in overall trend analysis; quantify impact. - Document data quality gaps with impact assessment - Template per issue: description, affected metrics, severity (blocker/high/medium/low), direction of bias, mitigation, residual risk. - Example: 18% quotes missing variant → affects variant comparisons; severity medium; mitigation join to assignment table reduces to 6%; residual risk noted; run scenario bounds assuming missing skewed entirely to control/treatment. - Contingency paths - If TTFQ timestamps inconsistent, compute from request created_at to earliest quote or first message time; cross-check with server logs. - If hire flags lag, use 3-day hire as leading proxy with historical uplift factor to 7-day (validated from prior months), and show sensitivity range. - If session-based assignment is noisy, switch to request-level intent cohorts and analyze difference-in-differences vs. categories not exposed to the new workflow. ## 4) Narrative and Persuasion - Deck storyline - 1-page exec summary - Decision ask; recommended option (A/B/C); headline metrics vs. thresholds; confidence level; risks and mitigations; rollout + rollback plan. - Context and method - What changed (new quoting workflow); cohorts, timeframe, unit of analysis; known data gaps and mitigations. - Metric deep-dives (speed, quality, volume) - Overall and key segments (top categories, new vs returning pros, peak vs off-peak hours, top geos). - Show distributions (e.g., TTFQ median and P90); include 90% CIs. - Trade-off analysis - Efficiency frontier: speed gains vs. hire rate/quality impacts. - Sensitivity to missingness/seasonality; scenario bounds (best/base/worst). - Recommendation with confidence intervals - Example framing: "Expand to 50% in top-5 categories; hold in long-tail; revisit in 2 weeks when lagging outcomes mature; expected TTFQ −12% [−15%, −8%], hire rate −0.2pp [−0.6pp, +0.1pp]; guardrails within limits." - Implementation plan - Owners, milestones, monitoring, rollback triggers. - Confidence intervals (quick methods) - Proportions (e.g., hire rate p): 90% CI via Wald or better, Wilson: - CI(p) ≈ p ± z_0.95 * sqrt(p(1−p)/n), with z_0.95 ≈ 1.645. - Difference in proportions Δ = p_t − p_c: CI(Δ) ≈ Δ ± z * sqrt(p_t(1−p_t)/n_t + p_c(1−p_c)/n_c). - Medians (TTFQ): bootstrap by request-level resampling; report median delta CI. - Small numeric example: If hire rate increases from 12.0% (n=10,000) to 12.5% (n=10,000), Δ=+0.5pp. SE≈sqrt(.125*.875/10k + .12*.88/10k) ≈ 0.0046 → 90% CI ≈ 0.5pp ± 0.76pp → [−0.26pp, +1.26pp]. - Pre-wire strategy - 1:1 pre-reads with PM (speed), Ops (quality), Sales (volume) by hours 36–60. - Ask for explicit buy-in on thresholds and rollout phasing; capture dissent and propose guardrails. - Handling live pushback - Prepare alt scenarios slides: - Option B (phased expansion) with narrower guardrails. - Option C (hold) with a dated plan to close gaps. - Use "If true, then do" logic: If hire rate CI lower bound < −1.0pp, then phase by categories with strongest speed gains and stable quality; monitor daily. ## 5) Decision and Follow-Through - Go/no-go criteria (example; customize to signed brief) - Go (partial): TTFQ median improves ≥ 10% with 90% CI lower bound ≥ 5%; 7-day hire rate CI lower bound ≥ −1.0pp; support tickets ≤ +10%; spam ≤ +5%; no category shows both hire decline >1pp and support increase >10%. - Go (full): Same as above across top-10 categories; no critical guardrail breached; data gaps <10% residual missingness on variant. - No-go: Any critical guardrail breached or speed gains <5% (lower bound), or substantial unresolved data issues that could flip sign. - Owner assignments - DRI for decision and measurement: You - Rollout: PM (schedule), Eng (feature flag/rollout tooling), Ops (pro comms/quality ops), Sales (messaging), Analytics IC (monitoring) - 2-week post-decision plan - Monitoring (daily for week 1; 3x per week for week 2) - Dashboards: TTFQ, hire rate, support tickets, spam flags, GMV per request, pro retention proxy. - Alerting thresholds: e.g., hire rate −1.0pp day-over-day for 2 consecutive days triggers review. - Validation tasks - Replace proxies (e.g., support rate) with lagging true outcomes (ratings, refunds) as they mature. - Category/geo deep-dive for outliers; tighten eligibility if needed. - Incident rollback plan - Keep feature-flagged; one-click rollback to previous workflow. - Runbook: who flips (Eng), who communicates (PM/Ops), who postsmortems (You + A), SLA: rollback within 30 minutes of breach. ## 6) Leadership Behaviors Under Time Pressure - Model calm and focus - Publish the plan and thresholds early; remind the team we need decision-quality, not perfection; use a cut list for non-essentials. - Timebox debates; decide with the best available data and documented assumptions. - Protect IC focus - Block calendar for heads-down windows; route stakeholder questions to you; batch feedback at checkpoints; freeze metric definitions after hour 12 unless a material error. - Escalate without churn - Escalate only with clear options and implications ("We can patch missing variant today with 6% residual missingness; accept medium confidence or delay 24h for a cleaner backfill"). - Document decisions in the channel; avoid revisiting closed topics unless new data changes the expected outcome. # Practical Analytics/Engineering Details - Metric computation tips - TTFQ: For each request, TTFQ = MIN(quote_at) − request_created_at; compute median and percentiles by variant and segment. - Hire rate: hires_7d / requests; ensure consistent lookback window; censoring: exclude last 7 days from window or use Kaplan–Meier for partial. - Support rate: support_tickets / requests; tag by request_id to avoid double counting. - Sampling/QA checks - Randomly sample 100 requests; manually verify TTFQ and quote counts across sources; check 5–10 edge cases (no quotes, many quotes, long TTFQ). - Compare to historical baselines; if deltas are implausible (e.g., TTFQ –70%), investigate timestamp timezone or duplication. - Sensitivity/robustness - Stratify by category volume quartiles; if long-tail drives negative effects, propose phased rollout. - Difference-in-differences: compare exposed vs. unexposed categories over the same time window to control for seasonality. - Communication guardrails - Always pair a metric with its CI and a plain-English interpretation. - Call out known biases and show scenario bounds: best/base/worst. This approach gets you fast alignment, credible decision-quality analytics despite data ambiguity, a persuasive story tuned to each stakeholder, and a safe operational plan with monitoring and rollback.