Align with PM on ranking goals

# Overview Goal: Ship a safe, measurable MVP quickly; instrument properly; protect customers and the business; learn fast; then iterate to ML once we have evidence and approvals. I’ll structure the plan around discovery (clarify goals and constraints), delivery (MVP), and derisked evolution (V2 ML). --- ## 1) Clarifying Questions Ask targeted questions to surface the real objective, constraints, and measurement plan. Phrase each with why it matters. - Objective and success - What is the primary goal? Examples: - Acquisition (new account sign-ups), cross-sell/upsell (adoption of additional products), or LTV/profit (risk-adjusted revenue). - What counts as a conversion? Define funnel events and window: - Click → product detail → application start → qualified submission → approval → activation within 30/60/90 days. - What is the minimum detectable effect (MDE) and acceptable time-to-detect? If unknown, I’ll propose an MDE based on traffic and risk appetite. - Guardrail metrics: page performance (TTFB, LCP), complaint/opt-out rate, customer support contacts, fairness/exposure diversity, eligibility errors, and any regulatory guardrails. - Scope and eligibility - Which products are in-scope initially? Any out-of-scope due to inventory/capacity or readiness? - Eligibility rules to avoid showing ineligible products (e.g., geo/state, age, income, existing relationships, KYC/KYB status). Are we ranking for logged-in, logged-out, or both? - Required disclosures or specific ordering constraints (e.g., required display of specific products or disclaimers)? - Users and segmentation - Key segments: new vs. returning, current customers vs. prospects, device/geo. Any segments we must exclude or treat differently? - Frequency caps or rotation needs to avoid over-exposure fatigue? - Data and tech - Available real-time signals at ranking time (consented only): device, referrer, session context, known user features? Any PII restrictions? - Latency budget for ranking (e.g., p95 < 150ms). Fallback behavior if service fails. - Current tracking: do we log impressions with rank position, exposure time, and downstream conversions tied to a user/session key? - Compliance/legal - Permissible features for personalization. Any prohibited attributes or proxies (e.g., protected-class proxies)? Required documentation/audit trail? - Review/approval process and SLAs: legal, compliance, risk, accessibility, brand. - Experimentation - Can we run an A/B test? Unit of randomization (user vs. session), expected traffic, seasonality constraints. - Any user segments that must be in permanent holdout or must not be randomized? Small numeric checkpoint (MDE sizing): If baseline application-start rate is 5.0% and we want to detect a +10% relative lift (to 5.5%) at 80% power, 5% alpha, a two-proportion test typically needs on the order of ~60k–80k sessions per arm. If we only have 10k/week, we should lower MDE expectations, extend duration, or use variance-reduction techniques. --- ## 2) Negotiation Plan: MVP vs. V2 ML and 30/60/90 Roadmap - MVP (ship in weeks): Rules-based ranking + robust logging - Ranking logic (example): - Filter out ineligible products by hard rules (geo, required disclosures, inventory/capacity gates). - If logged-in: prioritize products the customer does not already have and is likely eligible for (simple heuristics like “not-owned first”, then by business priority). - If logged-out: show a diversified slate: top 1–2 broadly appealing products, then rotate remaining to ensure diversity (e.g., at most 60% exposure to any single product per day). - Instrumentation and data quality - Log events: page_view (session_id), product_impression (product_id, rank, slot, timestamp), product_click, funnel events (application_start, qualified, approval), and user_id (hashed) when available. - Include feature snapshot used for ranking (versioned) to support auditability. - Experiment design - A/B test vs. current ordering. Primary metric: qualified application starts per 1,000 sessions. Guardrails: page load metrics, complaint rate, exposure diversity (e.g., Herfindahl index), eligibility error rate. - Rollout plan: 10% canary → 50% if guardrails green → 100% once significant. - Governance - Pre-launch checklist with legal/compliance sign-off on logic, features, and disclosures. Define rollback switch and on-call. - V2 ML (post-MVP, gated by evidence and approval) - Predictive model objective: maximize risk-adjusted conversion or LTV. Start interpretable (regularized logistic regression or gradient boosted trees with monotonic constraints if needed) and calibrated. - Constraints and fairness - Enforce policy constraints at serve time (filtering) and fairness/exposure constraints (min exposure for long-tail, max cap for top product, demographic parity where applicable and permissible). - Infra - Feature store with only permitted, consented features. Real-time scoring endpoint with p95 < latency target and caching/fallback to rules. - Validation - Offline evaluation with pre-registered analysis, then A/B test. Document model cards, feature importance, and SHAP summaries where appropriate. - 30/60/90-day roadmap - Days 0–30 (MVP) - Finalize goals/metrics and guardrails; implement rules-based ranking; instrument logging; ship canary; start A/B. Create dashboards and QA data quality. - Days 31–60 (Learn + extend) - Analyze results; refine heuristics; add simple personalization by segment; build offline ML prototype and fairness/constraint evaluations; prepare compliance package for ML. - Days 61–90 (ML pilot) - Productionize ML behind a flag; run controlled A/B with strict guardrails; monitor latency, fairness, and audit logs; decide on scale-up or revert. --- ## 3) Communication Artifacts and Handling Pushback - PRD addendum (metrics + constraints) - Problem statement, in-scope/out-of-scope products, user stories, primary metric, guardrails, data sources, latency budget, experiment plan, and explicit assumptions. - Decision log - Date, decision, options considered, rationale, owners, and links to evidence. Example: “Chose rules-based MVP due to low traffic and missing legal sign-off for ML personalization.” - Risk register - Risk, likelihood/impact, owner, mitigation, trigger, contingency. Include: compliance breach, biased exposure, data leakage, infra failure, metric misalignment, seasonality confounding. - Experiment design doc - Hypotheses, unit of randomization, sample-size/power, duration, pre-registered analysis plan, segmentation, and stop/go rules. - Data contract + event schema - Field names, types, IDs, PII handling, retention, and versioning. Include impression/click/application linkage and feature snapshots. - Runbook - Rollout/ramp, monitoring dashboards, alert thresholds, rollback steps, and contacts. Handling pushback when PM’s preferred metric conflicts with risk/compliance - Example: PM wants to optimize CTR; risk prefers qualified applications and fairness guardrails. - Use a metric hierarchy: primary outcome (qualified applications per 1,000 sessions), optimization proxy (CTR) allowed only if guardrails are met, with documented thresholds (e.g., complaint rate ≤ X per 10k sessions; eligibility error rate < 0.5%; exposure cap per product ≤ 60%). - Quantify tradeoffs: show scenarios where CTR gains could increase unqualified volume or harm fairness. Provide a side-by-side with expected outcomes and risks. - Escalate respectfully to the metric council/legal/compliance when irreconcilable; propose a dual-metric approach (optimize proxy subject to hard constraints). Keep the decision in the log with sign-offs. --- ## 4) Decision Framework for Trade-offs and No-Go Criteria - Trade-off framework (weighted decisioning) - Dimensions: speed-to-ship, expected impact (incremental qualified conversions or risk-adjusted revenue), accuracy (offline/online lift), fairness/diversity (exposure distribution), compliance risk, and operational complexity. - Use a simple scoring (e.g., ICE: Impact, Confidence, Effort) with explicit constraints: if any hard constraints fail, the option is ineligible regardless of score. - Guardrails and monitoring - Exposure diversity: cap top product exposure (e.g., ≤ 60% per day) and minimum exposure for others (e.g., ≥ 5% where applicable). - Page performance: LCP within target; fail → rollback. - Eligibility errors: impressions of ineligible products < threshold. - Fairness: agreed metric (e.g., exposure parity across permissible cohorts/segments or category diversity) within bounds; monitor drift. - Compliance: prohibited features absent; audit logs present; disclosures rendered. - Explicit no-go criteria (any one triggers stop/rollback) - Missing legal/compliance approval for the current logic/features. - Guardrail breach sustained beyond X hours (e.g., complaint rate > threshold, eligibility error > 0.5%, LCP degradation > 20%). - Evidence of discriminatory impact or proxy discrimination beyond agreed thresholds. - Data quality regressions (missing impressions/clicks > 2% vs. baseline; identity mis-link rate > threshold). - Severe infra issues (p95 latency over budget with no viable fallback). --- ## 5) Blameless Postmortem and Maintaining Trust If V1 underperforms or causes issues, run a structured, blameless review: - Process - Assemble timeline: requirements → launch → metrics by day → incidents. Include experiment logs and decision log excerpts. - Facts only first: what changed, when, and measured impacts (primary and guardrails). - Root-cause analysis: 5 Whys across people/process/tech, avoiding blame. - Findings: validated hypotheses (e.g., heuristic favored low-intent clicks, seasonality confounded results, mis-specified eligibility filter). - Actions: specific owners and dates. Examples: tighten eligibility filter, add variance reduction (CUPED), adjust exposure caps, fix logging gap, update PRD assumptions. - Communication: share with PM, design, eng, compliance. Publish a short summary to stakeholders. - Maintain trust with PM - Be transparent and fast: same day data snapshot, 48-hour deep dive, and a revised plan. - Extract wins: “We reduced ineligible impressions by 40%” even if conversion didn’t lift; use this to justify next iteration. - Reset expectations with data: “At 10k sessions/week, detecting a 0.3 pp lift needs 6–8 weeks; propose an interim proxy with guardrails.” - Co-own outcomes: “We agreed on CTR as a proxy; next, we’ll switch to qualified starts as the primary metric and add a minimum-exposure constraint.” Small numeric example (learning from miss): - Expectation: +10% relative lift from 5.0% to 5.5% application starts. Observed: 5.1% (Δ = +0.1 pp; not significant at 80% power over 2 weeks, n=20k/arm). - Hypotheses: over-exposed one product (80% of impressions) causing low intent. Action: cap exposure at 60%, introduce segment-based ranking, re-run for 4 weeks to gain power. --- ## Appendix: Practical Details - Event schema (minimum) - impression_id, session_id, user_id_hashed, timestamp, page_id, product_id, rank, slot, features_version, features_snapshot, treatment_group. - click/application events reference impression_id to avoid attribution ambiguity. - Sample-size formula (two-proportion, ballpark) - For baseline p1 and target p2, per-arm n ≈ 2 × (Z_{1-α/2}√(2p̄(1−p̄)) + Z_{power}√(p1(1−p1)+p2(1−p2)))^2 / (p2−p1)^2, where p̄ = (p1+p2)/2. - Fallbacks - If real-time features unavailable or service latency > budget, revert to static diversified list with cached rankings. This plan aligns stakeholders on goals and guardrails, ships an auditable MVP quickly, and creates a measured path to ML with explicit safety checks and learning loops.