Handle sales pressure with analytical integrity

## Overview Goal: Move from a promising but confounded correlation to a low-risk, testable change that protects quarterly targets while generating causal evidence. We’ll propose a brief, quota-neutral, randomized rollout with pre-registered analysis, clear guardrails, and a communication plan. --- ## (a) Diplomatic pushback while maintaining partnership Meeting phrasing: - "I’m excited there’s a strong signal here—higher call activity correlating with higher win rates is exactly the kind of lead indicator we want. To make sure we actually capture those wins rather than just the appearance of them, we need to separate correlation from causation. Some of the effect may be due to deal stage and rep mix—top reps on late-stage deals make more calls and also win more. The fastest, lowest-risk path is a short, controlled rollout that lets us quantify the true lift without jeopardizing the quarter." - "If we run a quick pilot now with clear guardrails, we can make a confident, org-wide decision in a few weeks. I’ll partner with RevOps and managers to keep it quota-neutral and operationally light." Written summary (exec-friendly): - "Finding: Call volume is positively correlated with win rate; current analysis is observational and confounded (deal stage, rep tenure/mix)." - "Risk: A forced 2× increase may shift time away from high-quality activities or over-index on late-stage deals, producing no lift or even harming conversion." - "Proposal: 4–6 week, quota-neutral, stratified randomized pilot across eligible reps with guardrails; primary metric = win rate; safety metrics include cycle length and customer complaints. Decision in ≤ 6 weeks." --- ## (b) Safe, low-cost follow-up: Pilot design Design: Rep-level, stratified randomized rollout (A/B), cluster by rep to avoid spillovers. - Eligibility: Exclude enterprise/strategic accounts, new hires in ramp (<60 days), reps under formal performance plans, and territories with active comp plan changes. Focus on SMB/MM where call volume is operationally flexible. - Randomization: Stratify by rep tenure (new/experienced), region, and current pipeline stage mix to balance confounders. - Treatment: 2× calls per rep day relative to their 4-week baseline, with quality cap (e.g., min connect rate or min avg talk-time) to prevent low-quality dialing. - Control: Business as usual. Quota-neutrality and operational guardrails: - Quota-neutral: No change to quarterly quota for either arm; treatment reps get credit for time spent meeting target call counts. - Activity cap: Max additional calls/day to prevent burnout (e.g., +20–30 calls/day depending on team norm). Allow substitution from low-value tasks to keep time constant. - Customer guardrails: Monitor opt-out/complaint rate, voicemail drop rate, and negative-feedback tags. Pause rules if rates exceed baseline by pre-set thresholds (e.g., +50% over 7-day rolling average). - Rep welfare: Track overtime hours or work-time proxies; set a stop-loss if >10% of treatment reps exceed threshold. Success metrics (pre-registered): - Primary: Opportunity win rate (Closed Won / total closed) on opportunities active at assignment time. - Secondary: Stage progression rates (e.g., Stage 1→2, 2→3), meeting set rate, meetings per opportunity, average talk-time per connect, average deal size (ACV), sales cycle length, pipeline coverage. - Safety: Customer complaints/opt-outs, connect rate degradation, rep attrition signals, NPS/CSAT from post-meeting surveys (if available). Small numeric example and MDE planning: - Baseline win rate p0 = 20%. Detect a +3–5 percentage point lift (δ = 0.03 to 0.05). - Approx sample size per arm for a balanced A/B on proportions: n ≈ (Z_{0.975}+Z_{0.8})^2 * 2*p0*(1-p0) / δ^2 Using Z’s ≈ 1.96 and 0.84: (2.8)^2 * 2*0.2*0.8 / δ^2 ≈ 7.84*0.32/δ^2. - δ = 0.05 → ~1,000 opps total (~500/arm). - δ = 0.03 → ~2,800 opps total (~1,400/arm). - Cluster inflation for rep-level randomization: Design effect ≈ 1 + (m−1)ρ. If average m=30 opps/rep and ICC ρ=0.02, inflate by ~1.58×. Plan sample accordingly or extend duration. If sample is constrained: - Use stage-progression as leading indicators; consider Bayesian sequential monitoring or group-sequential design with alpha-spending. --- ## (c) Timeline, data quality, and reporting SLA Definitions to lock before launch: - Call attempt: Dial event logged to an opportunity’s primary contact. - Connect: Human answered call (exclude IVR/voicemail) with talk-time ≥ 15s. - Conversation: Talk-time ≥ 2 minutes or outcome tagged "qualified conversation." - Call volume metric: Attempts per rep-day; also monitor connects and conversations. - Attribution: Calls within X days of stage are attributed to that opportunity and stage. - Win rate: Closed Won / (Closed Won + Closed Lost) for opportunities active at assignment. Instrumentation checks: - Event completeness: Call logs exist for attempts, connects, talk-time, outcome codes; timestamps with timezone; unique IDs linking call→contact→account→opportunity→rep. - Join keys validated between telephony and CRM. Missingness <2% or understood and MCAR. - Pre-pilot dry run: 1-week shadow logging to confirm metrics stability and latency. - Experiment flags: Persist treatment assignment on rep profile; immutable, timestamped. Timeline: - Week 0–1: Instrumentation validation, finalize analysis plan, train managers, randomize and lock cohorts. - Weeks 2–6: Pilot live. Weekly safety checks only; no directional efficacy reads until MDE or max duration. - T+5 business days after MDE or Week 6: Final analysis and readout. Reporting SLA: - Weekly safety dashboard: guardrails, compliance, data quality. - Final readout doc: effect sizes with CIs, subgroup analysis (pre-specified), decision recommendation. Delivery within 5 business days after pilot end/MDE hit. --- ## (d) Stakeholder alignment and decision criteria Stakeholders and roles: - VP Sales (DRI for go/no-go), RevOps (process and tooling), Frontline Managers (execution and coaching), DS/Analytics (design, analysis), Sales Enablement (training/comms). Operating cadence: - Kickoff: Agree on objectives, metrics, guardrails, MDE, stop/pause rules, comms plan. - Weekly 30-min safety standup (no efficacy reads): Confirm adherence and guardrails. - Steering review at end: Decide scale, iterate, or stop based on pre-defined criteria. Pre-registered decision criteria: - Scale: Primary metric improves by ≥ MDE (e.g., +3–5pp win rate) with 95% CI excluding zero; no material harm on safety metrics; cycle length not materially longer. - Iterate: Directionally positive but CI includes zero; or benefits concentrated in specific segments—consider targeted rollout. - Stop: Null or negative effect; or safety metrics breach thresholds; or significant displacement from high-value activities (e.g., demo completion down >10%). - Documentation: All criteria and results captured in a one-pager and appended to the experiment registry; share in sales all-hands notes. --- ## (e) Handling pressure to publish mid-pilot Principles and phrasing: - "To protect decision quality and avoid false wins, we agreed not to call efficacy early. We can share participation and safety metrics now, and a full, statistically sound result on [date]." - "Early peeks increase false-positive risk materially. With current sample, a 2–3 won deals swing can flip the sign. We’ll stick to the plan to avoid whiplash." - Offer a safe alternative: "Here’s a neutral pulse: adherence by team, connect rates, complaint rates. No efficacy claims until MDE." - If pressure persists: Escalate via steering committee, refer to pre-registered plan, and if necessary provide blinded arm labels (Arm A/B) to share neutral operational data without implying which is treatment. Guardrails on interim comms: - Only report compliance, safety, and data quality. - No win-rate deltas, no p-values/intervals, no segment deep dives that imply direction. --- ## Additional analysis notes and pitfalls - Confounding: Deal stage and rep quality drive both calls and wins; stratified randomization and rep-level clustering address this. - Interference: Avoid cross-arm coaching on call targets; keep territories distinct; monitor spillover. - Quality vs quantity: Include connect rate and avg talk-time to prevent low-value dials. - Multiple testing: Limit subgroup reads to pre-specified dimensions; adjust or present with CIs and a clear caveat. - Missing data: Define handling (e.g., exclude opportunities with missing close outcome; or use ITT at rep level to retain randomization integrity). This plan preserves quarterly goals, de-risks the change, and yields credible evidence to inform a scale decision within 4–6 weeks.