Analyze failed gym-collab credit card launch

# Structured, Teaching-Oriented Solution ## 0) Define success and set the stage - Primary metrics - Incremental enrollments (vs. counterfactual) and incremental gym spend per eligible cardholder within 30/60/90 days. - Unit economics: Incremental margin LTV − incentive cost − media/ops cost ≥ 0. - Secondary metrics - CTR/CVR through the funnel; approval/activation if new-card applications were part of the flow; time-to-first-gym-transaction; retention/churn of gym spend. - Guardrails - Don’t rely on raw lifts; always anchor on a credible counterfactual with checks for bias. ## 1) Establish the counterfactual (incrementality) Prefer randomized designs; if unavailable, use quasi-experimental methods. - If you had an RCT/holdout: - ITT (intention-to-treat): mean outcome difference between exposed and holdout groups. - TOT (treatment-on-the-treated): adjust ITT by take-up rate. - If no RCT, use quasi-experimental: - Difference-in-differences (DiD) across treated vs. matched control geos/branches/customers: DID = (Y_treat,post − Y_treat,pre) − (Y_ctrl,post − Y_ctrl,pre) - Ensure parallel pre-trends via event-study plots and placebo tests. - Matching/PSM: match on pre-period gym spend, tenure, demographics, credit risk, channel, seasonality exposure. - Synthetic control (if one/few treated units) to construct a weighted counterfactual from controls. - Negative controls: categories unlikely affected by the partnership (e.g., grocery spend) to detect spurious shifts. - Attribution hygiene - Validate exposure logs, MCC classification, partner referral tags, and that measurement pipelines were live at launch. Small numeric example: Suppose monthly gym spend per eligible cardholder moved from $9.0 to $11.0 in treated geos, and from $9.0 to $10.5 in controls over the same period. DID = (11.0−9.0) − (10.5−9.0) = $0.5 incremental per customer-month. ## 2) Data to collect - Exposure and funnel - Impressions, clicks, landings, enrollments (timestamped), approvals/activations (if applicable), first transaction, repeat transactions. - Channel/creative variant, device, geo, audience/eligibility flags. - Customer and history - Pre-period gym spend, overall spend, tenure, product type, credit bands, demographics (as permitted), opt-in status. - Partner operations - Store list and hours, POS capabilities, BIN recognition settings, coupon/offer code acceptance, training completion logs, signage audits, lead capture counts, referral IDs. - Offer and pricing - Incentive schedule (value, thresholds, caps), redemption/crediting latency, breakage, unit costs. - Marketing/media - Spend by channel/geo/day, frequency/caps, competing campaigns. - External factors - Seasonality (e.g., Jan/Sept peaks), local events, weather, macro shocks, competitor promotions. - Quality and integrity - MCC mappings, de-duped customer IDs, bot filters, logging gaps. ## 3) Analyses to run A. Measurement/QC - Reconcile exposures → enrollments → first gym transaction counts across systems. - Check MCC tagging for partner locations; spot-check with test swipes. - Latency checks: time from purchase to statement credit; broken credits depress observed value. B. Funnel and friction - Compute stepwise conversion: exposure → click (CTR), click → landing, landing → enroll (CVR), enroll → first gym swipe, swipe → repeat. - Segment by channel, creative, device, geo, and customer type (existing gym spender vs. non-spender). - Identify the biggest drop-off and investigate root causes (UX, eligibility, approval, redemption friction). C. Cohorts and retention - 0/30/60/90-day spend curves for enrollment cohorts; time-to-first-transaction survival curves; churn hazard after first month. - Compare to historical gym spenders vs. new-to-category customers. D. Incrementality and heterogeneity - DiD by geo/branch and by customer segment; cluster-robust errors at geo level. - Event-study plots to test parallel trends and dynamics. - Uplift modeling/CATE: E[Y|T=1,X] − E[Y|T=0,X] to find who actually benefits (e.g., fitness-interested, urban cores). E. Diagnose specific failure modes - Targeting failure - Exposure skewed to low-propensity audiences? Check pre-period gym spend propensity vs. exposure rates. - Low incremental lift in low-propensity segments indicates wasted reach. - Positioning/creative - CTR and on-site CVR by creative; run logistic regression controlling for channel to isolate message effects. - Survey/lightweight UX tests to detect misunderstanding of the benefit or hidden eligibility. - Incentive design - Is the offer salient and simple? Thresholds/caps too high? Delayed credit reduces perceived value? - Compute elasticity: enrollment lift per $ of expected value; compare to benchmarks. - Unit economics: Incremental margin × retention − incentive cost − media. - Partner operations - Redemption failures at POS, staff unfamiliarity, missing signage → high enroll but low first-swipe. - Compare store clusters with high vs. low training completion. - Seasonality/macros - Launch timing vs. gym intent cycles; add month fixed effects. - Placebo in non-gym categories to ensure effects are category-specific. F. Sensitivity and robustness - Alternative control sets, varying pre-period windows, excluding overlapping campaigns. - Synthetic control as a cross-check. ## 4) Likely synthesis patterns (examples) - Big drop from enroll → first swipe with partner stores showing low offer recognition → partner ops is primary driver. - Good CTR but poor landing → enroll CVR → positioning/UX issue. - Decent enrollments but low incrementality after DiD → high cannibalization (existing gym spenders switching cards, not increasing total spend). - Flat lifts in summer/off-peak months → seasonality; moving spend to Jan/Sept could materially improve ROI. - Weak or negative CATE in suburban low-density areas → mistargeting; concentrate on urban cores and existing gym spenders or New Year’s joiners. ## 5) Immediate mitigations (2–6 weeks) - Fix the leakiest funnel step - If POS redemption fails: enable auto-statement credit using MCC detection; refresh BIN tables; hotfix training with job aids; mystery-shop and audit top-50 locations. - If landing CVR is low: simplify copy, remove hidden eligibility, show "you’ll get $X credited within Y days" with examples; reduce clicks to enroll. - Targeting tightening - Suppress low-propensity audiences; prioritize customers with recent fitness app/device signals or prior gym spend; retarget cart-abandoners. - Incentive tweaks - Increase immediate perceived value (e.g., first-month credit, lower threshold) while capping total cost; shorten credit latency. - Media/geo reallocation - Shift budget to high-lift geos/segments; pause underperforming channels/creatives; ramp near partner flagship locations. - Measurement and guardrails - Stand up geo holdouts; implement unique referral codes; daily QA on crediting pipeline. ## 6) Longer-term pivots (6–24 weeks) - Redesign mechanics - From complex tiered thresholds to simple, auto-applied statement credit; or a limited-time high-salience New Year’s offer. - Performance-based economics with partner (rev share per incremental membership or per incremental spend). - Product integration - Add "Fitness" as a rotating or always-on accelerated category beyond a single partner to reduce concentration risk. - Bundle with digital fitness apps/wearables; tie benefits to membership autopay for persistence. - Targeting science - Always-on uplift modeling; lifecycle triggers around January/September and after life events. - Experimentation rigor - Pre-registered RCTs or geo-lift tests with power calculations; persistent holdouts for true incrementality. - Operational excellence with partner - Quarterly business reviews, standardized training, launch playbooks, signage SLAs, POS certification before mass launch. ## 7) Communication plans that preserve trust A) Executives - Structure: 1-page summary + appendix - What happened: Under target on enrollments (−X%) and incremental spend (−Y%). - Counterfactual and validity: Method (e.g., DiD with matched controls), pre-trend checks, sensitivity. - Root-cause attribution with effect sizes: e.g., partner ops issues explain ~60% of gap; incentive salience ~25%; seasonality ~15%. - Financial impact: Variance to plan, unit economics, risk to annual goals. - Action plan: Immediate fixes (2–6 weeks), pivots (6–24 weeks), owners, timelines, success criteria, and decision asks (budget reallocation, product changes). - Lessons learned: Launch timing, holdout design, QA gates. - Tone: Candid, data-first, solution-oriented; show learning and a path to ROI-positive relaunch. B) Partner - Acknowledge shared goals and early results; appreciate their investment. - Share a simplified, transparent readout focused on joint fixes (avoid blame): - What worked (e.g., high awareness near flagship locations). - Where we stumbled (e.g., POS redemption confusion in 40% of stores; credit latency messaging). - Joint action plan: Staff refresher training, signage refresh, auto-credit integration, co-branded New Year push; clear KPIs and check-ins. - Propose performance-based adjustments (reduce their perceived risk) and a quick-win pilot in top-performing geos before scaling. - Tone: Collaborative, specific, and time-bound; emphasize that the bank assumes co-ownership of issues and is committing resources. ## 8) Example quantitative wrap-up (illustrative) - Target: 50k enrollments; actual: 28k (−44%). - Incremental gym spend (DiD): +$0.50 per eligible customer-month; plan: +$1.50. - Biggest gap: enroll → first swipe CVR 64% vs. plan 85%, concentrated in stores lacking updated BIN tables. - Fix forecast: If ops fixes restore CVR to 80% and incentive tweak lifts enrollments +25%, modeled ROI moves from −12% to +8% at current media spend; +15% if we shift 40% budget to Jan/Sept windows. ## 9) Common pitfalls and guardrails - Confounding from overlapping campaigns; isolate exposure or exclude periods. - MCC misclassification leading to undercounted spend; validate with transaction sampling. - Cannibalization: Existing gym spend merely shifts to our card; measure net incremental category spend, not just partner spend. - Parallel trends violations; include event-study plots and alternative control groups. - Overfitting uplift models; validate out-of-sample and monitor stability. ## 10) What good looks like in the follow-up - A relaunch plan with powered geo holdouts, daily QA on redemption/crediting, pre-registered success criteria, and a joint QA checklist with the partner. - A 30/60/90-day readout cadence that transparently tracks incremental outcomes and unit economics, with kill-switch thresholds if incrementality is not achieved. This approach cleanly separates measurement (counterfactual), diagnosis (which link in the chain broke and why), actions (near-term vs. strategic), and communication (preserving executive and partner trust), which is essential for turning a miss into a durable learning and a path back to ROI-positive growth.