Analytics & Experimentation: Metric Design and Validation

Context

You are a Data Scientist working on analytics and experimentation. You are given business goals (not explicit metrics) and must define robust, time-bounded, non-vanity metrics and guardrails, then justify metric choices and validation plans.

Tasks

1. For each goal below, propose:
   • One primary metric (concrete, time-bounded, hard to game).
   • 2–3 guardrail metrics (also concrete, time-bounded, and resistant to gaming). Goals:
   • Improve Retention
   • Grow Mobile Usage
   • Increase Content Quality
2. Justify when to prefer rates/percentiles over means, and when that choice fails (e.g., freemium or outlier-driven businesses). Include brief examples.
3. For a Q&A platform, define a metric suite that captures "answering quality within 16 hours" without being skewed by late responses. Include at least two complementary percentages that reflect improvement in timeliness (e.g., from 15h to 1h).
4. When asked to optimize a long-term outcome (e.g., LTV), specify:
   • A short-term proxy set.
   • A validation plan to prove the proxy predicts the long-term outcome.
   • A backtest plan that could falsify your proxy choice.