Experimentally evaluate jogging-route recommendations

Quick Overview

This question evaluates competency in experimental design, metric selection and definition, statistical power/MDE calculation, contamination prevention, and operational monitoring for route recommendation features, targeted at Analytics & Experimentation for a Data Scientist role.

Experimentally evaluate jogging-route recommendations

Company: Google

Role: Data Scientist

Category: Analytics & Experimentation

Difficulty: hard

Interview Round: Technical Screen

Google Maps plans to recommend optimal jogging routes. Design the evaluation: pick a single primary metric (e.g., jog completion rate) and define secondary metrics (route adherence, repeat jog within 7 days, safety incidents, time variance vs baseline route, battery drain); specify eligibility and triggering logic—expose only when the user has opted into fitness mode, location accuracy <30m, speed 1–4 m/s, and a route suggestion is shown; define experimental unit and assignment, prevent contamination (multiple devices, social sharing), and list guardrails (crash rate, navigation reroutes, ETA accuracy); compute MDE and sample size for a baseline completion rate of 40% with an expected +3pp lift, 80% power, α=0.05, stating formulas and any clustering/novelty adjustments; propose a ramp plan, geographic stratification, novelty decay checks, and a rollback criterion.

Quick Answer: This question evaluates competency in experimental design, metric selection and definition, statistical power/MDE calculation, contamination prevention, and operational monitoring for route recommendation features, targeted at Analytics & Experimentation for a Data Scientist role.

|Home/Analytics & Experimentation/Google
Google logo
Google
Oct 13, 2025, 9:49 PM
hardData ScientistTechnical ScreenAnalytics & Experimentation
11
0

Design an Evaluation for Jogging Route Recommendations in Maps

Objective

Design an A/B test and evaluation framework for recommending optimal jogging routes. Your design should include:

  1. Primary metric selection (choose one) and precise definition.
  2. Secondary metrics and how to compute them (e.g., route adherence, repeat jogs in 7 days, safety incidents, time variance vs a baseline, battery drain).
  3. Eligibility and triggering logic for counting an exposure.
  4. Experimental unit, randomization/assignment, and contamination prevention (multi-device, social sharing).
  5. Guardrail metrics and thresholds (e.g., app crash rate, navigation reroutes, ETA accuracy).
  6. Power/MDE analysis: For a baseline completion rate of 40% and an expected +3 percentage point lift, compute the minimum sample size with 80% power and α = 0.05. State your formulas and note any clustering/novelty adjustments.
  7. A ramp plan, geographic stratification approach, novelty-decay checks, and a rollback criterion.

Provided Constraints

  • Eligibility/triggering: Expose the suggestion only when the user has opted into fitness mode, location accuracy < 30 m, speed in [1, 4] m/s, and a route suggestion is actually shown.
  • Consider how to prevent contamination across devices and via social sharing.
Loading comments...