Test Whether a Routing Experiment Reduced Pickup Time

An A/B test in San Francisco during July 2024 evaluates a new routing algorithm intended to reduce time to pickup. You have: ```text df_users(user_id, variant) df_rides(ride_id, user_id, ride_date, city, time_to_pickup) ``` ### Constraints & Assumptions - Use Python with pandas and a standard statistical library. - Filter to San Francisco rides from July 1 through July 31, 2024. - The requested calculation is a two-sided Welch's t-test comparing pickup time between variants. - State the independence limitation if users can contribute multiple rides. ### Clarifying Questions to Ask - Was treatment assigned at the user level, and is assignment unique and stable? - Which variant is control, and are there missing or duplicate assignments? - Is `ride_date` timezone aware, and are invalid or canceled rides already excluded? - What significance level and minimum practically important effect were chosen before analysis? ### Part 1: Compute the P-Value Filter the rides, join users to their assigned variants, run Welch's t-test on `time_to_pickup`, and return the p-value. Explain the important data-quality checks. #### What This Part Should Cover - Correct half-open date filtering, validated join cardinality, null handling, and `equal_var=False`. - Group counts and means alongside the p-value. - Recognition that ride-level observations may be clustered within users. ### Part 2: Interpret and Improve the Experiment Explain when the result is statistically significant and whether significance alone proves the new algorithm is better. Propose a stronger experiment and analysis. #### What This Part Should Cover - Comparison with a preselected alpha, an effect estimate, confidence interval, and practical significance. - Checks for randomization integrity, sample-ratio mismatch, interference, attrition, and metric quality. - An analysis aligned with user-level assignment or clustered repeated rides. - Primary metrics, guardrails, duration, segmentation, and a launch rule defined in advance. ### What a Strong Answer Covers - Correct code plus careful alignment between randomization unit, analysis unit, and causal claim. - A distinction between a small p-value, effect size, business value, and proof of superiority. - A design that can estimate impact without conditioning only on post-treatment ride activity when conversion may change. ### Follow-up Questions 1. What bias can arise if the algorithm changes whether users complete a ride? 2. How would you form a user-level metric for users with no rides? 3. When would a one-sided test be defensible? 4. How would you handle treatment spillovers through a shared vehicle fleet?