Welch's t-Test p-Value for a Routing Experiment on Time to Pickup, and Its Validity

This two-part task comes from a timed online assessment for a data science role at a ride service. A new routing algorithm intended to reduce time to pickup (TTP, the time until a vehicle reaches the rider) was tested in San Francisco in July 2024. Users were assigned to variants. You have two pandas DataFrames: - `df_users` with columns `user_id`, `variant`; - `df_rides` with columns `ride_id`, `ride_date`, `city`, `time_to_pickup`. Assume `df_rides` also carries the rider's `user_id`, which the join to `df_users` requires, and that San Francisco appears in `city` under a single label such as `"SF"`; confirm both before relying on them. ### Clarifying Questions - What are the variant labels, and is every user assigned to exactly one variant? - Should the test be two-sided, or one-sided in the direction of lower TTP? - What significance level should be used? - How should rides with a missing `time_to_pickup` be treated? - Is `ride_date` a date or a timestamp, and in which time zone is "July" defined? ### Part 1 — Compute the p-value Write a Python function that filters the rides to San Francisco in July 2024, joins each ride to its rider's variant, runs Welch's t-test comparing `time_to_pickup` between the two variants, and returns the p-value. ```hint Check the join Before testing, make sure the join neither drops rides silently nor duplicates them. ``` #### What This Part Should Cover - Correct filtering by city and by the full month, with date parsing and an exclusive end boundary - A join that neither duplicates nor silently drops rides, with a check for users in more than one variant - Welch's t-test (unequal variances) with the right inputs, and handling of missing values - What the test statistic and its degrees of freedom are ### Part 2 — Experiment validity Now interpret the result. Is it statistically significant at your chosen level? Can the team conclude that the new routing algorithm reduces TTP? If not, design a better experiment. ```hint What is independent here? Compare the unit that was randomized with the unit your test treated as independent observations, and ask whether the variants can affect each other. ``` #### What This Part Should Cover - Reading the p-value correctly, together with the direction and size of the effect - Threats to validity: mismatch between the randomization unit and the analysis unit, interference between variants through the shared fleet, and limits of one city and one month - A better design (for example switchback or geographic randomization), with its metrics, duration, power and analysis plan ### What a Strong Answer Covers - Code that is correct on edge cases, not only on a clean example - Awareness that statistical significance is not the same as a valid causal conclusion - A concrete, feasible redesign rather than a generic "run a bigger test" - Guardrail metrics beyond TTP ### Follow-up Questions - TTP is heavily right-skewed. How does that affect the t-test, and what would you use instead or in addition? - How would you compute a confidence interval for the effect when rides are clustered within users? - The switchback test shows a smaller effect than the user-level test. Which do you trust, and why? - How long should the redesigned test run, and what determines that?