Assess Routing Experiment Validity
Company: Waymo
Role: Data Scientist
Category: Statistics & Math
Difficulty: medium
Interview Round: Online Assessment
A ride-hailing team runs an A/B test in San Francisco in July 2024 for a new routing algorithm intended to reduce time to pickup, abbreviated TTP.
You are given two pandas DataFrames:
`df_users`:
- `user_id`: unique user identifier
- `variant`: experiment assignment, either `control` or `treatment`
`df_rides`:
- `ride_id`: unique ride identifier
- `user_id`: user identifier
- `ride_date`: ride timestamp or date
- `city`: ride city
- `time_to_pickup`: numeric TTP in minutes
Tasks:
1. Write Python code to filter to San Francisco rides in July 2024, join ride records to experiment assignments, run a Welch two-sample t-test comparing treatment versus control TTP, and return the p-value.
2. Given the returned p-value, how would you decide whether the result is statistically significant?
3. Is a statistically significant p-value conclusive proof that the new routing algorithm is better? If not, explain the main threats to validity.
4. Propose a stronger experiment design and analysis plan for this routing algorithm.
Overview: This question evaluates experimental analysis and statistical hypothesis testing skills, including data preparation and joining, interpretation of p-values, and understanding of causal inference and threats to validity in A/B tests.
Community answers
Answer by practicial
Filter the data and join using Pandas
Answer by juanipnc6
1)
data = df_rides[(df_rides["ride_date"].dt.year == 2024) &
(df_rides["ride_date"].dt.month == 7) &
(df_rides["city"] == "San Francisco") ].merge(df_users,
on = "user_id", how = "inner")
A = data[data["variant"] == "control"]["time_to_pickup"]
B = data[data["variant"] == "treatment"]["time_to_pickup"]
from scipy import stats
t_val, p_val = stats.ttest_ind(A,B, equal_var = False) # False to get Welch test
2)
reject H0: no difference if p-val < alpha significance level, alpha = 0.05
3)
It is not, we can get the significant effect to evaluate the practical significance:
(B.mean()-A.mean()) / sqrt(pooled_var)
pooled_var = [ ( (A.shape[0]-1)(A.var()) + ( B.shape[0] -1 )(B.var()) ) / ( A.shape[0]+B.shape[0]-2) ]
and also, and most important, we are assuming independent samples but may not be met because one group can influence the other one, we have multiple measures for each user which breaks iid, and drivers can learn during the experiment which may change the outacome after some weeks of using the new algorithm
4)
We can use a geo randomization to divide the city in areas where the drivers get the new algorithm and areas weher they don't. The areas need to be comparable in terms of streets, habitants, limit speed, terrain conditions etx.
Another idea could be switching the whole city between treatment and control over time so we have samples of the whole city under one condition or another, we can have samples representing the same time conditions like weekdays, peak times, wee
Answer by jzt5132
part 1
from scipy import stats
df_rides_sf = df_rides[(df_rides.city == "San Francisco") & (df.ride_date <= '2024-07-31') & (df.ride_date >= '2024-07-01')]
df_merged = df_users.merge(df_rides_sf, on='user_id', how='inner')
control_series = df_merged[df_merged.variant == 'control']['time_to_pickup']
treatment_series = df_merged[df_merged.variant == 'treatment']['time_to_pickup']
_, p_value = stats.ttest_ind(control_group, treatment_group, equal_var=False)