Design experiment on culture memo emphasis
Overview
This question evaluates competency in experimental design and statistical analysis for product metrics, covering hypothesis specification, randomization and exposure definition, metric selection with guardrails, invariant and SRM checks, MDE and sample-size calculations, seasonality adjustments, CUPED or covariate adjustment, heterogeneity analyses, and pre-registered decision criteria. Commonly asked in Analytics & Experimentation interviews, it tests practical application of causal inference and A/B testing principles within the Data Scientist domain, combining conceptual statistical understanding with hands-on experimental planning.
A/B Test Design: Prominently Feature the Culture Memo on Job Description Pages
You are designing an experiment to evaluate whether prominently featuring the Culture Memo on the job description (JD) page increases completed applications.
Specify the following:
- Hypothesis and expected direction
- Unit of randomization and exposure definition
- Primary metric and at least two guardrail metrics
- Invariant checks and Sample Ratio Mismatch (SRM) detection
- Minimum Detectable Effect (MDE) and sample size calculation (state assumptions and show math)
- Test duration and how you will account for weekly/seasonal patterns
- Analysis plan, including use of CUPED or pre-exposure covariates
- Heterogeneity breakdowns you will pre-specify
- Pre-registered decision criteria for ship/no-ship
Solution
Login required