Difference-in-Differences Evaluation and ROI of a State Availability Incentive Program

A booking marketplace ran an incentive program in Massachusetts to increase availability on its platform, paying \$200 per week. Assume availability is the bookable time that providers on the platform offer, that customers book that time, and that the incentive is paid to providers. Pennsylvania, which had no incentive, serves as the comparison state. The data cover 36 weeks: the first 24 weeks are before the launch, and the last 12 weeks are under the program. In the interview, the analysis results were provided in notebooks to interpret. The report does not include their numbers, so this version asks how you would set up, check and read a difference-in-differences analysis of the program. ### Clarifying Questions - What exactly is availability: open appointment hours, bookable slots, or the number of active providers? Is it recorded per provider per week or only as a state total? - Is the \$200 paid to each participating provider for each week, and does it require a minimum amount of availability? - Is provider-level weekly data available, or only one weekly number per state? - Why was Massachusetts chosen, and did anything else change in either state during the 36 weeks? - How much revenue does the company earn from one booking? ### Part 1 — Check the design before trusting the estimate What must be true for Pennsylvania to be a valid comparison for Massachusetts, how would you check it with the 24 pre-launch weeks, and what else could explain a difference between the states? ```hint Use the 24 weeks Twenty-four weeks before launch say a lot about whether the two states moved together when nothing was different. ``` ```hint Anything else that changed Ask what else happened in only one of the two states during the program weeks. ``` #### What This Part Should Cover - The parallel-trends assumption and concrete ways to test it - Confounders: marketing campaigns, product changes and other state-specific events during the period - Threats from how Massachusetts was chosen, anticipation and spillovers between the states ### Part 2 — Estimate the effects, with statistical inference Estimate the program's effect on availability, and then on bookings. The candidate's own doubt was whether the analysis is only the change in Massachusetts minus the change in Pennsylvania, with no statistical inference possible. Is that right? Show how you would estimate the effect and attach uncertainty to it. ```hint Write it as a regression Express the comparison as a regression on the weekly data and ask what the interaction coefficient measures. ``` ```hint Count the independent units With one treated state and one comparison state, ask how many independent pieces of evidence really back a standard error. ``` #### What This Part Should Cover - The difference-in-differences estimator and its regression form - Valid standard errors given weekly autocorrelation and only two states - An event-study view and placebo checks - How the availability effect relates to the bookings effect ### Part 3 — Does the program pay for itself? Convert the bookings effect into incremental revenue and compare it with the program's cost. Should the company keep, expand or stop the program? ```hint Every dollar paid Count the incentive paid for availability that providers would have offered anyway, not only the payments that created new availability. ``` #### What This Part Should Cover - Incremental revenue measured as margin, not gross booking value - Full program cost and a break-even condition - How uncertainty, durability after the program ends and generalization to other states affect the decision ### What a Strong Answer Covers - A precise estimand and a clear statement of the parallel-trends assumption, tested with the pre-period data - Identification threats named specifically, each with a check - A correct difference-in-differences estimate with honest inference given only two states - A link from availability to bookings to incremental margin, set against the full cost - A decision with conditions, plus a better design for the next test ### Follow-up Questions - If the pre-launch trends are not parallel, what would you do instead? - How would using several comparison states, for example through a synthetic control, change the analysis and its inference? - Availability rose but bookings did not. What does that tell you about the market? - What would you expect after the incentive stops, and how would you measure it?