Measuring the Incremental Impact of Instagram's NYC Subway Ad Campaign

Instagram's team is planning a display advertising campaign on the New York City subway. How would you design a test for the campaign, and how would you measure whether it succeeded? Explain what you would measure, what you would compare against, and how you would decide whether the campaign was worth running. ```hint What can you actually assign? You cannot show subway posters to a random half of New Yorkers the way an in-app test splits users. Think about which unit you can put in or keep out of the campaign, and what would tell you how New York would have behaved without it. ``` ### Clarifying Questions - What is the campaign meant to achieve: new sign-ups, bringing back lapsed users, more use of a specific feature, or better brand perception? - Is the media plan fixed (lines, stations and dates), or can we influence where and when the ads run to make a test possible? - Will other Instagram marketing, such as digital ads, TV or promotions, run in New York at the same time? - Which data can be tied to location, for example app installs, sign-ups and activity by metro area or ZIP code, and can we run brand surveys? ### What a Strong Answer Covers - A campaign objective translated into one primary success metric, with secondary behavioral and brand metrics - A test design that works without user-level randomization, such as comparison markets or a staggered rollout across lines, and the assumptions it relies on - Handling of seasonality, other marketing, product launches and spillover from commuters and visitors who live elsewhere - Power and duration, with the smallest effect worth detecting tied to the campaign's cost - A decision rule that says which results justify scaling, reworking or stopping the campaign ### Follow-up Questions - Brand awareness rises clearly, but sign-ups and activity show no measurable change. Is the campaign a success? - The campaign must cover the whole subway system at once and cannot be split within the city. How do you still estimate its effect? - How would you separate the campaign's effect from a major Instagram product launch that happens in the same month?