Judging Meta Ads Ranking Quality: Metrics, Experiments and Launch Criteria

You are a product manager at Meta working on ads ranking, the system that decides which ads a person sees and in what order. Design a framework for evaluating it: how the team should tell whether ads are being ranked well, and how it should decide whether a proposed change to the ranking is an improvement. ```hint Start from who ranking serves Every ranking decision affects the person seeing the ad, the advertiser paying for it and Meta's revenue. Define what a good outcome is for each of them before you name any metric, and look for where those outcomes pull against each other. ``` ### Clarifying Questions - Is the framework for judging individual ranking changes before launch, for monitoring the live system, or both? - Which surfaces are in scope, such as Feed, Stories and Reels on Facebook and Instagram, and should each be judged separately? - Does ads ranking here cover only predicting and ordering ads in the auction, or also deciding how many ads to show? - Which advertiser goals must the framework handle, for example purchases, app installs and brand awareness? ### What a Strong Answer Covers - The parties ranking serves (people, advertisers, Meta), what success means for each, and how they conflict - Layered evaluation, from offline model quality to online experiments to long-term effects, with what each layer can and cannot show - A concrete scorecard: primary decision metrics plus advertiser, user-experience and integrity guardrails - Experiment problems specific to ad auctions, such as advertiser budgets shared across test and control, auction price effects and delayed conversions - Pre-set launch criteria and post-launch monitoring, cut by surface, advertiser size and campaign goal ### Follow-up Questions - A new model raises ad revenue in an experiment, but advertisers' cost per result also rises. Do you launch? - How would you detect a ranking change that slowly teaches people to ignore ads, an effect a short test will not show? - How should the framework adapt when privacy changes leave fewer observed conversions to learn from and measure?