Success Metrics and Guardrails for edX Course Search

How would you measure whether the search bar on edX, the online course platform, is successful? ```hint Search is a means A search has only worked if the learner does something afterward that shows they found what they came for. Decide what that action is on edX before you pick any metric. ``` ### Clarifying Questions - Is this the site-wide search for courses, certificates and degree programs, or the search inside a course a learner is already taking? - Are we judging the current search bar, or setting success criteria for a redesign that is about to launch? - Which learners matter most here: people browsing free courses, learners after a paid certificate, or prospective degree students? - Should free enrollments and paid upgrades count equally, or is paid conversion the priority? ### What a Strong Answer Covers - The goal of search for learners and for edX's business, and how it shapes what success means - One primary success metric tied to a learner outcome, with the reason it beats clicks or query volume - Supporting metrics for search adoption and result quality, segmented by the kind of query and learner - Guardrails that catch misleading wins, covering speed, site-wide enrollments and what learners do after enrolling - How changes would be evaluated, combining experiments, offline relevance judgments and qualitative signals ### Follow-up Questions - Search-to-enrollment conversion rises after a ranking change, but total enrollments fall. How do you explain it? - How would you separate zero-result searches caused by poor matching from ones where edX simply has no such course? - How would you design an A/B test for a new ranking model, and what would make you stop it early?