Diagnose Search Issues with Relevant Metrics and Solutions
Quick Overview
Evaluates diagnosis of brand-name search relevance and safety issues using analytics and experimentation. Strong answers define relevance, satisfaction, and harmful-content metrics, find root causes, and validate fixes offline and online.
Diagnose Search Issues with Relevant Metrics and Solutions
Company: TikTok
Role: Data Scientist
Category: Analytics & Experimentation
Difficulty: medium
Interview Round: Onsite
##### Scenario
Brand clients report that searching their own name (e.g., 'Apple') on the platform surfaces irrelevant or harmful content.
##### Question
Which metrics would you examine to diagnose the search-relevance problem, and how would you structure an approach to resolve it?
##### Hints
Consider precision/recall, CTR, dwell time, harmful-content rate, brand-safety violations, time-series before/after model changes.
Quick Answer: Evaluates diagnosis of brand-name search relevance and safety issues using analytics and experimentation. Strong answers define relevance, satisfaction, and harmful-content metrics, find root causes, and validate fixes offline and online.
Brand-Name Search: Diagnosing Relevance and Safety Issues
Brand clients report that when users search for their own brand name, the platform surfaces irrelevant or harmful content. As a data scientist focused on analytics and experimentation, diagnose the problem and propose a resolution plan.
Constraints & Assumptions
Treat brand-name queries as potentially navigational, informational, ambiguous, or policy-sensitive.
Include relevance, satisfaction, and safety metrics.
Use both offline search evaluation and online experiment metrics.
Consider model, ranking, policy, entity resolution, and logging changes.
Clarifying Questions to Ask Guidance
Which brands, geographies, languages, and query variants are affected?
What content is considered irrelevant or harmful under policy?
Did a ranking model, policy, moderation, or indexing change coincide with the reports?
Are complaints about top results, all results, ads, autocomplete, or related searches?
Part 1 - Diagnostic Metrics
Identify key metrics to diagnose the search-relevance problem.
What This Part Should Cover Guidance
Include precision@k, recall where judged labels exist, NDCG, MRR, zero-result rate, CTR, long clicks, reformulation, dwell time, hides, reports, and user satisfaction.
Include harmful-content impression rate, policy violation rate, and brand-safety incident rate.
Segment by query intent, brand, language, region, surface, and result type.
Separate human-judged relevance from behavior-derived metrics.
Part 2 - Root Cause Analysis
Propose a structured approach to find root causes.