Find A Low-Quality Annotator From Label Data
Company: Meta
Role: Machine Learning Engineer
Category: Data Manipulation (SQL/Python)
Difficulty: hard
Interview Round: Onsite
You are given label data from several annotators and need to identify the person whose labels are likely wrong or low quality. Describe a practical pandas-based approach for cleaning the data, comparing annotator quality, and explaining the result. The interview also asks what product signals you would watch when rolling out a new AI product.
<details>
<summary>Hint 1</summary>
Start by naming the core entities, constraints, and success criteria.
</details>
<details>
<summary>Hint 2</summary>
Make the trade-offs explicit before going deep on implementation details.
</details>
### Constraints & Assumptions
- The exact dataset schema is not fixed, so define reasonable columns before solving.
- Assume labels include item id, annotator id, assigned label, and possibly a ground truth or consensus label.
- The goal is a clear analysis, not a fancy algorithm for its own sake.
- Use simple pandas operations where possible.
### Clarifying Questions to Ask
- Is there trusted ground truth, or only annotator agreement?
- Can one item have labels from multiple annotators?
- Are some items harder than others?
- What threshold defines an outlier annotator?
- What product rollout decision depends on this analysis?
### What a Strong Answer Covers
```premium-lock What a Strong Answer Covers
```
### Follow-up Questions
- How would you handle no ground truth?
- How would you avoid punishing annotators assigned harder items?
- How would you visualize the result?
- What AI launch metric would make you stop a rollout?
Quick Answer: Practice a pandas-style data analysis prompt for identifying a low-quality annotator from label data. The question emphasizes cleaning, agreement or ground-truth comparisons, sparse annotator handling, class imbalance, and AI product rollout metrics.