Explain surprisal and its units
Company: Amazon
Role: Applied Scientist
Category: Machine Learning
Difficulty: medium
Interview Round: Technical Screen
You are discussing a language-modeling / NLP project. The interviewer asks about **surprisal**.
1. Define *surprisal* for an event/token with probability \(p\).
2. What are the possible **units** of surprisal (analogous to meters for distance), and what determines the unit?
3. How is surprisal related to **cross-entropy** and **perplexity** in language modeling?
4. What common pitfalls or misunderstandings arise when interpreting surprisal values?
Overview: This question evaluates understanding of probabilistic information measures in language modeling, specifically the concept of surprisal, its units, and its relationship to evaluation metrics such as cross-entropy and perplexity within the Machine Learning/NLP domain.
