Explain surprisal and its units
Quick Overview
This question evaluates understanding of probabilistic information measures in language modeling, specifically the concept of surprisal, its units, and its relationship to evaluation metrics such as cross-entropy and perplexity within the Machine Learning/NLP domain.
Amazon
Nov 20, 2025, 12:00 AM
Machine Learning Engineer
Technical Screen
Machine Learning
0
0
You are discussing a language-modeling / NLP project. The interviewer asks about surprisal.
- Define surprisal for an event/token with probability .
- What are the possible units of surprisal (analogous to meters for distance), and what determines the unit?
- How is surprisal related to cross-entropy and perplexity in language modeling?
- What common pitfalls or misunderstandings arise when interpreting surprisal values?
Solution
Show