Explain surprisal and its units

Q: Explain surprisal and its units

This is a Machine Learning interview question from Amazon for Machine Learning Engineer roles. View the full question and solution on PracHub.

Q: How do I approach Machine Learning interview questions?

Machine Learning questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master machine learning interviews.

Question

You are discussing a language-modeling / NLP project. The interviewer asks about surprisal.

Define surprisal for an event/token with probability $p$ .
What are the possible units of surprisal (analogous to meters for distance), and what determines the unit?
How is surprisal related to cross-entropy and perplexity in language modeling?
What common pitfalls or misunderstandings arise when interpreting surprisal values?

Explain surprisal and its units

Solution

Comments (0)