# Interpret Language-Model Perplexity Define perplexity for an autoregressive language model, derive its relationship to average negative log-likelihood, and explain what it does and does not reveal about model quality. ### Constraints & Assumptions - Evaluate on a fixed tokenized dataset with a stated logarithm base. - Mask padding and any tokens that are not prediction targets. - Perplexities from different tokenizers are not directly comparable. ### Clarifying Questions to Ask - Is loss averaged per token, sequence, or byte? - Was the evaluation set seen during training? - How are long contexts and truncation handled? ```hint Start from likelihood Express perplexity as the exponential of mean token cross-entropy before discussing intuition. ``` ### What a Strong Answer Covers - Formula and interpretation as a geometric mean branching factor. - Correct masking, normalization, and tokenizer dependence. - Use for held-out predictive fit and training diagnostics. - Limits for factuality, safety, usefulness, calibration, and cross-model comparison. ### Follow-up Questions 1. Why can lower perplexity fail to improve a user's task? 2. How would byte-level normalization help compare different tokenizers?