Explain KL Divergence in Language Models

# Explain KL Divergence in Language Models Define Kullback-Leibler divergence, explain its directionality and support requirements, and describe why modern language-model training or post-training uses it. ### Constraints & Assumptions - Use probability distributions over the same events or tokens. - KL divergence is not symmetric and is not a metric. - Distinguish a token-level estimate from the ideal sequence-level objective. ### Clarifying Questions to Ask - Which distribution is the reference and which is being optimized? - Is the goal regularization, distillation, variational inference, or monitoring? - How is the expectation estimated from samples? ```hint Name the expectation Write which distribution supplies samples and which log-probability ratio is averaged. ``` ### What a Strong Answer Covers - Definition and nonnegativity, asymmetry, and zero condition. - Consequences of support mismatch and direction choice. - Use as a policy-to-reference penalty or distillation objective in language models. - Estimation, coefficient trade-offs, and monitoring limitations. ### Follow-up Questions 1. What changes when reverse KL is used instead of forward KL? 2. Why can a sampled KL estimate be negative even though true KL is nonnegative?