Explain the Bellman Equation

# Explain the Bellman Equation State and explain the Bellman expectation and optimality equations. Show how they decompose value into immediate reward and discounted future value, and connect them to policy evaluation and control. ### Constraints & Assumptions - Use a finite or discounted Markov decision process. - State whether reward is attached to the transition or current state-action pair. - Distinguish value under a fixed policy from optimal value. ### Clarifying Questions to Ask - Are you writing a state-value or action-value equation? - Is the policy fixed? - Are transition and reward expectations known or sampled? ```hint Condition on one step Expand return into the next reward plus the discounted remaining return, then take the appropriate expectation or maximum. ``` ### What a Strong Answer Covers - Bellman expectation equation for a fixed policy. - Bellman optimality equation and the role of maximization. - Contraction or fixed-point intuition under discounting. - Use in dynamic programming, temporal-difference learning, and Q-learning. ### Follow-up Questions 1. Why does replacing expectation with a maximum change the problem? 2. What breaks when undiscounted continuing rewards are unbounded?