Explain a Markov Decision Process

# Explain a Markov Decision Process Define a Markov decision process using states, actions, transition probabilities, rewards, and a policy. Explain the Markov property and how a trajectory produces return. ### Constraints & Assumptions - Use a discrete formulation unless continuous spaces are explicitly requested. - State the discount or finite-horizon convention. - Reward may depend on state, action, and next state. ### Clarifying Questions to Ask - What information must a state contain to be Markov? - Is the environment fully observed? - What objective is the policy optimizing? ```hint Test the state If the future still depends on omitted history after conditioning on the proposed state and action, the state representation is incomplete. ``` ### What a Strong Answer Covers - Formal components and the Markov property. - Policy, trajectory probability, reward, return, and discount. - Value functions and the distinction between environment dynamics and policy. - Examples of non-Markov observations and how to repair the state. ### Follow-up Questions 1. How does partial observability change the formulation? 2. What is the difference between episodic and continuing tasks?