Robot Learning

Markov Decision Process (MDP)

A Markov decision process (MDP) is the standard mathematical formalism for sequential decision-making, defined by a state space, action space, transition dynamics, reward function, and discount factor, under the Markov assumption that the current state fully summarizes history. Solving an MDP means finding a policy maximizing expected cumulative reward. Reinforcement learning addresses MDPs with unknown dynamics; partially observable variants (POMDPs) relax the full-observability assumption.

Why it matters for physical AI

Nearly all reinforcement learning theory and algorithms for robot control assume an underlying MDP, and understanding where robotics violates its assumptions, especially partial observability, explains many practical failures.