Policy Gradient

Skip estimating values — just sample actions and do more of what paid off.

Policy gradient learns the behaviour directly instead of first estimating how good each state is. The policy π(a | s) is a set of probabilities over actions, usually output by a neural network. The agent samples an action, sees the reward, and adjusts the probabilities to make rewarding actions more likely.

REINFORCE — the core idea

The classic method, REINFORCE, runs a full episode, then shifts the policy by gradient ascent: increase the log-probability of each action taken, weighted by the return G it ultimately earned. Good outcomes pull their actions up; bad outcomes push theirs down.

∇J  =  E[ G · ∇ log π(a | s) ]

# G > 0  -> push π toward action a  (do more of this)
# G < 0  -> push π away from a      (do less of this)
θ <- θ + α · ∇J        # gradient ASCENT: climb toward more reward

Why differentiate the policy directly?

Value methods like Q-learning struggle with continuous action spaces (you can't take a max over infinitely many actions) and force deterministic choices. A policy can output a continuous, stochastic distribution naturally — steer a robot, set a temperature — which is why policy gradients dominate robotics and are the backbone of LLM RLHF.

On-policy and noisy

1It is on-policy: each update uses data collected by the *current* policy, then that data is thrown away.
2Returns vary wildly from episode to episode, so the gradient is a high-variance estimate.
3Subtract a baseline (a value estimate) so only *better-than-expected* actions are reinforced — this slashes variance without bias.

Actor-critic and PPO

Pairing a policy (the actor) with a learned value function (the critic) as the baseline gives actor-critic methods. Adding a clip that stops each update from moving the policy too far gives PPO — stable, sample-efficient, and the algorithm behind RLHF for chat models.

High variance is the catch

Because the signal is a sampled return, naive policy gradients learn slowly and unstably. Baselines, advantage estimates, and trust-region clipping (PPO) exist precisely to tame that variance.

OperationTimeSpace

Per update · needs a fresh rollout — on-policyO(episode length)O(policy params)

Variance · baseline / critic reduces ithigh without a baseline—

Check yourself

What problem does subtracting a baseline solve in policy gradient methods?