Steering a Cart-Pole with Deep Q-Learning
A DQN agent that keeps the pole upright while driving the cart at whatever speed I command with the arrow keys.
The problem
The standard CartPole task only asks an agent to keep the pole from falling. I wanted to drive it: hold the right arrow and the cart should speed up to the right without dropping the pole, let go and it should come to a stop.
That’s harder than it sounds. To accelerate one way, the cart first has to move slightly the other way so the pole leans into the motion. Balancing and moving on command pull against each other, and the agent has to learn when to give up a little of one for the other.
Approach
- Observation: the usual four states (cart position and velocity, pole angle and angular velocity) plus the commanded velocity: −1, 0 or +1.
- Reward: each step earns between 0 and 1 depending on how close the cart’s velocity is to the command. If the pole falls, the agent loses 10.
- Training: a single run of 800 episodes, up to 2,000 steps each. The command switches randomly every 50–300 steps so the agent practices starting, stopping and reversing.
- Agent: Double DQN with soft target-network updates, Huber loss and a 500k-transition replay buffer.
- Demo: the trained policy runs in real time and reads the arrow keys to set the command.
Results
To test the trained agent, I gave it a fixed 20-second command sequence (0 → +1 → −1 → 0 m/s) and recorded the cart’s velocity and the pole angle. It finished the run without dropping the pole.
| Command | Time to settle | Error after settling |
|---|---|---|
| 0 → +1 m/s | 0.60 s | 0.13 m/s |
| +1 → −1 m/s | 1.84 s | 0.14 m/s |
| −1 → 0 m/s | 0.66 s | 0.10 m/s |
The “wrong way first” move shows up at every command change: the velocity dips the opposite way before heading toward the new target.
On a full reversal the pole leans to 11.2°, just under the 12° limit. Leaning harder lets the cart accelerate harder, and the reward only cares about velocity error, so the agent learned to use almost all of the available margin. A real system would need a penalty on pole angle to keep some margin in reserve, at the cost of slower reversals.
What broke
Two-stage training. My first plan was to train the agent to balance with the command fixed at zero, then fine-tune it with random commands. Fine-tuning was unstable, with episode lengths swinging widely from one episode to the next. Training on both goals from the start with one combined reward was simpler and learned more reliably.
The edges of the track. Holding a command long enough always drives the cart off the end of the track, which normally ends the episode. That punished the agent for following orders. I wrote an environment wrapper that moves the cart to the opposite edge instead, so only a dropped pole ends an episode.