Reinforcement Learning on LunarLander

• Tabular Q-learning (off-policy) and SARSA (on-policy) implementations
• State discretization of LunarLander's 8-dimensional continuous observation space
• Hyperparameter sweeps across ε and α
• Greedy evaluation with per-action distribution analysis
• ~250,000 Q-entries (≈ 62,500 states × 4 actions)

Overview

A comparative study of two tabular reinforcement learning algorithms on Gymnasium's LunarLander-v3 (the discrete-action, extra-credit environment). LunarLander has an 8-dimensional continuous observation space, so the notebook discretizes each dimension into bins and stores Q-values in a dictionary keyed by the discretized state. Course project for CSC 158 at Hofstra, completed with my teammate Yash.

State Discretization

Each of the 8 observation dimensions is binned: 5 bins for x, y, vx, vy, angle, and angular velocity, plus 2 bins for each leg-contact flag. That gives roughly 62,500 states × 4 actions ≈ 250,000 Q-entries, large enough to be interesting and coarse enough to fit in memory and converge in a few thousand episodes.

Algorithms

Q-learning (off-policy). Target uses max over the next state's actions, independent of what the behavior policy actually does next:

Q(s, a) ← Q(s, a) + α · [ r + γ · max_{a'} Q(s', a')  −  Q(s, a) ]

SARSA (on-policy). Target uses Q(s', a') where a' is the action the current ε-greedy policy will actually take, so the cost of exploration shows up directly in the value estimates:

Q(s, a) ← Q(s, a) + α · [ r + γ · Q(s', a')  −  Q(s, a) ]

Both agents share the discretizer, the ε-greedy action selector, and the same training-loop scaffolding; the only thing that differs is the update.

Results

All sweeps used 3,000 episodes, max 1,000 steps per episode, γ = 0.99. Random baseline: ≈ −351.

Effect of ε (α fixed at 0.5)

Effect of α (ε fixed at 0.5)

α = 0.3 was best for both. With LunarLander's noisy reward (swings of hundreds of points per episode), a smaller step size averages over more transitions before changing the estimate.

Greedy evaluation (ε = 0, 100 episodes)

Q-learning's greedy policy leans heavily on the main engine and rarely fires the side thrusters, so it crashes when the lander starts off-center. SARSA's greedy policy uses all four actions in roughly balanced proportions (~32% nothing, ~30% main, ~19% each side). That matches the on-policy / off-policy distinction: SARSA's value estimates absorbed the cost of its own exploration during training, so it kept side-thruster actions in its policy.

What the Numbers Actually Say

Neither agent learned an optimal policy. Both training curves plateau early, greedy evaluation is worse than the average training reward (a sign the argmax over Q is unreliable in rarely-visited states), and neither agent produced a single successful landing across 100 episodes. The bottleneck is the discretization: many physically distinct situations map to the same bin, so tabular methods cannot represent the fine-grained control LunarLander rewards. Cracking this environment properly needs either much finer bins (combinatorial state explosion), function approximation (DQN), or decaying ε / α schedules.

Tech Stack

• Python 3, NumPy, Matplotlib
• Gymnasium with box2d-py for LunarLander-v3
• Jupyter notebook for the full training and evaluation pipeline