Q-Learning Explained: A Guide to Reinforcement Learning Basics

Deep Q-Networks (DQNs) extend Q-learning by replacing lookup tables with neural networks, enabling continuous state spaces. Neural networks serve as function approximators, taking states as inputs and outputting Q-values for all actions simultaneously. This allows scaling to complex environments like Lunar Lander with continuous states (positions, velocities, angles) and discrete actions. Visualization reveals how networks learn meaningful mappings between inputs and outputs, such as firing left engines when veering right. DQNs represent a critical bridge from tabular methods to practical applications in continuous domains.

Deep Q-Networks combine reinforcement learning with deep neural networks to approximate the Q-function in Q-learning algorithms. DQNs use neural networks to estimate Q-values for high-dimensional state spaces, making them suitable for complex problems like image-based games. The algorithm uses experience replay and target networks to stabilize training and prevent divergence during learning.

Deep Q-Networks (DQNs) extend Q-learning to high-dimensional states using neural networks as function approximators. Key innovations include experience replay (breaking sample correlations by sampling random mini-batches from a buffer) and fixed target networks (using separate networks with periodically updated parameters to stabilize training). Frame stacking enforces the Markov property by providing temporal context. DQNs achieved superhuman performance on many Atari games but struggle with sparse reward environments requiring long-term planning. The transition from tabular to function approximation enables handling complex real-world environments with continuous state spaces.

Deep Q-Networks (DQNs) solve Q-learning limitations by using neural networks as function approximators instead of tables. Neural networks take observation values as inputs and output Q-values for each action, enabling handling of continuous observation spaces and scaling to complex problems. Key DQN hyperparameters include: policy network architecture (hidden layers and units), learning rate, replay buffer size, learning start steps, batch size, discount factor (gamma), train frequency, gradient steps per rollout, target network update frequency, and neural network layer sizes. The replay buffer stores previous experiences and samples from it to break correlations between consecutive samples, enabling more stable learning. Target networks provide stable targets during training by periodically synchronizing with the main network, preventing oscillations and instability. This transition from tabular to function approximation represents a fundamental shift in RL capability, enabling application to problems with high-dimensional or continuous state spaces.

Deep Q-Networks (DQN) replace traditional Q-learning tables with neural networks to approximate Q-values. The input is a one-hot encoded state vector (size 16 for Frozen Lake), and the output predicts action values (size 4 for up/down/left/right). This approach allows handling high-dimensional inputs like images, as demonstrated in the famous 2013 DeepMind Atari paper where CNNs learned directly from raw pixels to achieve superhuman performance on multiple games.

This section covers action-value methods and the on-policy vs off-policy distinction. Q-learning is a TD algorithm for action values using the maximum Q-value as the target, making it off-policy (learning about the optimal policy while following a different exploratory policy). SARSA is an on-policy method learning about the same policy it's following. Expected SARSA takes the expectation over all actions rather than the maximum, providing an off-policy version of SARSA. The speaker notes that off-policy learning is theoretically more challenging because you're learning about a policy different from the one generating the data.

Q-Learning and SARSA are two fundamental reinforcement learning algorithms that help agents learn optimal actions through trial and error; Q-Learning is an off-policy algorithm that evaluates actions by always choosing the maximum expected future reward (using max Q-value), while SARSA is an on-policy algorithm that evaluates actions based on the actual next action taken, with both algorithms updating Q-values using the formula: Q(s,a) = Q(s,a) + α[r + γQ(s',a') - Q(s,a)], where α is the learning rate and γ is the discount factor for future rewards.

On-policy algorithms use the current policy to generate samples and improve that same policy. Off-policy algorithms use the current policy to generate samples but improve a different target policy. Q-Learning is off-policy because it uses the maximum Q-value for the next state, which may not correspond to the action actually taken. SARSA is on-policy because it uses the actual Q-value of the action that will be taken next according to the current policy.

SARSA (on-policy) and Q-Learning (off-policy) are two fundamental TD control algorithms that differ by a single term: SARSA uses the actual next action chosen by the current policy (including exploration), while Q-Learning uses the maximum Q-value over all possible next actions (purely greedy). This distinction determines whether the algorithm learns from what it actually does (SARSA) or from what could be done optimally (Q-Learning). In the cliff-walking problem, SARSA finds a safer, longer path avoiding the cliff, while Q-Learning theoretically finds the optimal shortest path but suffers repeated failures during training due to exploration falling off the cliff. Q-Learning's ability to learn optimal values from exploratory behavior enabled the development of DQN and modern deep reinforcement learning.

On-policy algorithms evaluate or improve the same policy used for decisions, learning values for the policy being followed. Off-policy algorithms learn optimal policies while behaving with more exploratory policies, decoupling behavior from target policies. Q-learning is the most widely used off-policy TD algorithm: Q(s,a) ← Q(s,a) + α[R + γmax_a' Q(s',a') - Q(s,a)], using the maximum Q-value for the next state rather than the action chosen by the policy. In the cliff walking problem, SARSA finds a safer path while Q-learning finds the optimal path closer to the cliff. SARSA collects more rewards because it's conservative, while Q-learning's optimal path often causes the agent to fall into the cliff during exploitation. Off-policy algorithms are generally stronger because they find optimal policies quickly.

The Q-learning class implementation includes parameters: environment (OpenAI Gym), learning rate (alpha), discount rate (gamma), epsilon (exploration rate), number of episodes, number of bins for state discretization, and bounds. The Q-matrix is initialized as a multi-dimensional array. Key methods include discretize_state() mapping continuous states to discrete indices, select_action() implementing epsilon-greedy with adaptive exploration scheduling (random for first 500 episodes, then decreasing epsilon), and simulate_episodes() executing the complete training loop. After training, the optimal greedy policy (epsilon=0) is extracted from the learned Q-matrix. Performance is validated by comparing against a random policy baseline—trained agents achieve hundreds of rewards per episode versus 20-30 for random actions.

OpenAI Gym provides environments for RL training, including Frozen Lake. The implementation involves: (1) initializing a Q-table with dimensions matching states (16) and actions (4), (2) setting hyperparameters like learning rate (α) and discount factor (γ), (3) running episodes where the agent plays until termination, (4) tracking performance metrics like steps per episode and total rewards. The environment automatically handles state transitions and reward calculations. Key hyperparameters include learning rate (controls how much new information overrides old information), discount factor (controls how much future rewards are valued), and number of episodes (how many times the agent plays to learn).

Q-learning is a reinforcement learning algorithm that trains an agent to make decisions by learning from rewards and penalties through a Q-table that stores expected future rewards for each state-action pair. The algorithm uses an exploration-exploitation trade-off where the agent randomly explores new actions or exploits known good actions based on an exploration rate that decays over time. In this implementation, the agent learns to play the Frozen Lake game by receiving +1 reward for reaching the goal and 0 otherwise, achieving approximately 70% success rate after 10,000 episodes through iterative Q-value updates using the formula: Q(s,a) = Q(s,a) + α[r + γmax(Q(s',a')) - Q(s,a)].

The Q-learning update equation combines immediate rewards with discounted future rewards: Q(s,a) = Q(s,a) + α[r + γmax(Q(s',a')) - Q(s,a)]. In grid world environments, agents navigate 2D grids to reach goals, with state spaces defined by position and action spaces by movement directions. Reward functions can be designed as negative distance to goals, with positive rewards for goal proximity and large negative penalties for failures. The Q-table is initially unknown and must be learned through exploration. This framework allows agents to learn optimal navigation strategies through repeated interaction with the environment.

Q-Learning is a model-free reinforcement learning algorithm that learns an optimal policy by iteratively updating a Q-table with state-action values using the Bellman equation: Q(s,a) = Q(s,a) + α[r + γ × max(Q(s',a')) - Q(s,a)], where α is the learning rate, γ is the discount factor, and r is the immediate reward. The algorithm balances exploration (random actions) and exploitation (using learned Q-values) through an epsilon-greedy strategy. In this implementation using the Gym Taxi environment, the agent learns to navigate a 4x4 grid, pick up passengers at designated locations, and drop them off at correct destinations while minimizing movements and avoiding walls, achieving an average of 12.37 actions with no penalties after 100,000 training episodes.