Reinforcement Learning Snake AI: Stable Baselines3 Tutorial

In reinforcement learning, an agent is the learner/decision-maker (like a robot or game character), the environment is the world the agent operates in, actions are the commands the agent sends to the environment, states are the observations the environment returns describing the current situation, and rewards are feedback signals measuring the success of actions taken.

An agent is an entity that can take actions in an environment. The environment is the world in which the agent operates and provides observations back to the agent. A state (s) is a concrete, immediate situation the agent finds itself in—essentially what the agent observes from the environment. An action (a) is a decision the agent makes based on its current state. Rewards (r) are numerical feedback signals provided by the environment that measure the success, failure, or penalties of the agent's actions at each time step.

In RL, the learning entity is called an agent (not a model). The agent interacts with the environment in a closed loop: the agent takes an action, the environment responds, and the agent receives feedback. This feedback is called reward (a real number indicating how good or bad the action was). The environment also reports the next state to the agent. The sequence of states, actions, and rewards collected over time is called a trajectory. The policy (π) is a function that takes the current state as input and outputs an action. The policy can be deterministic or stochastic. The return (G) is the sum of rewards received over time, often discounted by a factor γ (0 < γ < 1) to give less weight to future rewards. The expected return is the expected value of the return given the policy, written as E[G | π]. The state must contain all information necessary for making decisions - if incomplete, the agent cannot make optimal decisions.

Reinforcement learning consists of four core components: (1) Agent - the decision-making entity that interacts with the environment; (2) Environment - the world in which the agent operates; (3) Actions - the set of possible actions the agent can take; (4) Observations - the information the agent receives about the environment. The reward function R(t) measures immediate feedback at each time step, but reinforcement learning considers cumulative rewards over time. Future rewards are discounted because they are less valuable than immediate rewards, ensuring agents prioritize immediate gains while still considering future benefits.

Key RL concepts include: Action Space - all possible actions an agent can take (for CartPole, actions are 0 or 1 representing left/right movement). Observation Space - the state representation of the environment (CartPole has four attributes: cart position, cart velocity, pole angle, and pole angular velocity). Step - each action taken by the agent. Reward - feedback given for each step (CartPole gives +1 per step while balanced). Episode - one complete simulation from start to finish. Termination/Truncation - conditions ending an episode (pole falls beyond threshold, cart goes off edge, or maximum steps reached).

Gymnasium provides a standard API for reinforcement learning environments through reset() and step() methods. Reset initializes the environment and returns initial observation; step accepts actions and returns next observation, reward, termination status, and truncation status. The framework uses episodes—discrete interaction sequences where environments are reset between episodes. This standardization enables agents developed for one environment to work across different environments. Practical demonstration shows how to interact with the CartPole environment, sample random actions from action spaces, and accumulate rewards over episodes.

OpenAI Gym is a standard interface for reinforcement learning environments. It consists of two main functions: reset() which returns the initial state, and step(action) which takes an action and returns the next state, reward, and a boolean indicating if the episode has terminated. This interface allows researchers to implement and test reinforcement learning algorithms in a standardized way across different environments.

OpenAI Gym environments consist of: (1) An environment that defines the simulation space (like a building model or game); (2) A reset function that initializes the environment to a starting state; (3) A step function that advances the simulation by one time step, returning observations, rewards, and whether the episode has ended. The step function is where most of the environment implementation effort goes. The environment defines what observations are available, what actions are possible, and how the system evolves over time. This structure allows researchers to focus on algorithm development rather than environment implementation.

OpenAI Gym provides a standardized interface for reinforcement learning environments with methods including reset() to start a new episode, step(action) to take an action and receive the next state, reward, and done flag, and render() for visualization. The step() method returns a tuple (next_state, reward, done, info). The done flag indicates whether the episode has ended. The environment can be closed using close() to release resources. This API is used by many reinforcement learning libraries and is essential for implementing algorithms. The grid world environment is particularly useful for learning RL concepts as it captures essential elements: states, actions, rewards, and terminal states.

Gym provides a standardized API for reinforcement learning environments. Key functions include make() to create environments, step() to take actions and receive observations/rewards, and reset() to restart environments. Available environment types include Atari games (pixel-based observations), robotic control tasks (joint sensor observations), racing games, and browser tasks (file system navigation, form filling). The API works for most tasks except those requiring physical robot resets due to environmental variability.

Deep reinforcement learning is a computational framework where an agent learns to make optimal decisions by interacting with an environment through trial and error, receiving rewards as feedback to maximize cumulative expected return; the core concepts include the RL loop (state → action → reward → next state), the reward hypothesis, Markov property, exploration-exploitation trade-off, and two main solution approaches: policy-based methods (learning a policy function that maps states to actions) and value-based methods (learning a value function that estimates the expected return of each state); deep reinforcement learning extends these classical methods by using deep neural networks to approximate either the policy or value functions, enabling agents to handle complex, high-dimensional environments.

Deep reinforcement learning uses neural networks to approximate Q and V functions when state spaces are too large for tabular methods. Key concepts include: (1) temporal difference learning for improving value estimates; (2) SARSA (on-policy) and Q-learning (off-policy) for updating Q values; (3) neural network approximation of Q functions; (4) experience replay for stable training; (5) target networks for stabilization; (6) exploration-exploitation trade-off management. The agent learns by minimizing the difference between predicted and actual outcomes, improving predictions iteratively without complete environmental knowledge.

Stable Baselines 3 is a Python library that simplifies reinforcement learning by abstracting away complex algorithm implementations, similar to how scikit-learn simplifies general machine learning; it allows users to quickly experiment with various RL algorithms on custom or pre-built environments (like those from OpenAI Gym) by defining the environment, selecting an appropriate algorithm based on whether the action space is discrete or continuous, and training the model through repeated interactions where the agent receives observations, takes actions, and receives rewards until the episode ends.

Deep neural networks address the fundamental challenge of approximating functions in reinforcement learning when dealing with large or continuous state spaces. Traditional tabular methods like value iteration and Q-learning become impractical when states cannot be enumerated, as demonstrated by the astronomical state space in Go (3^361 states) and continuous state spaces in robotics. Neural networks provide a universal function approximation approach for Q functions, policies, and value functions. A neural network consists of interconnected computational units (neurons) with weighted edges. Each unit receives inputs, multiplies by weights, adds a bias, and applies a nonlinear activation function. Common activation functions include threshold, sigmoid, Gaussian, hyperbolic tangent, and identity functions. The Universal Approximation Theorem states that neural networks with at least one hidden layer using sigmoid, tanh, or Gaussian activation functions can approximate any continuous function arbitrarily closely.

Deep reinforcement learning has exploded in recent years, exemplified by DeepMind's AlphaGo demonstrating human-level performance in complex games. Deep neural networks serve two main purposes: either to learn environmental models for model-based reinforcement learning, or to represent model-free concepts like Q-functions and policies. Deep neural networks enable gradient-based optimization through automatic differentiation and backpropagation, allowing efficient training of complex policy representations.