Academic Databases
Explore Our Semantic Databases
STEM & Tech
Math, Physics, CS & Engineering...
Economics
Macro, Micro, Econometrics...
Business
Management, Marketing, Finance...
Health & Life Sciences
Biology, Medicine, Neuroscience...
Humanities & Social Sciences
Psychology, Law, Philosophy, History...
General Studies
General & interdisciplinary studies...
Explore Topics
Features

Create Your Own Reinforcement Learning Environments: Gridworld Tutorial

Added: 2026-08-07

Deep Dive

Object-Oriented Programming (OOP) in Python, specifically class inheritance and method overriding.

Python full course tutorials session 20

Python full course tutorials session 20

109

This section covers class inheritance and method overriding in Python. Child classes can inherit from parent classes using class ChildClass(ParentClass): syntax. Child classes automatically inherit all attributes and methods from parent classes. Child classes can add new attributes and methods specific to their purpose. The super() function helps child classes call parent class methods. Child classes can override parent methods by defining methods with the same name. Method overriding allows child classes to customize behavior while maintaining parent class functionality. Understanding inheritance enables code reuse and hierarchical class design.

Click to watch →

Python OOPs Tutorial 6| +91-9872993883 for query|

Python OOPs Tutorial 6| +91-9872993883 for query|

120

Class inheritance allows child classes to inherit attributes and methods from parent classes. The syntax is 'class Subclass(ParentClass):'. Child classes can access parent class methods and attributes, and can also add new functionality. Method overriding enables subclasses to change the implementation of inherited methods. The subclass redefines methods with the same name but different behavior. This allows customization of inherited functionality while maintaining the parent class structure. Objects of the subclass can call both their own methods and inherited methods from the parent class.

Click to watch →

Python Inheritance | Inheritance In Python | Intellipaat

Python Inheritance | Inheritance In Python | Intellipaat

6.8K

Object-Oriented Programming (OOP) is a paradigm using objects with attributes and behaviors to solve problems. Python implements OOP through classes (blueprints) and objects (instances). Classes are defined with 'class ClassName:' syntax, and objects are created by calling the class name. The __init__ constructor initializes object attributes. Python supports five inheritance types: Single (one child from one parent), Multiple (one child from multiple parents), Multi-level (chain inheritance), Hierarchical (multiple children from one parent), and Hybrid (combinations). The super() function calls parent class methods within child classes. Method overloading (same name, different parameters) is not natively supported in Python - defining a method with the same name overwrites the previous definition. To simulate overloading, use default arguments, *args, or runtime type checking. Method overriding (same name and signature in child class) is fully supported and allows child classes to modify parent class behavior.

Click to watch →

OOP in Python | Object Oriented Programming

OOP in Python | Object Oriented Programming

2M

Method overriding occurs when a subclass provides a specific implementation of a method that is already defined in its superclass. When you call the method on a subclass object, the subclass's version executes instead of the superclass's version. This allows subclasses to customize or extend parent class behavior while maintaining the same method signature. The overridden method in the subclass must have the same name and parameters as the method in the superclass.

Click to watch →

Python ООП за 20 минут | Наследование | Магические методы | Слоты

Python ООП за 20 минут | Наследование | Магические методы | Слоты

109

This section covers inheritance, a core OOP concept. A child class inherits from a parent class by specifying the parent in parentheses (e.g., class Child(Parent)). The child class can access all public attributes and methods of the parent. To properly initialize inherited attributes, the parent's constructor must be called using super().__init__() within the child's __init__ method. Method overriding allows the child class to provide a specific implementation of a parent's method, replacing the parent's version when called on child instances. However, private methods and attributes from the parent class are not inherited due to Python's name mangling mechanism.

Click to watch →

Deep Dive

Fundamental Reinforcement Learning (RL) terminology, including Agent, Environment, State, Action, and Reward.

Reinforcement Learning 2026 - Session 1

Reinforcement Learning 2026 - Session 1

127

In RL, the learning entity is called an agent (not a model). The agent interacts with the environment in a closed loop: the agent takes an action, the environment responds, and the agent receives feedback. This feedback is called reward (a real number indicating how good or bad the action was). The environment also reports the next state to the agent. The sequence of states, actions, and rewards collected over time is called a trajectory. The policy (π) is a function that takes the current state as input and outputs an action. The policy can be deterministic or stochastic. The return (G) is the sum of rewards received over time, often discounted by a factor γ (0 < γ < 1) to give less weight to future rewards. The expected return is the expected value of the return given the policy, written as E[G | π]. The state must contain all information necessary for making decisions - if incomplete, the agent cannot make optimal decisions.

Click to watch →

MIT 6.S191 (2023): Reinforcement Learning

MIT 6.S191 (2023): Reinforcement Learning

148.8K

In reinforcement learning, an agent is the learner/decision-maker (like a robot or game character), the environment is the world the agent operates in, actions are the commands the agent sends to the environment, states are the observations the environment returns describing the current situation, and rewards are feedback signals measuring the success of actions taken.

Click to watch →

Reinforcement Learning Basic Definitions - Machine Learning

Reinforcement Learning Basic Definitions - Machine Learning

3K

Reinforcement learning involves four core components: (1) Agent - an entity that takes action, typically implemented as an algorithm or software program (e.g., a drone making deliveries or Super Mario navigating games); (2) Action - the move an agent makes in a given state, chosen from discrete possibilities like running, jumping, or standing still; (3) State - the concrete situation the agent finds itself in, representing the configuration of surroundings, tools, obstacles, enemies, and prizes; (4) Reward - immediate feedback from the environment measuring action success or failure. The environment processes the agent's state and action to return reward and next state. The agent's objective is to maximize cumulative reward, with future rewards valued less than immediate ones through the discount factor mechanism.

Click to watch →

Fundamentals of Reinforcement Learning - Part I

Fundamentals of Reinforcement Learning - Part I

2.6K

The agent is the learner that performs actions to maximize rewards. The environment is the scenario where the agent acts, described by parameters including states. The state is an environment parameter that changes with agent actions, representing the current situation. Actions are agent properties that cause state transitions. The robot navigation example illustrates these concepts: a robot in a grid environment can move up, down, left, or right, with states representing its position and whether cells are charging or discharging stations. Different states may have different allowable actions. The agent observes states, takes actions, transitions to new states, and receives rewards based on its actions.

Click to watch →

MIT 6.S191 (2024): Reinforcement Learning

MIT 6.S191 (2024): Reinforcement Learning

123.9K

An agent is an entity that can take actions in an environment. The environment is the world in which the agent operates and provides observations back to the agent. A state (s) is a concrete, immediate situation the agent finds itself in—essentially what the agent observes from the environment. An action (a) is a decision the agent makes based on its current state. Rewards (r) are numerical feedback signals provided by the environment that measure the success, failure, or penalties of the agent's actions at each time step.

Click to watch →

Deep Dive

Familiarity with the OpenAI Gym/Gymnasium API lifecycle (specifically the reset, step, and render methods).

[Week Extra] Introduction to Machine Learning (noc25-cs46)

[Week Extra] Introduction to Machine Learning (noc25-cs46)

190

OpenAI Gym provides a standardized interface for reinforcement learning environments with methods including reset() to start a new episode, step(action) to take an action and receive the next state, reward, and done flag, and render() for visualization. The step() method returns a tuple (next_state, reward, done, info). The done flag indicates whether the episode has ended. The environment can be closed using close() to release resources. This API is used by many reinforcement learning libraries and is essential for implementing algorithms. The grid world environment is particularly useful for learning RL concepts as it captures essential elements: states, actions, rewards, and terminal states.

Click to watch →

Jonathan Mugan: RLlib, a Python Library for Deep Hierarchical Multi-Agent Reinforcement Learning

Jonathan Mugan: RLlib, a Python Library for Deep Hierarchical Multi-Agent Reinforcement Learning

10.7K

An OpenAI Gym environment implements four methods: reset (start new episode), step (execute action and return next state/reward/done flag), render (visualize), and close (cleanup). The step function receives an action dictionary and returns observation, reward, done status, and additional info, enabling interaction between RL algorithms and custom environments.

Click to watch →

How to train our first Reinforcement Learning agent! | Reinforcement Learning Part-3

How to train our first Reinforcement Learning agent! | Reinforcement Learning Part-3

13.7K

OpenAI Gym provides a standardized framework for RL environments with four core methods: make() creates environments by name, reset() initializes and returns initial observations, step(action) applies actions and returns next_observation, reward, done, and info, and render(mode) visualizes the environment. The library categorizes environments into classes including toy_text (for simple problems like cliff walking). Understanding this interface is essential for interacting with any RL environment in Gym.

Click to watch →

Hands-on workshop with Reinforcement Learning | GYM Library | DataHour - by Krupa Galiya

Hands-on workshop with Reinforcement Learning | GYM Library | DataHour - by Krupa Galiya

335

Interacting with a Gym environment involves a standard workflow: (1) Create the environment using gym.make(); (2) Reset the environment to initialize it to a starting state; (3) Render the environment to visualize it (optional); (4) In a loop, repeatedly: (a) choose an action, (b) step through the environment to execute the action, receiving observations, rewards, done flag, and info, (c) render the updated state. The step() function returns four values: observation (current state), reward (feedback), done (whether the episode ended), and info (debugging information).

Click to watch →

Everything you need to get started using OpenAI Gym to train RL agents.

Everything you need to get started using OpenAI Gym to train RL agents.

3.7K

OpenAI Gym is a toolkit that provides standardized environments for testing Reinforcement Learning algorithms, allowing developers to focus on learning algorithms rather than environment implementation; it offers diverse environments like CartPole and LunarLander, with core functions including gym.make() to create environments, env.reset() to initialize, env.step() to perform actions and receive state/reward/done signals, and env.render() for visualization, enabling researchers to easily test RL agents across different environments without modifying environment code.

Click to watch →

Deep Dive

Basic understanding of 2D coordinate systems or matrix representations for grid-based layouts.

Grid System 2D - Pikachu Unity2D - E4

Grid System 2D - Pikachu Unity2D - E4

2K

In 2D grid systems, matrix coordinates (integer positions) differ from screen coordinates due to cell dimensions, requiring an offset calculation to map grid positions to actual screen positions; the offset value controls the spacing between grid elements, where smaller values result in closer elements and larger values create more distance between them.

Click to watch →

What Are 2D Coordinates (Ordered Pairs)?

What Are 2D Coordinates (Ordered Pairs)?

149

A coordinate is a set of numbers describing a single location in relation to a reference. On 2D grids, coordinates are typically two numbers (ordered pairs) where significant digits indicate measurement precision. Precision differs from accuracy—precision refers to measurement detail, while accuracy refers to closeness to true values. Coordinates describe single points, not lines or shapes. The word 'coordinate' derives from Latin 'co-' (together) and 'ordinate' (in order), explaining why coordinate order matters critically. Three main 2D coordinate grid types exist: (1) Mathematicians' grid: origin at (0,0), X increases right, Y increases up, X-first order, zero direction on right, angles measured clockwise; (2) Geospatial/surveying grid: northing-first order, axes called northing/easting, origin typically not at (0,0) to avoid negatives, zero direction straight up, angles measured clockwise; (3) SVG coordinate system: origin at top-left, Y increases down (backwards from others), X-first order. Each system has different conventions for order, axis labeling, and origin placement.

Click to watch →

درس المرصوفة الجزء الثاني لتلاميذ السنة الثالثة ابتدائي

درس المرصوفة الجزء الثاني لتلاميذ السنة الثالثة ابتدائي

4.4K

A grid (مرصوفة) is a table divided into equal squares consisting of horizontal rows (سطور) and vertical columns (اعمده). Coordinates (احداثيات) are a system to locate items in a grid, consisting of two numbers separated by a comma: the first number represents the row, and the second represents the column. To locate an item, identify its row and column, then write them as coordinates. Navigation within a grid uses directional movements: right/left changes the column, up/down changes the row. Each item's position is defined by the intersection point of its specific row and column.

Click to watch →

[INFO188-2021] Reunión 24 - Programación en GPUs (parte 3)

[INFO188-2021] Reunión 24 - Programación en GPUs (parte 3)

111

For 2D matrix operations, the grid uses two-dimensional coordinates (x, y) instead of one-dimensional. Each thread's position is determined by both its x and y coordinates within the block. The global coordinates are calculated as: global_x = block_x * blockDim.x + threadIdx.x, and similarly for y.

Click to watch →

2 ubicacion de objetos en la cuadricula

2 ubicacion de objetos en la cuadricula

704

A grid (cuadrícula) is a structured arrangement containing rows (horizontal lines labeled a, b, c, d, e, f) and columns (vertical lines labeled 1, 2, 3, 4, 5, 6). An ordered pair (par ordenado) uses two elements to locate positions: a number (column) and a letter (row). The number always comes first, followed by the letter. To locate an object, first find the column number, then find the row letter, and place the object at their intersection. For example, 2D means column 2 and row D.

Click to watch →

Grid Overview

Setup & State

Magic Squares

State Checks

Env Dynamics

Reset & Render

Wrap Up

Grid Overview

0:00

Playing Section

1
Introduces the goal of building a custom reinforcement learning environment.
2
Describes the grid world setup with teleporting magic squares.

Deep Dive

Implementing tabular Reinforcement Learning algorithms, such as Q-learning or SARSA, to solve the custom Gridworld.

How To Create Your Own Reinforcement Learning Environments | Tutorial | Part 2

How To Create Your Own Reinforcement Learning Environments | Tutorial | Part 2

15K

This tutorial demonstrates how to implement a Q-Learning agent for a custom grid world environment, covering key concepts including the Q-learning update equation (Q(s,a) = Q(s,a) + α[r + γ*max(Q(s',a')) - Q(s,a)]), epsilon-greedy action selection with linear decay, optimistic initial values for exploration, and hyperparameter tuning (alpha=0.1, gamma=1.0, epsilon=1.0). The implementation shows how the agent learns to navigate a 9x9 grid with teleporters, achieving optimal performance with total rewards approaching -11 (the minimum number of steps to escape), demonstrating that Q-learning can solve grid world problems through iterative experience accumulation.

Click to watch →

Q-learning

Q-learning

5.1K

The Q-learning update equation combines immediate rewards with discounted future rewards: Q(s,a) = Q(s,a) + α[r + γmax(Q(s',a')) - Q(s,a)]. In grid world environments, agents navigate 2D grids to reach goals, with state spaces defined by position and action spaces by movement directions. Reward functions can be designed as negative distance to goals, with positive rewards for goal proximity and large negative penalties for failures. The Q-table is initially unknown and must be learned through exploration. This framework allows agents to learn optimal navigation strategies through repeated interaction with the environment.

Click to watch →

Q-Learning từ A-Z: Cách áp dụng vào Gridworld 5x5 (Python) | Reinforcement Learning cho người mới

Q-Learning từ A-Z: Cách áp dụng vào Gridworld 5x5 (Python) | Reinforcement Learning cho người mới

949

Q-Learning is a reinforcement learning algorithm that learns optimal behavior by building a Q-table, which stores expected rewards for each state-action pair. The algorithm updates Q-values using the formula: Q(s,a) = Q(s,a) + α[r + γ × max(Q(s',a')) - Q(s,a)], where α is the learning rate and γ is the discount factor. In a 5x5 Gridworld environment, Q-Learning can be implemented in Python to find the shortest path from a starting position to a goal, with rewards of +10 for reaching the goal, -10 for hitting obstacles, and -1 for each step taken. The implementation involves initializing the Q-table, training the agent through episodes, and extracting the optimal path once convergence is achieved.

Click to watch →

Reinforcement Learning Lecture: Q-Learning and SARSA in a simple language

Reinforcement Learning Lecture: Q-Learning and SARSA in a simple language

49.4K

Q-Learning and SARSA are tabular reinforcement learning algorithms that learn optimal actions through trial and error. Q-Learning uses the Bellman optimality equation, updating Q-values based on the maximum future reward (off-policy), while SARSA uses the Bellman expectation equation, updating based on the actual next action taken (on-policy). Both algorithms use a Q-table to store state-action values, with updates following the formula: Q(s,a) = Q(s,a) + α[r + γmax_a'Q(s',a') - Q(s,a)] for Q-Learning and Q(s,a) = Q(s,a) + α[r + γQ(s',a') - Q(s,a)] for SARSA. The learning rate α controls how quickly the agent updates its knowledge, while the discount factor γ determines how much future rewards are valued. Epsilon-greedy exploration balances exploration (random actions) and exploitation (best-known actions), with epsilon decaying over time.

Click to watch →

آموزش یادگیری تقویتی عمیق - -DQN - توضیح تئوری و کد نویسی با پایتورچ

آموزش یادگیری تقویتی عمیق - -DQN - توضیح تئوری و کد نویسی با پایتورچ

1.1K

A custom grid world environment for reinforcement learning can be implemented by defining: (1) Grid dimensions (e.g., 4x4); (2) Start position (e.g., top-left corner); (3) Actions (e.g., up=0, down=1, left=2, right=3); (4) Reward function (positive reward for reaching goal, negative for hitting obstacles); (5) Terminal state detection; (6) Reset function to restart episodes. The environment returns the next state, reward, and done flag after each action.

Click to watch →

Deep Dive

Transitioning from discrete environments to continuous state and action spaces using environments like Pendulum or CartPole.

Q-Learning Tutorial 6: Train Gymnasium Pendulum-v1 on Continuous Action and Observation Spaces

Q-Learning Tutorial 6: Train Gymnasium Pendulum-v1 on Continuous Action and Observation Spaces

2.2K

To apply Q-learning to environments with continuous action and observation spaces, such as the Pendulum-v1 environment, the continuous values must be discretized into discrete buckets using a binning function that divides the range into equal segments (e.g., 15 divisions), allowing the agent to learn a discrete Q-table that can be converted back to continuous actions when executing.

Click to watch →

Benjamin Rosman Reinforcement Learning Part 2

Benjamin Rosman Reinforcement Learning Part 2

337

Reinforcement learning traditionally uses discrete state spaces (like grid worlds) where value functions can be stored in tables. However, real-world problems involve continuous state spaces (GPS coordinates, robotic joint positions, camera inputs) that cannot be represented in tabular form. This transition requires fundamentally different approaches to representation and learning, as agents must now generalize to unseen states rather than relying on revisiting discrete states to refine estimates.

Click to watch →

Reinforcement Learning with Stable Baselines 3 - Introduction (P.1)

Reinforcement Learning with Stable Baselines 3 - Introduction (P.1)

110.8K

In reinforcement learning environments, action spaces can be either discrete or continuous. Discrete action spaces involve classification-like decisions where only specific choices are available (e.g., left or right movement with no intermediate values). Continuous action spaces involve regression-like decisions with a range of nearly infinite possibilities (e.g., setting servo torque between -1 and +1). Continuous environments are generally harder to learn but are often required in robotics applications where precise control is needed.

Click to watch →

Introduction to OpenAI Gym (Gymnasium): Cart-Pole Environment - Reinforcement Learning Tutorial

Introduction to OpenAI Gym (Gymnasium): Cart-Pole Environment - Reinforcement Learning Tutorial

29.1K

The cart-pole environment has a discrete action space with two actions: push left (0) or push right (1). The observation space includes four continuous state variables: cart position (X, -4.8 to +4.8 m), cart velocity (X dot), pole angle (Theta, -0.418 to +0.418 radians or -24° to +24°), and pole angular velocity (Theta dot). Initial states are randomized uniformly within [-0.05, +0.05] for all variables. Episodes terminate when pole angle exceeds ±12°, cart position exceeds ±2.4 m, or maximum steps (500) are reached. The reward structure provides +1 per time step while balanced, making cumulative reward a direct measure of control performance.

Click to watch →

Q-Learning Cân Bằng Con Lắc Ngược – Giải Thuật Học Tăng Cường AI [Matlab]

Q-Learning Cân Bằng Con Lắc Ngược – Giải Thuật Học Tăng Cường AI [Matlab]

497

The reinforcement learning environment for the inverted pendulum consists of the pendulum-cart system that receives actions and returns new states. The environment function takes current state variables and control force, then computes new state values using Euler's method. State discretization converts the continuous state space into discrete bins: theta into 6 bins, x into 3 bins, x_dot into 3 bins, and theta_dot into 3 bins. This discretization is necessary because reinforcement learning algorithms like Q-learning require discrete state spaces. The state index is calculated by combining bin indices, creating a unique identifier for each state in the discretized space.

Click to watch →

Deep Dive

Integrating the custom environment with Deep Reinforcement Learning (DRL) libraries like Stable-Baselines3 or Ray/RLlib to train DQN or PPO agents.

Custom Environments - Reinforcement Learning with Stable Baselines 3 (P.3)

Custom Environments - Reinforcement Learning with Stable Baselines 3 (P.3)

63.3K

To create a custom environment for reinforcement learning with Stable Baselines 3, you must convert your environment to follow the gym interface by defining the action space (discrete actions like up, down, left, right), observation space (features such as head position, apple position, snake length, and previous moves), and implementing the reset() and step() methods; the key challenge lies in designing effective observation features and reward functions that enable the RL agent to learn optimal policies efficiently.

Click to watch →

Reinforcement Learning with Stable Baselines 3 - Introduction (P.1)

Reinforcement Learning with Stable Baselines 3 - Introduction (P.1)

110.8K

Stable Baselines 3 is a Python library that simplifies reinforcement learning by abstracting away complex algorithm implementations, similar to how scikit-learn simplifies general machine learning; it allows users to quickly experiment with various RL algorithms on custom or pre-built environments (like those from OpenAI Gym) by defining the environment, selecting an appropriate algorithm based on whether the action space is discrete or continuous, and training the model through repeated interactions where the agent receives observations, takes actions, and receives rewards until the episode ends.

Click to watch →

Building a Custom Environment for Deep Reinforcement Learning with OpenAI Gym and Python

Building a Custom Environment for Deep Reinforcement Learning with OpenAI Gym and Python

157.7K

After building the custom environment and adapting the model, you can train the DQN agent using the custom environment. Training involves running multiple episodes where the agent learns to maximize cumulative rewards. After training, evaluation tests the agent's performance by measuring average rewards across multiple test episodes. The presence of random noise in the environment means perfect performance may not be achievable, but the reward should improve over training iterations as the agent learns optimal policies.

Click to watch →

Алгоритм DQN — учим нейросети принимать решения // Демо-занятие курса «Reinforcement Learning»

Алгоритм DQN — учим нейросети принимать решения // Демо-занятие курса «Reinforcement Learning»

393

Stable Baselines is a library providing well-implemented RL algorithms with optimized code. Using Stable Baselines is recommended over custom implementations because it handles experience replay, target networks, and other implementation details. The library includes DQN and other algorithms with tested code. Advanced algorithms include Actor-Critic methods (combining policy and value functions), A3C (asynchronous actor-critic), and PPO (proximal policy optimization). These algorithms are designed for continuous action spaces and provide better sample efficiency and stability than DQN.

Click to watch →

Ray RLlib: How to Use Deep RL Algorithms to Solve Reinforcement Learning Problems

Ray RLlib: How to Use Deep RL Algorithms to Solve Reinforcement Learning Problems

14.6K

This tutorial demonstrates how to solve the CartPole-v1 environment using Ray RLlib's PPO algorithm, covering the four essential components of an RL experiment: initializing Ray, selecting an RL algorithm (PPO), configuring the environment and hyperparameters, and running experiments with Tune; the process involves alternating training and evaluation phases where training improves the policy through iterative updates while evaluation assesses performance using metrics like episode reward mean, with successful training indicated when rewards approach the environment's goal threshold of 500 per episode.

Click to watch →

Deep Dive

Designing more complex custom environments, including handling partial observability (POMDPs) or multi-agent systems.

【ReHacQvs東京大学】超AI時代到来！人間を超えた人工知能で未来はどうなる？【ひろゆきもびっくり・AI松尾研】

【ReHacQvs東京大学】超AI時代到来！人間を超えた人工知能で未来はどうなる？【ひろゆきもびっくり・AI松尾研】

102.2K

In many multi-agent systems, each agent has partial observability, meaning it can only perceive a limited portion of the overall environment. Agents cannot see what other agents are doing or what the complete state of the system is. This partial information creates additional complexity because agents must make decisions based on incomplete information while still coordinating effectively with other agents.

Click to watch →

Lecture 1: Part 2: Course Introduction

Lecture 1: Part 2: Course Introduction

100

Planning difficulty depends on environmental characteristics: (1) Deterministic vs Stochastic - deterministic is easier because actions have single predictable outcomes; (2) Observability - fully observable is easiest, partially observable is worst because agents must gather information; (3) Static vs Dynamic - static worlds remain unchanged while agents think, dynamic worlds change even when agents aren't acting; (4) Single Agent vs Multi-Agent - multi-agent scenarios require modeling other agents' mental states. Partial observability is the worst case because agents must perform additional actions to gather information. Mental models are crucial for multi-agent planning, as demonstrated by the Sally-Anne test showing children develop understanding that others can have false beliefs around ages 3-4. Classical planning is the simplest case: static, deterministic, fully observable, discrete state spaces with instantaneous actions. Adding durative actions leads to metric temporal planning. Partially observable environments require confident planning (solving without sensing) or contingent planning (gathering information first). Stochastic environments require handling probabilistic outcomes. Markov Decision Processes (MDPs) handle stochastic environments with full observability, while Partially Observable Markov Decision Processes (POMDPs) extend this to handle both stochasticity and partial observability.

Click to watch →

KI1 V02: Intelligent Agents (part 1/2)

KI1 V02: Intelligent Agents (part 1/2)

418

Environments have key properties affecting agent design: Partially observable environments (like poker) hide relevant information from agents. Single-agent environments contain only one optimizing agent, while multi-agent environments have multiple agents that may compete or cooperate. Deterministic environments have predictable outcomes, while stochastic environments have probabilistic outcomes. These properties determine how complex and challenging an agent must be to function effectively.

Click to watch →

Rich Sutton (DeepMind Alberta, University of Alberta, Amii) - Experience and Intelligence

Rich Sutton (DeepMind Alberta, University of Alberta, Amii) - Experience and Intelligence

2.6K

Reinforcement learning is particularly suited for real-world systems where the agent must actually care about the rewards it receives and cannot afford to make mistakes. The framework can handle partial observability by assuming the world is a complicated MDP that the agent cannot fully observe. The agent produces an internal state representation that acts as the true state. This approach is general and handles all the sophistication of real-world environments, including partial observability and non-stationarity. The framework is designed to handle the general case where the world may be fully observable or non-stationary.

Click to watch →

Multi-Agent Learning Kickoff Meeting

Multi-Agent Learning Kickoff Meeting

255

This section covers fully observable and partially observable stochastic games, the most general forms of MARL. In fully observable stochastic games, agents have complete information about the current state (which game they are in) and the history of states and actions. The policies are conditioned on the full state-action history. In partially observable stochastic games (POMDPs), agents receive incomplete information about the environment. The observation function takes the actual state-action pair and produces a limited observation for each agent. The history is now a history of observations rather than the full state-action history. In partially observable MARL, agents can try to construct what they think the full environment state is based on the action history of other agents. This is called a belief state, and it can be updated using Bayesian inference. However, for complex problems, this becomes computationally intractable.

Click to watch →

Grid Overview

Setup & State

Magic Squares

State Checks

Env Dynamics

Reset & Render

Wrap Up

Grid Overview

0:00

Playing Section

1
Introduces the goal of building a custom reinforcement learning environment.
2
Describes the grid world setup with teleporting magic squares.

39.1K views51820:58@MachineLearningwithPhilOriginal Release: 2019-04-02

To create a custom reinforcement learning environment compliant with OpenAI Gym standards, you need to implement several core components: (1) a class-based structure with an initializer that defines grid dimensions and magic squares, (2) state space and state space plus concepts to distinguish between non-terminal and terminal states, (3) an action space mapping actions to position changes (up/down/left/right), (4) a step function that handles movement, reward calculation (-1 per step, 0 at terminal), magic square teleportation, and illegal move detection, (5) a reset function to return the agent to the starting position, and (6) a render method for visualizing the environment. This framework enables agents to learn optimal policies through trial and error, such as discovering shortcuts via magic squares in a Gridworld environment.

[00:00:00]welcome back everybody to neural net Dante I I am your host Phil Taber previously a subscriber asked me hey Phil how do I create my own reinforcement learning environment I said well that's a great question I don't have time to answer it in the comments but I can make a video so here we are what we're going to do in the next two videos is create our own open a gym compliant reinforcement learning environment the good world it's going to be text-based and if you're not familiar with it the grid world is aptly named a grid of size M by n where the agent starts out in say the top left and that's to navigate its way all the way to the bottom right the twist on this is going to be that there will be two magic squares that cause the agent to teleport across the board the purpose of doing this is to create a shortcut to see if the agent can actually learn the shortcut kind of interesting the agent receives a reward of minus one with each step except for the terminal step or receives a reward of 0 therefore the agent will attempt to maximize its reward by minimizing the number of steps it takes to get off the grid world what other things to other concepts we need are the concept of the state space which is the set of all states - the terminal state and the state space Plus which is he set of all states including the terminal state this gives us a bit of a handy way to find out the terminal state as well as to find out if we're attempting to make illegal moves it also follows the nomenclature and terminology from the Sutton and Bardot book reinforcement learning which is an awesome resource you should definitely check out if you have not already so in part 1 we're gonna handle the environment and in part 2 we're gonna get to the main loop and the agent for which we will use cue learning not deep cue learning because this is a very straightforward environment we don't need a functional approximation we just need the tabular representation of the agents estimates of the action value function so if you're not familiar with cue learning I do have a couple videos on the topic one where the Q learning agents solved the card poll game as well as a an explainer type video that talks about what exactly cue learning is so let's go ahead and get started we only have a couple dependencies we're not doing anything on the GPU so just numpy and Matt plot line we want to close everything up into a class called grid world and our initializer will take the M&N which is the shape of the grid as well as the magic squares so we represent our grid as an array of zeros in shape and by n we want we want to learn to type we want to learn to sorry we can want to keep trying we want to keep track of the M in the end for handy use later so let's go ahead and define our state space and that's just going to be a list comprehension for all the states in the range self dot M times self dot n now the as I said the state space does not include the terminal state and the terminal state is the bottom right so we have to go ahead and remove or pop off that particular state from the list next up so now let's go ahead and again learn to type go ahead and define our state space plus also we need to know the the way that the actions map up to their change on the environment so we'll call that the action space a little bit of a misnomer but we can live with it for now so moving up will translate the agent up one row which is distance M and moving down will advance the agents position downward by also M moving the left will translate the agent one step will decrement the agents position by one and moving right will increase it by one we also want to keep track of the set of possible actions you could use the keys in the action space dictionary but let's go ahead and use a separate structure and we use a list up down left and right the reason is that the q-learning agent EQ learning algorithm sorry can it can choose actions at random so it's handy to have a list from which you can choose randomly next we need to add the magic squares because it's a little bit more complicated than it may seem and finally when we initialize the grid we want to set the agent to the top-left posy Shaun let's go ahead and add those magic squares so of course you want to store that in our object now there's a little bit of HO keenest that I must explain so the agent is represented by a zero when we print out the grid to the terminal and MD squares are represented by a one and so excuse me and so that means we need something other than zero and one to represent these magic squares I want to when I render the environment I want to know where the entrance and where the exit is so we use different values for the entrance and exit so that way we can render it correctly so just by royal decree we set I which will be the representation of the of the magic square in the grid world 2 2 to start and I'm gonna go ahead and iterate over the magic squares so now what we need to know is what position we are in so you sorry its if the color is off indicating something is wrong I have screwed something up royally which I do not see because I am blind anyway so the the exposition is just going to be the floor of the current square and the number of rows and Y will be the modulus of the number of columns so then the grid we want to set that x and y position to I and since we want to have a different representation for the entrance and exit go ahead and set the increment I by 1 recall that the magic squares are is represented as a dictionary so we're iterating over the keys and the values are the destinations so the keys are the source values our destinations so next we want to find out precisely that what the destinations are so butt in and then set the grid that Square to I and then increment hi again and I'm only gonna do I'm only going to do two magic squares you can do any number but in this case we're just gonna do too so okay so the next thing we need to know is if we are in the terminal state and as I said earlier the state space and state space plus concepts give us a very easy way of doing that so let's go ahead and take care of that so since the state space plus is all the states and the states space is all the states - the terminal state we know that the difference between these two sets is the terminal state so state in state space + and not in the state space how does that look you scroll down a bit okay so next up let us go ahead and get the agent row and column and we're going to use the same logic as above so next we want to set the state so that we'll take the new state as input and we're going to go ahead and assume that the new state is allowed so the agent if it is along the left edge and attempts to move left it just receives a reward of minus 1 and doesn't actually do anything likewise if it's on the top row and attempts to move up it doesn't actually do anything that just gets a reward of minus 1 for wasting its time so we want to get sorry the row and column and set that space to 0 because 0 denotes an MV square and the agent position then is the new state and again we want to get the new ex a new Y there is a typo there let's fix that and then set that position to one because that is how we represent the agent by royal decree so the next thing we have to know is if we're attempting to move off the grid that's not allowed the agent can only stay on the grid so let's take care of that so we want to take the new and old States as input and the first thing we wanna know is if we're attempting to move off the grid world entirely that's so I hate these editors so if we are the new state is not in the new state space plus we are attempting to move off the grid so you return true otherwise if the old state modulus M equals zero and new state modulus this self dot M equals self dot M minus 1 then we return true and for brevity and I could explain this but the video is running long already so for brevity the reason this is true is left as an exercise to the reader bet you didn't know this was gonna be like a college course so now basically we're trying to do here I'll just give you a hint what we're trying to do here is determine if we're trying to move off the grid either to the left or to the right we don't want to wrap around so so if you're for instance if we have a 9 by 9 grid it goes from 0 to 8 so then if you add one right you would get 9 which would teleport you to the other row and the 0th column you don't want that what you want to do is waste waste to move and receive a reward of minus 1 so that's what we're doing here old state modulus good grief so if neither of those are true then you can go ahead and return false meaning you're not trying to move off the grid so let's see can you see that you can so next function we need is a way to actually step so let's go ahead and do that let's say the only thing we need is to take in the action so the first thing you want to do is get the X&Y again and here's what we're going to check to make sure it's a legal move so the resulting state is then agent position plus the mapping so the agent position is whatever it is and recall that the action space is this dictionary here that map's the actions to the translations in the grid so we're doing down here then is saying the new state is equal to the current state plus whatever the resulting translation is for whatever action we're attempting to make so next thing we need to know is are we on a magic square and if we are and if we are then the agent teleports to its new position okay so next up we need to handle the reward so it's minus one if not is terminal state so it's minus one if we haven't transitioned into the terminal state otherwise it is 0 if we're not trying to move off the grid then we can go ahead and set that state so if not set state salting state and then we're ready to go ahead and return so in the opening I Jim whenever you take a step it returns the new state the reward whether or not the game is over and some debug information so we're gonna do the same thing resulting state reward and the whether or not it is the terminal state and our debug info is just going to be none so if we are attempting to move off the grid what do we want to do nothing so we want to return agent position and the reward and rather not its terminal and the null debug info we're almost there so next thing we need to know is how do we reset the grid because at the end of every episode we have to reset right first thing to do is set the agent position to zero we set the grid to zeros and go ahead and add the magic squares back in and return the agent position which is of course zero oh wow that's real close all right so next up one last function I swear just one more I promise all right I wouldn't lie to you all right next up we want to provide a way of rendering because hey that's helpful for debug I like to print a big string of you know dashes because it's party we want to iterate over the grid for column in row column equals zero in other words if it's an empty square we're just going to print a dash and we're going to end it with a tab if the column is one meaning we have an agent there we're going to print an X to denote the agent now if the column is two then that is one of the entrances to our magic squares so print the a in with a tab-delimited a tab end if the column equals three you print a out and he goes tab and if the column equals four then we know we're at the other magic square entrance and finally if it's five then we know we're at the other magic squares exit after each row we want to print a new line and at the end we'll go ahead and print another chunk of pretty dashes oh yeah that's it that is it okay so that is it for our agent class that only took how long 20-some minutes Wow okay I hope you're still with me basic idea here is to make your own environment you need and initialize a reset a state space state space plus a way to denote possible actions a way to make sure the move is legal and a way to actually effect that environment the step function needs to return the new position the reward whether not the state is the new state is terminal as well as some debug information you also need a way of resetting and printing out your environment to the terminal so in part two we're actually gonna fire this baby up with aq learning algorithm and see how she do that's actually quite exciting it moderately exciting anyway it actually learns it does quite well and it does find the magic square spoiler alert if you made it this far it finds a magic square and gets out of the minimum number of moves required it's pretty cool to see so that will come in the next video on redness day I hope to see you all then if you liked the video make sure to leave a thumbs up subscribe if you have not already for more reinforcement learning Khan and I will see you all in the next video

Up Next

MIT 6.S191: Convolutional Neural Networks for Computer Vision

@AAmini

176.7K views•2025-03-17

Building Real-Time ML Pipelines with Feature Stores and MLOps Frameworks

@ODSCAI

5.1K views•2022-02-20

PPO Implementation With PyTorch | Reinforcement Learning Tutorial

@MachineLearningwithPhil

85.7K views•2020-12-24

Neural Networks Explained: Math, Layers, and Learning Fundamentals

@3blue1brown

21.9M views•2017-10-05

Related Study Plans & Knowledge Roadmaps

Structured learning paths in Artificial Intelligence

View All Artificial Intelligence Topics

artificial-intelligence

Building Recommenders: Two-Tower & Vector DBs

01Foundations of Python & Machine Learning Basics

02Introduction to Recommender Systems

03Deep Learning & Representation Learning

+ 3 more learning modules

Explore Roadmap

artificial-intelligence

Latent Diffusion: PyTorch, UNet & CLIP

01Deep Learning & PyTorch Foundations

02Convolutional Networks & Autoencoders

03The Math and Mechanics of Diffusion Models

+ 3 more learning modules

Explore Roadmap

artificial-intelligence

NeRFs: 3D Reconstruction & Instant-NGP

01Math & Python Foundations for 3D Vision

02Photogrammetry & Camera Pose Estimation

03Neural Radiance Fields (NeRF) Core Concepts

+ 2 more learning modules

Explore Roadmap

artificial-intelligence

TFT Forecasting: Energy Demand & Deep Learning

01Programming & Data Science Foundations

02Introduction to Machine Learning & Time Series

03Deep Learning & Sequential Models

+ 3 more learning modules

Explore Roadmap