Building a Custom Reinforcement Learning Environment with Q-Learning

[00:00:00]what is going on everybody and welcome to part 4 of the reinforcement learning series in this video we're gonna be doing is building our own cue learning environment so the first thing I wanted to do when I learned to learning yes it was useful to use the open AI gym environment but the first thing I wanted to do is make my own environment and I didn't really intend to make a tutorial out of it but people were asking and then it's kind of like well obviously that's what I wanted to do so of course that's what other people probably want to do too so anyway here we are let's make our own environment so I'm actually just going to kind of run through the environment that I made and explain it as I go so well probably we're just going to do the program linearly but if you have questions whatever as always comment below join the discord GG slash syntax and feel free to ask any questions but it should be pretty self-explanatory it's not a super complicated environment and just in case I guess I'll explain it a little bit before we get in but so first of all I love blobs every I use blobs for like examples of everything so if you haven't noticed by now I'm a blob a file anyway so I always want to make blob things and so the idea was to make blobs and in this case you've got a player blob of food blob that's the objective we're trying to achieve and then just for you know complexity I added an enemy blob as well so the idea is that you have this player blob and and we'll start by having the enemy and the food not move it just stays stationary and they just kind of initialize in random positions and then the the player blob has to move to get to that position and then later we can add in movement it's just that letting the things move is kind of irrelevant to the model learning how to actually move except in one example which is so if the enemy is moving and you're moving you could in theory move into each other if the enemy is not moving in training you you you might learn that it's just totally fine to get close to the enemy as long as you don't hit the enemy but if the enemy can also move they could collide but anyway not really worried about that but you guys can feel free to tinker with that anyway let's get into it so we're gonna be using OpenCV so the first thing that I want to do is open up a command prompt and let's go ahead and pip install Python - open CV let's make sure we have that is it open it might be open CV - Python open CV - Python okay I feel like I always get that wrong anyway okay pip install OpenCV - python if for whatever reason python - open CV did install for you you'll probably just installed something really bad so don't use it get rid of it moving along so alright so we are also going to use numpy but you should already have that and then we're gonna use the Python imaging library and I want to say that would be a pip install pillow this is a virtual machine and it should be clean so if that's not what I want we'll find out soon enough but we'll get red pillow and then to use pillow you import capital PIL so that should be everything we need let's go ahead and import import numpy as NP sorties numpy for the array types of stuff from PIL we're going to import import how to type image with capital I import CV - we're gonna import matplotlib pipe lot as PLT we're going to import pickle and that's to save and load our cue table then we're going to from matplotlib will import style just to make our graph pretty and we'll do style dot use ggplot and then finally we will import time we're gonna use time purely to set dynamic queue table filenames and so it has like some order what's your deal probably - under 500 is its type of style huh okay great so the first thing I'm gonna do is just try to run this real quick make sure it actually runs it does okay so all our imports worked and we're good to go so so now we're gonna have some oops did I just delete totally did okay so now we're gonna start with some of our constants so we're going to say size and we'll say this is 10 so we're going to skip the whole make a huge environment and then boil it down to action or observation spaces discreet observation spaces I'm gonna skip that step I'm just gonna make it a grid so in this case it's going to be a 10 by 10 grid so the player the food and the enemy will be initialized at a random location on a 10 by 10 but we can change this as time goes on I'll talk about how some of those changes will impact things but it obviously as you increase this size especially depending on the size of your action space as well that is going to just exponentially explode the number of you know possible combinations in your cue table so anyway we'll start with a 10 by 10 that should be pretty simple how many episodes so HM episodes will say 25,000 we're going to say it we're gonna give a move move underscore penalty that's gonna be a 1 an enemy underscore penalty so this is if we if we hit the enemy we're gonna say is a 300 so we'll subtract that penalty basically we're going to say a food reward 25 this was I haven't really decided where I want this I don't know if I really want it to be 1 0 like we had with the mountain car 25 I haven't really decided I don't really know what's the best way to what's the best one to use so I'm just gonna throw in 25 I haven't noticed anything you know huge to tell me one way or the other then we're gonna have epsilon lowercase because it's gonna change over time we'll start at 0.9 another thing that we could change a ton EPS decay so it's epsilon decay zero point I went with triple nine eight and the way I came into this was I just made a for loop and I literally just decayed so I just took epsilon times or I'm sorry for you know for I in range say 25,000 multiply the I just kind of looked and I was like okay that looks good so so I literally just pulled that number out of nowhere moving along now we'll say show underscore every so this is just like before how often do we want to show I'm gonna say every 3,000 episodes if I recall right this actually my environment is actually a little quicker than the mountain car one okay show every 3,000 okay so now what we're gonna say is start will have a variable here start cue table table and for now we'll say none but then it could or a file name so if you happen to have an existing cue table that you want to like load in and train from that point you would throw that right in there now you might want to do this for a variety of reasons you want to continue training or or you know at least we haven't really coded in a good way to like decay epsilon to zero in them you know train for a little bit and then reintroduce epsilon we don't really have a good way to do that so if you really wanted to do that you could decay epsilon to zero or train for a little bit as you know with very low or no epsilon and then load it in a DEP salon and continue and I've actually found that that works really well to like decay epsilon let it go for a little bit and then set up salon back to 0.9 or something indicate again and then let it train for a while and then decay again and keep repeating that that that seems to actually learn pretty well so anyway for now it's none because we don't have a cue table but if we had one we just passed the filename so now we'll throw in learning underscore rates we're gonna say zero point one discount we'll set again zero point nine five these two variables I really haven't played with much so couldn't couldn't really tell you how it's going to impact things but now that you know how to do analysis from the previous video feel free to thinker with them now what I want to do is I'm gonna give a player N equals one we're gonna say food n these are just numbers for labels and a keys rather in a dictionary that's it so clear and food and end enemy n3 again it's just definitions for what number of these things represent in a dictionary so we're gonna say D equals and then we're gonna have one colon something well I'll just fill it in so these will be the colors so 255 175 0 now I'm actually defining these colors in BG or format even though I'm really try was really trying to use RGB so side-project is if anybody can tell me why is it BGR because i don't think it's like the I don't know maybe I'll figure it out as we go but that's that was a problem I haven't solved yet anyway it's 255 0 so let the player be is mostly blue some green so it's kind of like a lightish blue food is B G so full green so the food is green and then the enemy will be 0 0 255 so BG R so maximum R so it'll be very red cool so you don't like that huh all right I'm surprised it doesn't surprise it's accepting my spacing over of the dictionary I don't think that's Pepe maybe it is must it must be if it's accepting it ok so now what we need is we need a blob class because really all of these blobs are gonna have a lot of the same attributes at least they're gonna need to be able to move they're gonna need a starting location like they need to be initialized randomly they're gonna need to be able to be moved and then later basically our observation I didn't really want to use because it you would have like a huge observation space if you needed to pass what is the location of like I felt like it would you have to either the problem would be much more complex I guess if you passed these the physical location of everything so my plan is instead the observation space is actually going to be the relative position of the food and then the relative position of the enemy to the player that's going to be the observation so to get that we need to be able to form some sort of a operation that will produce that value so anyway we're gonna make a blob class so that we can handle for that I get asked all the time hey why don't you code an object or in your programming all the time and it's like well I just use it where it's at actually useful and in this case it seems to me that making a blob class is actually useful so anyway class blob so define our init method and if you don't know object-oriented programming in Python I would go to Python programming dotnet go to Python fundamentals intermediate Python and then in this series we get into object oriented programming here and I would strongly recommend you you go through this if it's any if it's confusing to you and then we also are going to do some operator overriding and stuff like that so you might want to check that out if that's confusing to you so or init method this is just gonna run immediately so we're gonna say self dot X is going to be equal to numpy dot random dot R and int between zero and the size all caps and then we're gonna do the same thing for self dot Y so that'll just initialize randomly it's gonna cause a slight issue so if well in one case the player could spawn right on the food that would be lame the the other one is the player could spawn right on the enemy that would also be lame and finally the enemy could spawn on the food or the food on the enemy server the way you want to look at it glass half full half empty kind of thing those with that that kind of stinks that that's a possibility I'm not gonna handle for that because I don't really feel like writing the code for that define we're going to make a string method I'm not even sure I use this for debugging purposes but then I don't not even sure we need it now but anyway I'm gonna return an F string of mmm self dot X and then we'll just say comma self-doubt why so if we want to print the blob it'll just print the blobs location so now what we want to do is I'm gonna override define the subtraction operator here or over I think it's operator overloading anyway someone will correct me actually anyway return so this just allows us to subtract a blob from another blob so in fact we need another blob so I'm gonna say other so we'll Pat so when we subtract when you do it - and then the thing that thing will get passed here and then in here we're going to say return and in this case what we're gonna say is return and then in parentheses we're gonna say self dot X - other dot X comma self dot Y - other dot Y cool it looks like Pepe doesn't care either way if you want to space here or not I'm gonna give it a space okay so that's our subtraction operator overloading I think it's operator overloading I'm gonna go with that now I'm going to define basically this blob needs to the player is gonna take a discrete action right so we're gonna have an action method but really the player is the only one that's gonna use it but there might be a time where you want to have many players so I'm just gonna I'm gonna go ahead and do this like in this like q-learning like we're not going to have many players I don't think I can't really see a good reason why I would do that but later we might want to do that and then so action will actually interact with another method I'm just gonna pass for now and we're going to call this move method and you know it'll be self and then probably x and y and in fact I'm gonna set X&Y just default to false before I forget okay so in action we want to be able to pass a discrete value so we'll say self choice and then we're just going to if choice equals zero now we're gonna start taking some actions so we're gonna say self dot move and we're gonna say X equal x equals 1 y equals 1 so it's gonna move 1 in 1 now what I'm going to do is I'm just gonna copy this paste this and this will be L Elif choice equals 1 and we're gonna say negative 1 and negative 1 and then again we're just gonna copy paste paste set this to to set this to 3 and then negative 1 and 1 and then 1 and negative 1 and those are gonna be all the choices possible I wanted to do this just to keep the action space relatively small but this means the player can only move diagonally now if I had thought of this when I was coding this I wouldn't have continued I would have actually done each of sorry I just punched the mic I wish I wouldn't have done that I would have also added all of the X only in Wyoming movements as well so it could move up and down because to me I would not have expected q-learning to be able to handle for this but like I said I actually made this environment before I actually thought I was gonna show you guys this environment so I already know it's gonna work and it works by using the wall so if it gets to the wall and then tries to move diagonally it it moves up or down right or side to side depending on which wall it's at so so it still actually works and I'm still amazed that it learns to use the wall because as you'll see it has no idea about the wall it doesn't know there's any boundary to its environment so the fact that it learns to use the wall is fascinating and I don't know if it's because because it's not smart enough it's not like deep learning where if it it knows it's on evens or odds and then the food is on and off ok let's say the X is even for the agent and then you've got like you know like it's a exact combination that like a deep learning model could figure out and be like oh we need to flip this let's go to the wall but for q-learning it really I don't think I should be able to learn that because this is not how it works but it actually does work in this model can be trained for like almost a hundred percent because like in theory half of the time when it gets initialized randomly the food should be in a location inaccessible without using the wall but this model actually learns to succeed almost a hundred percent of the time so that's mind boggling but anyway if you want to add more choices go for it I'm gonna leave it this way but just take note that it can only move diagonally and it's amazing so now let's code the move method and basically we want to be able to move randomly if that's if there's no value passed here or move very specifically so we're gonna say if not X then what we want to do is we're gonna say self dot X plus equals NP dot random dot R and int between negative 1 & 2 so it's from negative 1 up to 2 so that just means that we'll either do a negative 1 0 or a 1 so keep in mind it has the ability to do zero so this random movement can move up or down fascinating if not X else self dot X plus equals x then we want to do the exact same thing for the Y so I'm actually just gonna copy this come down here paste so why why why and why why though okay so now so that's it so that's handling for X&Y the problem here now becomes when it hits the wall so recall our size is a 10 we're trying to make a 10 by 10 grid so if the agent attempts to move beyond that we need to need a thort it's same thing with the food if we allow the food to move spoiler alert I'll will turn on movement you'll get to see it or at least I'll show you video of it it'll take probably too long to do it but you can train it on your own so okay so if self dot X is less than zero because it could be at the zero position it just can't be less than zero we're gonna say self dot x equals zero no colon required there L if self dot X self dot X is greater than the size minus one rather so again the size is 10 but the positions are 0 through 9 so there's 10 positions but so that's why so if it's greater than 9 we're actually at the 11th position so we say size minus 1 there then what we're gonna say is self dot x equals size minus 1 we need a fix sighs there okay now we're going to do the exact same thing for y so say again y and then we'll do why why and why fabulous so now that should be everything we need for our blob class which I will now refer to as blob jects which you've altered if you've seen any of my tutorials with blobs so now what we want to do is either create the cue table or load the cue table so what we're gonna say here is if start cue table is none so if we don't have the start cue table we need to create one so we're gonna say Q underscore table equals a we're gonna just make it a dictionary we could we could do like something similar to before but we're gonna do it different so all we're going to say here is for I in range of negative size plus 1 to size this needs to be all caps size so so again negative size plus 1 because we again we need to shift and then size and we're not doing minus one here because all the ranges are always up to so we're not actually including the 10 so we were going to do that now what we're going to do is do this three more times so we've got one two okay because we need every cut so think of it this way we have two coordinates right so we have the basically the observation space is going to be two coordinates where the first one is going to be the Delta to the food so the relative difference because we're going to subtract the blobs right we're gonna subtract these two blobs from each other so where we overrode here so it's going to be the subtraction of the plate current player about the player to the food so that's going to return one tuple of x and y-values and then the other one is going to be the difference or the subtraction or the Delta of the player to the enemy so it'll be another tuple so that'll be XY so it'll be like x1 y1 and then x2 y2 okay it'll look like that that's our observation space now to get every combination we're going to iterate through them so we've got I I okay we could also get fancy we could say x1 y1 x2 y2 okay that might make a little more sense since I just showed that other example so now we have every combination we want to add every combination to our table here so so the way that we're gonna do that is I'm gonna look at my code because I want this because this is something I would I didn't say it I would fudge up cue table and then the for the table we're actually gonna pass the tuple so it'll be a tuple and I changed my values so I already kind of screwed myself here like this right so that's gonna be the key is a tuple of tuples so we're gonna say x1 y1 x2 y2 and let's fix this right okay cool so that will save that out you hmm there you go Pepe okay so then now we just need to initialize with random values so what kind of we'll just initialize the same way pretty much as before so that's going to be equal to and then so basically each each observation space needs four four random values because we have our action space is four we have four discrete actions surface a MV dot random dot UNIF uni form and I'm gonna say negative 5 to 0 and then we're gonna say for I in range for cool I think that's everything looks good make pep eight happy except for the line limit petition to remove the line limit from pep eight there's just like so many times when the line limit is stupid and it like it would be dumb to to not exceed the line limit okay so cute eh okay so we've got our key table values that's it our cue table is initialized now we're gonna say else with open we basically well we're gonna do start cue tables so it's that file name RB as F we're gonna say now cue table equals pickle dot load F so if we have one pre-trained awesome will load that or rather if we don't have one pretty trained fine we'll make one otherwise awesome blowed that so then we just have our Q code or our training or whatever you want to call it to do so we're gonna say for episode in range of how many episodes do we want we're going to say the player equals a blob the food equals blob and enemy equals a blob easy then we're gonna say if episode modulo show every is equal to zero we're gonna say we're gonna print an F string on number episode just so we know kind of where we where we are and then we're going to say the epsilon also so we kind of know where we are epsilon wise then we are going to let's just print one more we're going to say whoops show every episode clips mean and we're just gonna display the mean episode reward so we're gonna say NP mean and that's episode underscore rewards negative show every : I just don't recall making the episode rewards I don't think we did oh now it I saw it show up initially anyway app is oops episode underscore re words that's just gonna be an empty list okay and then the other thing we'll do is show equals true else we'll set show equals false and this is just so we can actually display the thing so now we're going to get to our actual so that's four Episode now what we want to say is episode underscore reward duh equals zero because we start at zero and then for I in range we'll say 200 these will be how many steps we take I'm hard coding that you could later add that as a parameter if you'd like so the first thing is we need an observation so we're gonna say abs equals and that's going to be player - the food comma player - the enemy so that's our operator overloading in action then we're gonna have our random movement stuff so we're just going to say if NP dot random dot random is greater than the epsilon as long as that's the case we're gonna do a regular action so we're gonna say action equals NP arg max of the queue table at the location of our observation otherwise if that's not the case we're gonna say else action equals NP random dot Rand int between 0 and 4 again it's up to 4 so it's 0 1 2 or a 3 so now that we've got the action we're ready to take the action so where is a player dot action action so we take that action which based on the action passed moves based on these things that we passed and again I just kind of wanted to separate these because sometimes like you might want to move a thing maybe that thing we want it to move more than one value for whatever reason like the movement and then the action to move I think should be separate for this simple example at the stage that we're at maybe it's unnecessary but I think it's good moving forward anyway no one was complaining about that and then what I'm gonna do is I'm just gonna say maybe maybe later we might want to say something like this enemy dot move and then food dye move right you might want to let them move that would complicate things I think for training purposes it's better to not let them move initially Excel just like really complicates things I think again haven't actually tested either way I just initially thought no no point and then later we can we should be able to add movement and the agent should be able to handle for that pretty well except for the one case that I was explaining where what about you know in training if the enemy can't move it's okay to even like brush alongside the enemy but if the enemy could move you wouldn't want to take actions like that'd be too dangerous because it could move into us so now what we're going to do is code for the actual reward process so after we take our action what was the results so what we're gonna say is if player dot x equals enemy dot X and player dot y equals enemy dot y you done screwed up reward equals a negative enemy penalty remember the negative otherwise your agent is gonna love the enemy okay so if that's the case now we're gonna say Elif because we're gonna assume like we definitely don't want we we always want to avoid the enemy so if the enemy is sitting on top of the food or something we we'd rather just avoid the enemy hmm so Elif player dot x equals food on X and player dot y equals food ly then we're gonna say reward equals what was it food reward yeah food reward all caps else so if those aren't the case reward equals negative move penalty so I think it's negative one great now to do our Q function we need to know we need to be able to like move we we need to make a new observation based on the movement so we're gonna say here is new underscore observation equals because we've we've made the move so when we took that action now we've moved so now we're going to come up here and say the new observation is equal to player - food player might player - enemy and yeah and then after that move we need to we get we need to grab the max future queue that's equal to NP dot the max value of Q table for the new observation so that gives us the maximum future queue and then the current queue is equal to RQ table at table at observation and the action that we took because it won't necessarily always be the the maximum or the Arg max or whatever so okay so current queue max not ordered max anyway okay so now we can actually calculate our queue so so now what I would say is if reward is equal to food reward then the new so if reward equals food reward new underscore Q is equal to food reward because once we've achieved food we're done the same thing would be true l if reward equals enemy penalty if reward that should be equals negative enemy penalty if L afford equals negative penalty then we're gonna say new Q equals negative don't forget it I really badly want to make that mistake else new Q will be the Q formula so that'll be equals 1 minus the learning learning underscore rate 1 minus the learning rate times the current Q current Q plus the learning rate times parentheses reward plus the discount times the max future Q let's fix our pep 8 cool so that's our new Q function I'll zoom out pause if you need to catch up okay so once we've got that we want to update the Q table so we're gonna say Q table at Bob's action equals whatever that new Q is okay so at that point we're actually done with the Q learning now what we want to do is show the environment if we want to show it so because we kind of want to see it over time and then also we want to track some of the metrics and then maybe graph or whatever as needed so Q table ops actually using new Q so the next thing is if show now we want to show it now we haven't even created the environment but it's just a grid so it's pretty simple environment so we're gonna say the the base environment it's equal to NP zeros and it's a so actually this will be a tuple here it's a size by size so a 10 by 10 by 3 because it's RGB data and then we're gonna say D type equals NP u int 8 so this is just basically 0 to 256 0 to 256 by 3 so let's fix that so now what we want to do is so it's all zeros so this would be an all black environment basically and then we're just gonna mark so we're gonna say N and we're gonna save food dot X food dot y equals and I'm just kind of thinking now I think actually in so in an array it's at the Y by X I think rather than X by X Y as we would traditionally think of X Y right because you've got your depth then your with basically so I don't think in the environment though it would really it's gonna matter like it's not gonna make any difference whether we flip that or not because one it's a square and two we're just trying to visualize so it would just be like rotated I think right so I don't think it actually matters yeah it would only matter if we were allowing use or input but I actually think it makes better sense to say Y X off to fix that in the tutorial or the text-based version anyway I think that's correct I think it would be Y then X in any array anyway so where is a food Y food x equals the dictionary food and so that's just our way of getting the color initially I had a really great reason for that but as long as we only have one food it kind of makes doesn't look you could just hard code this or you could just say food color like so you could just say the food color is something I had bigger plans for the dictionary and then it changed so anyway that would probably make more sense but just show must go on so we've got food then we have the player so we're gonna say player player Y player X is the player player in and then this will be the enemy and move this mouse enemy and and then here will be enemy dot y and then enemy dot X okay so that'll change that grids color and by grid I really it's really an image now so it's a it's an RGB image so but Italy it's only a ten by ten so so the next thing we're gonna do is we're gonna make this an actual image image and we're gonna say image equals capital I image dot from or a and then n and then here's where I'm saying hey this is our G frickin beat but for whatever reason it doesn't care it's BGR and the way I found that out is by is yeah I'll let the player moves and the food and the enemy doesn't move and it's so clear it's BGR it's 100% BGR but it's crazy anyway not n its n Vividh so that gives us our image now what we're gonna say is image equals image dot resize and then we're just gonna resize this - you can pick anything you want I'm gonna say a 300 by 300 just so it's like I can see it and then see v2m show and then the title is whatever you want it to be numpy array image and then I'm gonna throw in some kind of shoddy code here but that's okay everybody accepts it if reward is equal to the food reward or the reward is equal to a negative enemy penalty this means the simulation ended and we we really screwed up so so not if we didn't get the food but if we got the food no we didn't screw up necessarily if we got the food yay or we hit the enemy really bad so if that's the case I just wanted to pause it just for a moment longer so I'm just gonna say if C v2 dot wait key and in this case it's 500 milliseconds and then this little batch of Awesomeness oxf F equals Ord Q basically if you hit the Q key it breaks I'm gonna throw that in and this is just how you let that cv2 window just refresh live basically so if we either made it to the food or we really screwed up by hitting the enemy we want to run that otherwise else we'll just say wait key one so it's frame-perfect frame and then at the very end if we either get the food or this should be a double equals should not be an assignment yeah okay so the frames will go really really quick unless we hit the enemy or get the food and then we'll just hang there for a second just so we can catch it because this is one millisecond so well it's not actually gonna be 1 millisecond it's gonna take a little longer than that but it's gonna be quick so so that's our show so now we can see the environment now what we're gonna say is episode reward plus equals reward and then if reward is equal to the food reward or reward is equal to the negative enemy penalty it's over we break and then finally in our episode episode episode loop so just one tab over we're going to say episode rewards uh dot append episode reward and then we're gonna decay the epsilon so epsilon x equals our EP s decay and then finally we just want to graph things at the very end so I'm going to say moving underscore average is equal to the number of episode rewards uh so this will just create a moving average and numpy ones show every and then divide by show every and then does this mode I don't even know let's tab that over yeah mode is valid and I believe that would be a mode equals valid so if you don't have enough to do the calculation should you still do it or not I'm pretty sure that's what it that's going to be but I could be wrong anyway it's just a way for us to make a moving average that's totally a Google operation there because you wanted to be like a rolling moving average you could also do it in chunks like we did in the previous environment I just think a moving average is a better idea so anyway there's a moving average and then we're going to PLT dot plot and then for X I'm just gonna say I for I in range of the Len of moving average and then for our Y we will plot moving average beautiful and then PLT dot y label we're gonna say reward and then show every so the moving average that we'll use is whatever the show every is so if it's 100 if it's a thousand whatever it'll be a thousand moving average and so on PLT e-elt dot X label we'll say this is our episode number PLT touch show and then finally with open and then we'll make an F string and we're gonna say cue table - int value of timed time dot pickle with open that pickle comma WB as F pickle dot dump and we'll dump our cue table into F hmm woo okay coded that all the way through how many reward on food reward Oh so this should be a lower case lower case reward so what are the odds that I coded this without any other errors I don't know I don't know we're gonna find out though we're gonna find let's go into here CMD Python Python q-learning 4 come on baby ok so it looks like we do very well there do we have an error are we still making it wording invalid I think I think we're fine cool awesome show me my environment dang it so so yeah this is your enemy this is your food this is your player so it's running up Oh looks like we got at that time so if you see blue and red really quickly that means we got to the food if you see green and red that means we ran into the enemy at the very end so everything's happening sometimes very quickly so we're only on episode what 9000 yeah so he's learning quick ok he's doing pretty good I forget how many episodes we said to go on probably 25,000 I think yeah he's moving fast so anyway though the order that we drew things the reason I made that order was so I could figure out what happens at the end so these are all victories so this is pretty cool to see I'm just waiting for the end so we could load it in and I can show you guys you know how we can start to kind of tinker tinker with things I also can bring up the text-based version of this tutorial for some other longer simulations as well ok so here's our graph we can see here we started these this is basically the rolling average of our accuracy or not really our accuracy are rewards basically oh we type out reward anyway cool so you can see it definitely learned things really quickly and again half of the time it should not be able to get there but it uses the wall to get there which is super cool ok so what I'm going to do is close out of here guess I'll just oh oh I had to save ok so it saves our cue table so as you can see our cue tables like thirteen point six megabytes as we continue to train because we're using pickle and a dictionary that will expand over time and you can actually use that expansion to know how many more random or yeah how many more random initialized values are being updated kind of an interesting outlook on my flaw of using a dictionary and pickling a dictionary as opposed to using numpy correctly anyway yeah so oh so so let's say okay we like what we we think this is good so then we could load in the cue table so let me copy this and then rather than none will load in the cue table itself so now we'll load this and then now let's just set epsilon to zero what else would I need to change trying to think I think we're good actually let's say show everyone I think that's everything I'm probably missing something but let's run it let's see how we look so now you're seeing it play every time he's doing really good get it get it get it Oh beautiful beautiful again more ah solved that's pretty cool okay so I think I think he passed around so those those worries like bouncing around that's a perfect example of the failure because he can't move I see he oh man either so cool but a lot of times you can see it actually go it specifically it could have gone straight there but instead are not really straight so I can only move diagonally but it could like zigzag right but instead it goes to the wall and bounces off the wall and I I just can't believe it's learned behavior like that I just I just can't it's crazy but anyway it has so now what I want to do let me break this always the other thing I wanted to show Oh let's turn let's turn movement on and see how we do there so let me just do this whoops this let's save that and then run it again and so so he died there he's surviving now pretty well he's catching so fast that's just crazy died there once died again ouch anyway yeah that I just think this is really cool so I'm super happy that I made the environment and I hope you guys are as impressed as I am with this because if I made a neural network that was like I say a continent and it looked at this and it tried to make these learn to do this it would take a while and in this trained in like I don't even know like what three minutes I just think that's pretty cool I mean I understand its basic stuff but I think it's really cool so so now we could take things further and like we could make a bigger environment we could do more steps or whatever you want to do and so feel free to tinker with the environment the next thing or the only other thing I want to show you guys is some of the results of using much much larger environment so in so we're doing a 10 by 10 if you just change that to a 20 by 20 your cue table size goes from about 15 megabytes to like 250 megabytes just for comparison's sake and then if I figure out where the heck I promise you there's some cool videos and photos we just have to get to them so let me see what this one is this one's probably what we were just looking at yeah and then I just showed how to how to get it live how to get movement blah blah blah blah nobody cares here so here I start to actually train for a 20 by 20 space so this is a 20 by 20 with no movement I want to say yeah so just seeing could it actually learn a much larger environment and obviously it does pretty well and then I turned on movement for both the enemy in the food still no problem and some of those are pretty complex where it's got to go around the enemy or deal with the enemy which again it's pretty cool that it can even do that and I almost wonder if allowing movement makes it easier because he's always trying to get toward he he's always trying to get towards the food and since the food is capable of moving in ones like up-down left-right not just diagonally that that might actually even make it easier for the for the agent let's say rather than he okay so that's our environment I hope you guys enjoyed if you've got questions comments concerns whatever you think I did something wrong feel free to let me know I I didn't even think about all it can only move diagonally how is it actually getting food until I showed the code to Daniel and he's like oh it could only move diagonally how does it get the food like half of the time and then I'm like oh oh that's so cool because it uses the wall but it has no idea about boundaries like it doesn't know about the wall so I'd have no idea why that works because again it's just a cute it's like brute force right it's just the fact that the how do I just don't get it I don't know how it could learn to use the wall or if that's just random like if the wall being there it's just random like it hasn't actually learned to use the wall it just happens to hit the wall you know that could totally be the case but anyway yeah that's pretty cool because that's pretty complex behavior that I'm surprised it would learn but anyway pretty cool it does it apparently does so that's it yeah like I said questions comments whatever feel free to leave them below if y'all want to support the channel you can click that blue join button get some shout outs good a special rank in the discord channel special channel hangout in and support me in the free education and all that kind of stuff so check that out if you want otherwise I will see you guys in the next video where we will be getting into we'll probably be leaving basic Q learning behind and moving into DQ ends or deep hue learning so I will see you there