Deep Q-Networks: Reinforcement Learning with Neural Networks

[00:00:00]okay so so far Peter and rocky have done a great job covering a lot of the basics and fundamentals and now we're going to basically jump into deep RL proper now and just yet just to introduce myself I'm Vlad me I'm a research scientist at google deepmind and this lecture specifically will be about work we did on deep Q networks a few years ago okay so a quick recap of the last lecture so we looked at Q learning and the basic form of key learning is in the an agent now is interacting with an environment so you're collecting experience online and typically what you get is a a temple or a transition where you have a state in action you took the next state and from that you get a reward from the environment so in key learning you define a target for your current Q value which depends on the reward you got and the Q value of the best action and the next state and really key learning is just iterative regression where you're fitting Q values to the targets you computed and as we saw there is basically a nice connection that you can see both tabular key learning and key learning with function approximation is optimizing a lot of squared loss function on these targets so the question is okay if you want to apply Q learning to real-world problems where you maybe need to take in your observations through a camera with images what can you do so the way we do vision these days is with neural networks of course so can you actually do Q learning if your Q function its parametrized by a neural network and that will be what this lecture is about so it turns out that yes you can make you learning work with neural networks but it requires some care and some tricks so I guess again to look a little bit more closely a tabular key learning we have this loop where you're collecting experience so taking actions and getting next States and you're computing a target at every step if the state is terminal your target is just the reward you got this is important I remember this for your labs if you forget to basically treat the terminal stage separately your key learning might not converge because it doesn't have sort of this fixed target to fit at the terminal states and it can essentially blow up so if the state is not terminal your target is the reward and the discounted value of the best action in the next day and then the update at least in the tabular setting just changes the value of just this action in just the state to be closer to the target so what happens if we do approximate key learning so now our cue function is parametrized by a function approximator so let's assume it's in there all network so the algorithm looks essentially the same we're collecting experience we compute targets in the same way but now we do this gradient update on the function approximator so how can this go wrong and there are two specific ways which you can get in trouble so the first is that you're now fitting basically your cue values are computed by the neural network and neural network neural networks generalized between states so basically when you change the value of one Q value so the key value for a specific state in action because a neural network generalizes it will change the values of the other states and actions and you end up with this non-stationary aspect so if you're computing the value if you're looking at the Q value for the next state and the best action in the next state even if you haven't changed this key value yourself in the previous iterations it will keep changing because you're applying updates to other key values so because of the state aliasing or generalization your targets are inherently unstable the other thing to note is that we have this loop and as we're collecting experience online we now have correlations between targets and time so if you look at stochastic gradient algorithms and theory for them and basically how it all works in their own networks there is a common assumption that the data is iid so you sampling data from some figs distribution but in RL as you're exploring an environment maybe you know one trajectory will go into one part of the maze and another trajectory will go into another part of the maze so what ends up happening is for one trajectory you might have a lot of correlations because you're looking at similar States but in another trajectory you'll have very different states which are similar to themselves so you essentially have these large swings and distributions in the States which introduced these weirds correlations and the targets and in your data so if we want to successfully apply key learning with neural networks we have to somehow deal with these problems so I'm going to cover the deep Q networks this is a method we developed for specifically for playing Atari games with our own neural networks and 2013 and when we started looking at this problem the idea was well let's try something simple key learning is to me kind of the most basic control method that could possibly work on this problem so we wanted to see if we could make it work and the main motivation behind the specific approach was well we know supervised learning work Steve so even in 2013 we knew how to fit confidence we knew how to train their own network so can we make reinforcement learning look a lot more like supervised learning and apply our knowledge from training they're all nuts so the training method for training DQ networks essentially does key learning with two specific tricks for stabilizing training so the first idea is to use something called experience replay so instead of applying our updates online on the data as it comes in what we do now is we take data so you take an action you get the reward the next state and we put this into what's called the replay buffer so this can maybe hold something like the last million frames you've seen so you just put this data in this buffer and when it comes time to learning you sample a bunch of transitions a mini batch of transitions from this experience replay buffer so what's happening now is by something say uniformly from all your previous experience or from your recent experience you get a much more stable update so you get something that looks like you essentially have something that looks like a data set it's changing because you keep taking new actions and throwing away the older experience but it's changing slowly so this essentially breaks these local correlations you get from doing on nowadays we know sample mini batches of past experience and this is a much steadier learning signal than doing these online gradient updates now the other idea was to specifically for dealing with this correlation between your Q value and the target so here the idea was to not use your current the ways of the current weights of your neural network to compute the target but instead to have a separate copy of the weights which is held fixed for some amount of time like say maybe 100,000 updates or something like that and to use these fixed weights for computing the targets for this period of time and then we would periodically update them so why is this good well if we fix these target weights then basically we have a fixed Network that's producing our estimates and we have this experience replay buffer so we have something like a fixed data set from which we're sampling data so now what we're really doing is we're fitting these key values to something that almost looks like a fixed set of targets and this really is yeah this helps train than they're all Network much better because you're almost in the supervised learning case you have these targets and now of course we're doing our L so we have to keep updating our estimates of the Q values with our better estimates as we get them and this is why yeah an experienced replay you keep saying the last 10 frames so you keep throwing away the older ones and taking new ones and then you're also periodically updating your Q function by taking the latest weights and copying them into the target Network and then you start fitting this better improved q-function yep question you can still do this on a robot basically yeah if you're running a robot that can send data off to a server and then the learning might happen on the server and it would periodically send the weights back yeah so this is I guess where the art comes in yeah so you definitely don't want replay buffer that's too small so if it's too small say 10,000 transitions you might end up the the problem becomes too non-stationary again so I guess yeah when we run dqn on a new problem we typically take a million transitions that seemed reasonable but yeah you might I don't know if you'd need more than that you might be able to get away with less in terms of mini-batches yeah we always use them in a batch of 32 if you go higher I don't think it will necessarily get worse so that one I think is less less fiddly like I can I don't imagine that you have to adjust it too much from problem to problem like 32 might be fine yeah so I'm trying to remember the exact number it's something like every 100 now every time every 10,000 frames maybe or every 100,000 frames this is again something if if you this one is kind of important if you set it too high like a million then you're not incorporating newer information into your target so you won't learn and if you set it too low like a hundred or something then again you're incorporating it too quickly and it becomes non stationary so I think it's on the order of like tens or hundreds of thousands you could try a couple of values yeah it does involve experimentation probably yes there are interactions between these parameters but yeah we haven't noticed that being - yeah you probably don't need to worry too much yeah but Dad augmentation so yeah we tried this an Atari a while back and surprisingly yeah it didn't really work so and supervised learning you had like nose to your inputs or something like that probably in real world applications this is something that might help on Atari it didn't because it sorry is sort of very deterministic and there's not that much variation but I think that's something that could work yeah it seems a bit different so there is a form of target networks and continuous control people instead of making a fixed copy of the way it's you sort of take a running average of your ways as a target network so that's a lot more like its momentum like but I think it is quite different like in fact you might want to use both momentum and target networks in practice so here there's a young the whites teeth are you never reset them so you copy them into your target Network but then you continue training the way it's from that value so that you don't reset them yeah so there is a more complete algorithm maybe it may be one more question yeah so that's a good question basically what do you look at when you train these things sonar all yet monitoring say the average episode rewards like the total reward in an episode averaged over a number of trials is like that's the main thing that you look at we were looking at say I can supervise learning you look at your training error so you're the loss you're minimizing is a squared error so you can look at that and you if you do that then it does make sense to have a holdout set of states so we would run a random policy for some number of states yeah for some number of stubs say take a thousand states and then compute the ones on these something to monitor if you look at your loss online as you train it doesn't necessarily make sense so we saw that it actually certainly keeps going up and we thought oh that this isn't good but what's happening is as your agent is learning to play the game better it discovers States it doesn't know about so it looks like the error is going up but on a fixed set of states you see it sort of initially spikes and then starts coming down yeah okay so probably move on so I think yeah the target network idea I think it's good to get a bit of intuition for why you need it so here's an example of two states from the game of space invaders you have two consecutive States importantly so this is your state s state s Prime and you got a reward between status and Status Prime so what you will do is because you've got a reward then you will probably increase your estimate we'll try to increase your estimates for the key value of action and status because it gave you a reward but because you're using a neural network to represent your key function you know you if you plug in these two images into your continent they look very similar it's sort of hard to say well what's the difference now maybe an alien disappeared the lasers moved a little bit so it's hard for a neural network to give very different Q value estimate for states that look similar because I guess the thinking is the inductive bias of neural networks is smoothness so if you give similar inputs that should produce similar outputs but here what we're asking it to do is to basically give a higher I would put something much higher for this state than this state because you got a reward for taking this action so what ends up happening without a target Network if you increase the Q value for this state in action you will also probably raise your estimates for the Q value of all the actions in the next state because they just look so similar so the next time you come back to this example you will have a higher target because you inadvertently increased your prediction for the Q value in the next state so this can put you in the sense shell loop or you're chasing your own tail you're coming up with higher and higher estimates of the Q values you think you should you can get and training might blow up now if we use a target network and if we're computing the target with a fixed set of weights you know you see this transition and you will increase your estimate over the Q value and stay down but next time you come back if your target Network weights have been fixed the target doesn't change so doesn't that that's kind of how it yeah helps prevent things from blowing up so one interesting anecdote is when we were still trying to get this stuff to work in order to run more experiments quickly we were using smaller confidence because for Atari you don't really need the confidence to do reasonably well but what that ended up doing was actually making the problem harder for us so when you use a smaller confidence to represent your Q function you essentially add more aliasing into your state representation because you have fewer parameters it's harder for the network to make drastically different predictions for similar inputs so because we were using a small network it was actually the target networks were very important for stability once we were happy with the approach and we started you know we wanted to get the best results we could we started using bigger continents and when we went back and redid experiments to see how important is the condom Porton is the target network it was what was important and was smaller in that so this is yeah just an example of being careful so yet doing experiments with small networks is good but it might have sort of unforeseen consequences on how difficult the problem is okay so this is a high-level view of the dk1 training algorithm I guess yeah it's probably it's probably worth going through it and some level of detail so we started off with basically initializing a replay buffer to some capacity M so n will be the number of frames it fits and our Q function starts with parameters theta and we initialize a targets Q function I guess Q hat with parameters theta - which initially are the same as theta so then we have a loop over episodes where same Atari and episode is one play of the game until you run out of lives and then in the episode you iterate over frames so you first select an action use your policy so because we're learning a cue function we have to define the mapping to policy so we were using just epsilon greeting action selections so with probability 1 minus epsilon you take the action with the highest key value with probability up so on you sample uniformly random and we ended up using a schedule for epsilon which I'll mention on the next slide right so then we execute our action a in the simulator and in return we get a rewards and the next state so here yet there is a bit yeah it's important to think about sort of what the state of the agent is and what the state of the world is so when you're playing a video game you might need to remember things from the past but your observations are just say that the current screen shot so in DQ n the state of the agent was a concatenation of the four previous frames of the game which gives just enough information to sort of see which way different objects are moving and this is an imperfect state so it doesn't capture the entire history of the agent which you might need but it's sufficient for learning so we yeah right so we do pre-processing of these concatenated frames to give us these times and we store these transitions in the experience replay buffer so transition here is a pre process state t the action the reward and the pre process state T plus one and you put that into your replay buffer and you sample another mini batch of transitions so the one you just acted on almost certainly will not be in the mini battery sample and none for the entire mini batch you compute targets using the familiar equation where it's just a reward if it's the terminal state and it's this sort of bellman backup otherwise and then again we just do a gradient descent stuff on the squared launched towards the computed target and every C steps we copied the weights of our key function to the target Network so this is essentially the whole loop where the agent is interacting with the environment it's putting data into the replay buffer and then the learning algorithm is basically sampling from the replay buffer and updating the neural network and the agent acts using the epsilon greedy policy for the neural network so there are a couple of details that were left off so we don't actually use the squared bellman laws we found using something called the hueber loss to work better so the squared loss here is shown in blue Hueber loss is something that's the same as the squared loss between minus 1 and 1 and it's sort of the absolute value loss otherwise so using the heber loss penalizes the network less for making really bad mistakes but it puts more emphasis onto getting sort of the smaller errors this focuses on smaller areas more so we also used our mess prompt instead of standard as Gd again this is probably beyond the scope of this lecture but different optimization algorithms matter a lot and reinforcement learning so deep learning was supervised learning I think many people still believe that if you tune your learning rate well the newsing stochastic gradient descent will still get you a good solution and rmsprop oh sorry in reinforcement learning this is not the case because how you update your neural network determines which actions you will take which determines which data you will see so it's not just a matter of fitting a fixed data set the optimization algorithm you use affects the dynamics of your agent and what how it will explore what data it will see and we find that yeah things like our mess prop and other people offend methods like atom get you to much better solutions in many cases than is Gd so optimization really matters and deep reinforcement learning so we also found that it helps turn the ol the exploration rain Epsilon so we started at 1:00 which means you take uniformly random actions with probability one where you start so you're just trying anything really and over the first million frames you linearly anneal it to something like point one or point O five so there's this initial learning phase where exploration is very high and then it remains fixed for the duration of learning and yeah maybe the there almost certainly better annealing schedules but yeah it helps to have some sort of phase with a lot of exploration at the beginning right so yeah all right so this is yeah I can probably beyond scope of this lecture but yeah the theory of optimization basically the proofs for convergence of like stochastic gradient methods rely on like iid updates this is I think still a big open problem in their own networks how how do you solve how do you address the continual learning problem so if you're building a system that's you know something in production the data go change over time almost certainly as your users change and yeah this is a problem for neural networks I think there's been some recent progress but yeah I think it's still not fully understood okay so specifics of yeah they had this application of DQ and so we applied it to 49 Atari 2600 games the idea was yet to use a convolutional neural network to map pixels to Q values which gives you your actions the reward signal between every two consecutive frames was the change in the score for most of these experiments we clipped this between minus 1 and minus 1 so in most cases what they're actually optimizing is the number of rewards as opposed to the magnitude of the rewards and again we used the same algorithm for all of them key learning with the DQ n tricks we used the same architecture which I think the big one had something like three million parameters and the hyper parameters were fixed for all games so there was no specific tuning for any of the games [Music] yes and not in this case that this is something people have been looking at so yeah transfer learning and the parole yeah it's definitely important future direction doing it on the atari ends up being harder than you might think so there are similarities between games like there's space invaders and i think something called alien no not alien assault which you know in both cases you're controlling a ship and you're shooting lasers it's aliens but the similarities are actually very high-level like it's very hard for the neural networks to generalize between these games so for us yeah it seems so you can transfer the knowledge easily but the neural networks here like they're learning to see essentially for the first time by playing these games and it's hard for it to generalize between the ship and space invaders in the ship and assault right so it's some it depends on the game of course so some games where the exploration problem is more difficult will be more sensitive to the seed so sometimes the agent will discover behavior is sometimes not these Atari results are pretty stable so if you run them there will be differences on the individual games but like the overall performance over 50 games is pretty stable to different seeds tiendas seems to be more robust to parameters than some of the other or save value based methods like he learning seem to be somewhat more stable to hyper parameters than some of the policy gradient methods people have been looking at so yeah people are deep mind still you know when they change the algorithm in some way they still run it off with the same hyper parameters and all the Atari game so there's been surprisingly little tuning you can almost certainly do better by doing more tuning but it's been I think quite robust yeah so people have looked at that that that seems to be a good technique like if you fill your buffer if you have a separate buffer that says has has only human data and then you still have your own replay buffer and you sample from both this accelerates learning quite a bit and really yeah it helps it learn on some games where it doesn't it can't learn just through it in a random exploration there are yeah weird artifacts that we didn't put that much thought into the specific number of frames on some of these Atari games you know movement might only happen on every other frame so it's not yeah and yeah there's all sorts of weird stuff so for frame seems to seem to work well yeah yes so I guess what we're giving to the network here is not a proper Markov state so and it still seems to work reasonably well so for the theory it's important but technically what we're doing here yeah technically this application is not an MDP so to get an MDP an Atari you would have to condition on the entire history of frames to be proper to have a proper Markov state and yeah using recurrent in their own map works what do you so yeah that gives it I guess so that gets it closer to having a Markov state than having just one frame I guess the link between frames sort of happens through you know getting a reward for taking an action and seeing which state you ended up in next that's sort of yeah that's important yeah and so for most games you're pretty close to having a homework I've stayed because say for Space Invaders you know they're I don't know if we have screen shot here well this is a good example right like it it visually shows you how many lives you have left so it's true that you don't really need that much memory for these games you do need some because again things you need enough to see the motion and sometimes there are like weird flickering artifacts so I think Atari had a limit on the number of sprites it could draw on one screen so you would often draw some sprites on even frames and others and odd frames and like this is annoying to deal with but yes there was pre-processing most in down sampling and dry scaling so yeah color can be useful we ended up throwing it away here but it's just down sampling and greyscale yeah yeah so typically yeah in many domains people end up using and recurrent neural networks to represent the cue function or our policy but yeah I think I'll try to move along now the interest of finishing yeah so here the Q function was represented using a consonant here yeah we're showing two convolutional there is the final results whose three convolutional hilarious and one fully connected layer and the outputs were sort of one I would put per action with the key value of that action so our D stability techniques important so here we tried different permutations of using replay or no replay and target or no target on five Atari games and I think yeah here we retuned we retune the learning rates for each combination so again we found that the replay was the most crucial of the two components because this is now using bigger confidence the best results were always with both sorry yeah these are scores on these five specific games averaged over I think a hundreds plays or something like that so yet the one case where target networks were very important was Seaquest which happened to be the game we were trying to get to work with the small confidence when we started using target meds but yeah overall doing replay gets you most of the benefit but using both really is gets you the best results in the end so here's a visualization of the performance over 49 Atari games so the x-axis is showing essentially the percentage of the human score that DQM gets on each game and they're sorted by performance and you can see that I think something like 29 games out of 49 the performance was over 75% human which for us was a measure of like are you in the ballpark of a human games tester on these games so the goal here wasn't really so much - for any specific game to like build the best possible agent that will be like the best human the goal was really can we build one method that without really any tuning you can throw a bunch of problems and it will work as well as you know a reasonably competent human so the architecture was the same but we trained it on each game separately so for each game you run the algorithm and it you know trains a confident that plays just that game I don't know five answers so yeah that's another artifact of neural nets so typically you if you're doing say regression you would like normalizer targets to maybe have zero mean and standard deviation 1 it's a bit trickier with rlb so yeah it's basically on Atari the scale of rewards is like kind of crazy like on some games you might get a reward of a thousand for blowing up a tank and I'm you know in pong you get a reward of one two for scoring a point so if you want to use the same hyper parameters across many games it's pretty hard if yeah the scale is that different so in some sense what most games rewards were only positive so it was really optimizing like discounted frequency everywhere it's as opposed to a discounted score but yeah so yeah we don't have too much time don't know if I should show it's all 45 ok yeah so this is a video of the original dqn agent from 2013 playing a bunch of the Atari games and if you yeah if you look at videos from papers being published now an Atari agents are way better but this was still yet quite exciting at the time so we'll look at some of these examples a bit more closely on the next slides and the kinds of things the agent is learning to do like giving sacrificing short-term rewards for a more long-term reward and things like that so again these are mostly games in which dqn was good which happened to be for the large part the more reactive games we're driving it helps to have a fast reaction time and their own that's well it's pretty easy for them to have a fast reaction time over time as deferral methods improved now agents are quite good on games with that involve more of the longer-term planning like say Pac Man where you have to navigate mazes and avoid ghosts but yet the original dqn with a simple confident was mostly excelled at reactive games so maybe I'll yeah skip the last few I don't know so yeah so this is pretty basic but I think it's still a good example of really understanding what how the network plays these games so these are four screenshots from an episode of pong the trajectory of the ball is superimposed for for clarity and what we're showing at the bottom are the predictive key values for the different actions for these frames so the agent is controlling the green paddle on the right and you can see as the ball is approaching here it's predicting a positive Q value for all actions so what it's saying is it doesn't matter what - what I do in this state because I think I can still get the point almost certainly so if you fast forward a few frames now it looks like you know the you're going to lose a point unless you and the predictions capture that so now it says if you go up you will still get a point but if you do nothing or you go down you will lose a point and as you fast forward again the agent starts going up and it's still I guess now that it's closer to the ball it's maybe more confident that it can get a point and more confident than going down will lose a point and here is it basically it returned the ball and it's about to score on its opponents and here again you can see that it doesn't matter what action to take the Q values are all essentially 1 so the agent knows it's about the scoring point so that's how basically a cue function transfers into a policy with greedy action selection so yeah I mean it's react I guess it does something at every action there is yeah yeah no so this is simply because yeah this state basically again it doesn't matter what to do the agent can do nothing and it can still go up in the future and have it has enough time to get a point so it's not some of these policies look a little bit like jerky and random because it's not trying to act in the most efficient way it's just trying to get the point so yeah it it doesn't care that well it has slight preference for a reward sooner but it's more important that you get the rewards at all then you get it sooner yeah yes so that's a huge area of research that continuous control and robotics yeah we'll hear a lot more about it I think later today and tomorrow like policy gradient methods okay so I'll move on yeah maybe I'll show this one quickly so this is again an agent playing break head where we're showing again the screen the screen shots and this is a prediction of the state value function which is sort of the maximum Q value at each state so you can basically see that the agent sort of knows when it's about to get a reward and what when it does get a reward the estimate typically goes down so this is yes space invaders so this was trained without reward clipping so you can see whenever it shot an alien it's estimate of the key value would go down so now we see this purple mothership show up which is worth way more points than everything else and we see that basically it's action value estimate started climbing until the point where it shot the mothership so that's the point where it knows okay I got this big reward so now I can expect a lot less because another one isn't about to show up so yeah as I mentioned dqn has even though it's reactive and has some ability to sacrifice immediate rewards for a longer term rewards so again we see the value function on the top right so the agent is shooting these fish but you can see that it's running out of oxygen so at some point the agent will have to go up to the surface to get more oxygen and there is no reward for getting oxygen the only signal for learning to do that is that if you don't get more oxygen you will lose the life and you won't be able to get more reward into the future but actually in the short term it's much better to just keep shooting fish because that's guaranteed short term reward and as you see ya trained training with the discount of 0.99 is enough for key learning to discover this behavior okay so let's see if there is yeah there is some time I said the dqn source code is available online that's in blue and torch if you want to play around with it it's there but yet there are more data efficient methods now that will save you a lot of time but it might still be interesting to learn about so I don't have that much time left yeah so this is probably worth discussing so there are our people have been training neural networks with key learning for many years and the question is okay well like did we need to come up with anything to actually do this so neural fitted Q iteration is one method for training neural nets with key learning and the way it achieved stable training of neural networks is somewhat similar to TQM but it's a badge method so this is a method from Martin Reed Miller from 2005 and the way the algorithm works is you basically you collect some data and train you to basically you iterate between playing another episode and then fitting a new neural network so after every episode you take all your previous experience you compute targets using your current estimate of the cue function and you turn this into a fixed data set so all your previous data with your current predictions and then you apply a batch neural network training method called our prompt which does not many badge but batch optimization of a new neural network to this fixed data set so here each iteration involves training and new neural network entirely from scratch but each iteration is essentially supervised learning you have fixed targets and yeah so this is a very stable method and it's quite data efficient but the reason it's not in completely practical for confidence is that you don't want to train many confluence like training one continent is expensive enough so if you need to train a bunch of them that's not really going to work but you can actually see the queuing as a practical online variant of neural fitted queue duration so we don't reset the weights of the neural nets ever we're sort of we trained one confident but we sort of simulate this effect of having a fixed data set with a target network and a replay buffer so yeah it's quite similar in spirit to neural fitted queue duration okay this is all skip so there have been many improvements to DQ n since it was published so one important improvement is called double DQ n I had a run hasselt and few of my other colleagues so here the idea is that DQ n and Q learning in general actually has an overestimation bias when you're losing a function approximator to represent Q so if your Q function if you assume that it's say the true Q function for your current policy plus some noise then if you look at the value of the if you look at the Q value with the highest value you will probably overestimate it because it will be the true value plus some positive noise most likely so because of this using this estimate for the Q value has this upward bias and you can see on the bottom here this shows and basically some measure of the estimate of the value on for games and the red curve is showing declines estimate and you can see that sometimes it might even collapse and sometimes and goes a little bit crazy and the GG queuing methods I shown in blue and you can see that it's often much more stable and has lower estimates of key values and dqn so so how does double D kyon do it so what it does is it exploits the fact that you now have two networks your online network and your target Network and the idea now is to use your online network to select the best action so you look at your online network and you ask what's the best action in the next stage and you take the Arg max but then you actually use your target Network to get the value estimate so this is less biased because you're not using the value of the Q well yet the Q value with the highest value is the estimate you separate the Arg max from selecting the value so this is there's theories showing that this is less biased in practice in theory and in practice it helps a lot so we're showing here basically if you look at the average median our mean score normalized by the human performance on Atari then you see just doing this doubling trick gets the queuing from xli 47% median score overall game stereo d8 so it's a huge improvement in practice yeah so I think they both help and practice but this one I think was slightly better I don't know if that might be in the paper I'm not sure so right now so another huge improvement over the UN was dqn with prioritized experienced replay so this is from Tom Shaw and others a deep mind so here the idea is okay we have a replay buffer but we're sampling from it uniformly and this is actually probably very wasteful in practice because if you already if you can already predict the target well why should you waste time updating it in practice so the idea now is to prioritize the data you sample from the buffer using the absolute bellman error so they abused the sample states on which you're currently more wrong in order to essentially prioritize your learning on parts of the state space were yeah you're still not very good and again this was a massive improvement so there are two ways of doing it you can sample like directly proportionally to the absolute bellman there or you can sample based on the rank so you yeah it's a I guess a slightly different sampling scheme and again in practice they both work really well and yeah lead to big improvement so just prioritization alone improves dqn again from something like 48 percent to 106 percent in the median so it's an even bigger improvement so another useful trick is dealing TQM so here so we know what the Q value is State value we know about Q and V so there is another quantity called the advantage which is simply the difference between the Q value and the state value so this essentially tells you how much worse an action is done the best action in this state and it turns out that we can drastically improve Q learning with neural networks by just changing the architecture of the neural networks on gqn again we simply had this fully connected output layer which outputs one value per action but what you can do instead is make two separate channels where one outputs a single number which we will call the value even though it doesn't necessarily have to be and another channel which outputs one number per action and we'll call these the advantages and if you essentially sum these up to get your output estimate of the Q value so you just changed your neural network and then run DQ n as is that will work much better in practice the only trick is you have to subtract the average value from the advantages inside your neural network and back propagate through that process but again this is something that I guess yeah changes the architecture in a way which makes it easier to separate the values of different actions probably don't have time to show the video is there one more okay this one is quick to explain so this is again very recent work there were two similar papers so one from deep lines and one from opening I and Berkeley so people are now finding that you can go way beyond Epsilon greedy exploration and one good way of doing exploration with neural networks is just to add noise to the parameter of the neural networks so you can go yeah you could do this and a number of ways but yeah this simple technique gives you much better exploration and practice on most Atari games so yeah if you're interested in exploration these two papers are definitely worth looking in yes so that's everything for this lecture [Applause] yeah so you could think about different ways of doing that I think how he had like sorry the question is yeah I guess can you be more clever about filling the prioritizing the experience replay buffer I mean yeah I would say yes but the most basic way seems to work quite well because yeah there isn't that much to go by so you could start learning generative models of your data or the inputs or something like that but yeah yeah so I think when you sample a yeah when you sample from the buffer and you learn on this transition you update you update the stored error I don't remember if you also have to sweep all the other ones but yeah that that's in the paper yeah I think that's probably all the time we have so can take the rest offline [Applause]