Tiny Reinforcement Learning for Robotics: Edge AI & Control Theory

[00:00:00][Music] [Music] [Music] he [Music] [Music] [Music] [Music] [Music] hey [Music] [Music] hello everyone our next speaker is a devil and electronics educator and he's been living out on the edge with AI but he is here today to tell us why some of the lessons of classical control theory don't need to be thrown over just yet please join me and welcoming to the hackaday supercon stage Sean himl good afternoon Super Con thank you Elliot I have been working with machine learning for a couple of years now doing things specifically on the edge because it's fun I enjoy this I come from the embedded background embedded world and I really enjoy the machine learning side and combining them is a ton of fun so the last year I've been working specifically with reinforcement learning and that is a a deep dive in its own realm and this is a ton of fun because now we're getting into the idea of can robots not just do a thing you tell them to but can they learn to do the thing you tell them to so that's where I've been really excited about and you've probably seen some of these in the news on YouTube and you probably recognize a few of these this is the Disney's new imagineering robot if you've seen this it has a beautiful bipedal gate it walks around it's really cool to see I honestly don't know what they use to make this cuz he controls everything this is uh copters the idea is what if we can have robots Learn to Fly themselves same idea with self-driving cars not just let me program you to avoid pedestrians but what if it actually learns to avoid pedestrians and it's rewarded or penalized based on that we also know Boston Dynamics we're familiar with spot Atlas all these good ones that they're working with now there were a lot of conversations because Boston Dynamics does not publish a lot of papers if any at all so there was a lot of conversations over the past few years whether Boston Dynamics actually does reinforcement learning or not when they had an interview I learned a couple of days ago there was an interview with the CEO on the Lex Freedman podcast they mentioned they do do a mix of classic control theory algorithms along with reinforcement learning so they are actually using some despite what random redditors may tell you then there's also this last one that came out eth Zurich put this really cool robot out last year they published a paper on it using reinforcement learning and this is a a four robot and it can do things like climb up stairs it can it learned itself how to raise itself up and balance on two wheels this is a cool video go check this one out if you haven't seen it um I forget the name of the robot but just search for eth Zurich robots these are really fun the dream behind all of this and I want to say Boston dynamics's dream as well is I want to go buy like a pet dog excuse me a robotic pet dog I already have a pet dog I want a robotic pet dog that I can put on the table I can push a button and overnight it just teaches itself tricks or teaches itself to walk and that's where we're trying to go with some of these but as it turns out reinforcement learning is really really difficult we're not at that point yet Boston Dynamics is what multi-millions of dollars in funding dozens of of phds and Engineers trying to even remotely achieve this dream of having robots teach themselves and get something close to um uh biology in the way the motion happens since I am not Boston Dynamics I wanted to start with something very simple here's me being very proud of myself doing an inverted pendulum who has done an inverted pendulum project a few people uh did you do it in school keep your hands up all right have who did it on their own just for fun like outside of school oh like at least one okay well done well done this was me doing it outside of school because I never took a control theory class but apparently this is the thing you do when you do a control theory class you one of the projects usually your end project you learn all about control theory control loops and all of that plants what have you and you do an inverted pendulum it might not look exactly like this but the idea is to have this thing balance and do it in such a way that you don't have control over the primary axis here so in my case this is a kit I know I'm losing like maker cred because I didn't build my own balancing bot here but the reality is is I just wanted to get working on the algorithms so it moves back and forth and the idea is just by moving the stepper motor on this axis it controls the pendulum that's on this axis and this is a quadrature encoder that measures the angle and so the hope is that you move it back and forth and the real goal is to have it swing up and then stay balanced and that's the goal and the balancing part was actually fairly straightforward using a PID controller um you can get it apparently to do the swing up if you do a nested P ID controller or some of these more advanced ones um linear quadratic Regulators model predictive controls not that I know anything about those because I just played with the PID side but you can get it to do the swing up and hold there so this is a solved problem this is nothing new like students are doing this like like J like freshman sophomore junior year in their college courses I wanted to make my life hard so I went and used AI to do it can we have a robot that learns to do the swing up on its own so a little bit of History not really history but background behind reinforcement learning here's the essential diagram that if you look up anything with reinforcement learning you will see there is an environment that is the world that could be a virtual world that could be the real world that might be the rules of the game chess or maybe go we have some sort of interpreter that gets observations from that and in our case our observations are going to be four values that's this doesn't work very well with four four values and we have the encoder angle so that's my quadrature encoder that's the angle of the pendulum we have the angular velocity of that encoder how fast is this moving we have the stepper motor I want to know the position of the stepper as it relates to where it started the relative position and that angular velocity of that stepper motor those are the four values that I'm going to look at for my observation those get fed into my AI agent this is the decision-making body this is just the code that says I take observations in and I give you some sort of action as an output in this case I started with continuous action space I said figure out how much you need to move that stepper motor by in order to achieve some goal in this case the goal is Swing the pendulum up and balance it to achieve that we need to create a reward function and this is The Interpreter is in charge of taking those uh either observations or some other part in the environment and saying based on what I see you get some sort of reward so we incentivize the agent to learn to do something over over time and the agent not not just wants to maximize this is the tricky part of reinforcement learning the agent doesn't want to maximize the reward from taking one action it wants to figure out which action to take in order to achieve maximal rewards in the future probabilistic rewards in the future is where AI gets very tricky actually our brains are really good at doing this already um you know with some hiccups here and there but we're generally good at maximizing our rewards and can predict into the future with some probably probabilities involved in there and so the idea is let's create math to do that for us and now we have robots that can learn or AI agents that can learn so here's the reward function that I use this came right out of farama gymnasium their whole framework for doing reinforcement learning and this is the reward function so we just Square the uh angle of the encoder we Square the angular velocity we Square the angle of the stepper motor and we square that velocity and then put some constants in there the constants tell me that I care most about trying to get that pendulum upright hence that one and everything else is kind of secondary to that goal or even tertiary on that side uh the hope is that I want to get the pendulum up as and not moving as much as possible hence why I care about angular velocity and so that negative sign says that every step you're losing points so zero is the best you can possibly do for any particular step during this process and any other than that it's going to be negative so it's going to incentivize the agent to quickly as possible try to solve this to make this zero and so I've normalized it so the angle of the pendulum is zero when it's up and it's either one or negative one but because of that square it doesn't really matter when it's down and then same with the stepper motor it's zero when it's zero where it starts and then one or negative one 180° past and there's this hard limit that says if it rotates a full 180 de in either direction stop the episode and you get penalized pretty hard by 500 points and the reason for that is because there's this cable that comes off here and as you can see that gets nicely turned up um if you come find me later I'll show you this I don't want to get into why there's a whole bunch of stuff here because I had to resolder it because I started breaking internal wires and then I had to put this nice little strain relief on here so this was this was a problem that I had from doing this over and over again even with this the skips some and it still Tangles itself up which is not a lot of fun but it was good enough for what I needed I used the proximal policy optimization algorithm which is considered state-of-the-art the paper for that was written in 2017 it's still used today for a lot of things it works with continuous action spaces discrete action spaces continuous observation spaces it's pretty robust um but as you'll see it's it's not great we're still not there yet um with state-of-the-art you can find this I use the the stable baselines 3 framework which has these algorithms and then I had to go through this whole process of optimizing hyper parameters so if you're familiar with machine learning hyper parameters are anything outside of your model that you can tune that doesn't get autotuned things like learning rates how big your your your actual model architecture and there's a bunch of stuff in po that you can tweak and tune reinforcement learning is very sensitive to hyper parameters I mean like you you move something by like a couple of percentage points in one way or another and your algorithm goes from not working to working or vice versa so it's very sensitive so tuning hyper parameters became a whole thing I use Bean optimization using the axe framework I won't get into the specifics of that come find me later if you want to talk about Axe and why it's actually kind of slick or weights and biases but speaking of weights and biases it didn't work which you know it doesn't work on the first time not not with anything that's fun right so what I'm hoping to see here is each one of these lines is a trial where it has a number of steps and this could be any number of episodes and so it just even though an episode might end here it just keeps going and keeps counting the step up as the steps up as that algorithm is updating and learning that agent is updating my hope is that each one of these lines some of these lines as it would get better and better with the hyper parameters because each line is its own set of hyper parameters I want to see them going up and to the right and approaching zero because it's the reward that it gets um I don't think this is not this is not cumulative reward because it'd be going down it's reward per step I want to see this going up towards zero as it learns to kick that pendulum up and balance it okay so why did it fail we got to talk about some reasons why this failed first of all this roundt trip Loop because I don't know if I I don't think I mentioned it but the training portion on the agent was happening on my GPU was happening locally on my laptop this interpreter is an AR Arduino and the action taker is also an Arduino that's controlling that stepper motor so it's all going through serial over USB which even at a a megahertz or uh yeah million bits per second it's still it would take over 30 milliseconds to go to the agent do any sort of updates with that staple baselines and then come back and perform that action and based on my very very particular testing uh or or like you know generic testing empirical evidence it you need at least you need low than 30 milliseconds in order to get that to balance so the best hope I had was like maybe I could do that swing up but balancing right now isn't possible without like tearing apart stable baselines with the two months I had for this project it was not going to happen also continuous action spaces are notoriously difficult in reinforcement learning so we are going to Maxim we're going to reduce that scope and change how we maximize that reward first all I care about is doing that swing up right now I'm going to say okay I'm going to put off the whole balance balancing part for later and come back to that let's just prove that we can do reinforcement learning and get it running on a microcontroller just to demonstrate that so the first thing we do is we add this this area about Plus or finus plus or minus 5 degrees from the Top If it reaches that and it's not going too fast we'll call that good that's a win let's restart the episode give it some reward to incentivize that if it goes too fast in that area then we're going to say you've crashed we're going to penalize the the agent and we're going to start over okay so we' reduced the scope for this the other thing is now we can start talking about let's just change this to discret action space instead of that wide range of minus 60 to plus 60 degrees that you can move the stepper by you only have three possible actions that's minus 10° do nothing or plus 10° this worked a lot better so after a few trials with good hyperparameters we can start to see it's approaching zero okay something's actually working here what is so now that we have something trained we have the RL agent we can go and look at what's the actual decision making body and that is the actor this is an actor critic network if you've heard of that PO uses an actor critic it's two different machine learning models or two different neural networks dense neural networks three layers each 256 nodes in each layer and the actor is in charge of taking those four observations and then giving me an action to perform so hence the 1x3 down here this is the actual Network by the way I put this into nron and just took a screenshot of it and so the three here is my values just generic logic values that I get for minus 10 0 and plus 10 so you figure out whatever one of those is the highest value and that's the action you should perform the critic on the other hand is a regression model that gives you one value out and it's used during training to try to predict how much reward do I think I'm going to get in the future and it's used as part of that training process but the good news is once we're done with training we can just get rid of that critic side and we just take that actor and once we have that actor we can go ahead and deploy it to the microcontroller I'm just using Arduino because I'm lazy and it makes my life easy um there's a number of tools you can use for compression and optimization I know Edge impulse very well so I'm going to take my model here I'm going to use Edge impulse to help me compress and optimize that for an Arduino and deploy it to an Arduino I used an esp32 S3 um from Seed Studio because those are very fast very powerful they have neural network um optic ized Hardware in them which is super cool and I love to see that so we have the loop now this happens inside the Arduino no more CPU or GPU down here but notice I took away that reward function cuz we don't need it anymore we're not training we've deployed it it's trained okay this is the fun part live demos okay so first thing I'm going to do is I'm going to give that stepper motor 18 volts and avoid the feedback from the microphone all right come on stepper motor you got this come on thing I bought off of eBay or not eBay but Amazon there we go 18 volts going to that stepper motor and let's power on that micro all right remember this is AI doing it not a control not a control Loop yes it did it swing up [Music] hooray I know it's I know it's fairly simple and fairly straightforward but it gives us a way forward to prove that yes we can run reinforcement learning algorithms on microcontrollers yes it's a solved problem but now I'm doing it the hard way and it taught itself how to do that which I find really cool this is in case it failed which it didn't here's some of the output um if you look at this the observations that it's collecting so this first one is the angle of the uh quadrature encoder this is the angular velocity remember it's been normalized so minus minus one means it's pointing down zero means it's pointing up so this is an example of where it worked we have the angular velocity we've got the angle of the stepper motor which it does a pretty good job of staying pretty close to where it started and then the velocity of that stepper motor we also have the time between the steps and the interesting thing here is the inference time is about 8 milliseconds which is pretty good for that three- layer dense neural network and you'll notice that it actually waits for the stepper to complete so if it was minus 10 which is one of the possible actions then it waits for that stepper to complete but if it's not doing anything then we do 8 milliseconds honestly I found that waiting for the steper to complete was a lot easier to deal with because if I tried to create a new inference action while the stepper was doing its thing it would just start skipping and freaking out I didn't tune the stepper motor to do that it has an acceleration and deceleration time and it would skip like crazy when it did that so where do I see this going in the future um and actually what we're doing right now is there's a number of Frameworks like Nvidia what was Isaac Jim is now Omniverse and we also have things like Unity where we can create virtual simulations of this train the algorithms in there and then deploy them this gives you the ADV Vantage of creating thousands of copies and I can train in parallel which is nice it's faster and the big issue with Hardware is it's very hard to reset this is great because a stepper motor I can just say go back to home but think about like a Rover or something like what do you do you have to like pick it up and move it over somewhere when it's done with the episode or it has you have to like train it to go back home which is like defeating the whole purpose of having it trained so that's a problem in in the virtual environments solve that for you so where we're going where we're are where we're at now and I've got um my friend Dimitri who let's see if this works so he was actually working with the Biddle which is the little $250 you know spotlike dog and he was doing this in a virtual environment and training them to work to to actually walk or scooch along when I gave this presentation to a group including him uh on Wednesday he was like oh I was working on this he sent me this video I'm like oh this is so cool he's like but I never got it working on the hardware and you just solved the hardware problem so he came from the software side of of simulating it I'm coming from the hardware side of like can we finish the training in the hardware and hopefully together we can make a tiny Boston Dynamics Team yeah right but that's you know that's where we're getting I'm I'm hoping people can use this and create their own robots using reinforcement learning so I'm way out of time um I don't know if we have time for questions maybe I don't I'm looking at Angela who's the next talk next speaker five okay couple of questions um my my repo's up here if you want to see my code it's a mess I apologize I usually make better repos but I was finishing this up like a weekend ago um but follow me on any of the the social media platforms reach out to me I'll be walking around after this but we have time for a few questions you're talking about doing for are are you saying doing the trading Loop in the Arduino and then evaluate or update after yes so that's actually that would that that thought occurred to me as well as like a way to do this um and and so you could get that better uh latency time was to have it do latency here basically do your Monte Carlo simulations collect your data from that update the model and then reflash the ESP so that it can do more Sim like what we call simulations but actual uh actual episodes but yeah that's a that's a idea and hopefully that would fix the latency problem yeah yes so it's collecting data and rewards at each time step um but reinforcement learning what it tries to do is it says based I I receive a real reward but based on my knowledge of the universe and what I expect what prob like what's the probabilistic or expected value based on an action that I'm going to take not that I'm going to get a good reward next but I'm going to get a good reward until the end of the episode or until infinity and so it's trying to maximize the future expected value of the rewards does that make sense it does it by trial and error so it has to do a whole bunch of stuff where it just tries random things and it says well based on what I did and rewards I received I think that if I do these in the future I can maximize my rewards and then it updates the model based on that and then it just tries that and so you it's an iterative process as you get better and better at say at approaching that local Optimum of saying now I'm maximizing my rewards in the future based on previous things that I've done and learned does that help this is why it gets like really weird in in reinforcement learning it's all probabilities it's it no because the processor is actually fast enough for the training portion of it I was using a GPU the the problem was the the training framework I'm using like stable baselines and all of these I don't have much control over what they're doing so i' you'd have to strip all of that away and rewrite it you also run into issues where you're moving memory back and forth to the GPU so to be able to do your updates and get under 30 milliseconds is a problem hence that gentleman's idea of let's do our our simulations or our trials on the device itself and then train like take the model take what I learned from that and then train elsewhere and then update the model because we want to close that latency Loop that's the big issue um but it's it's not a processor speed problem if you're asking what would it take to train on here um You probably don't want to because like even getting like a Raspberry Pi which is relatively way more powerful than this to like train on like mobile net like it takes days it's it's usually pretty silly to try to train on embedded systems there's there's work being done so you have like neuromorphic processors which can actually update its own weights um so there's some work being done to get something like that happening but we're still a ways away from that right now all right yeah oh sure um that was a that was a fun Rabbit Hole so there's there's three big um algorithms that are used for tuning hyper parameters in an automated fashion well for if you count manual like I really know this algorithm I know what I'm doing if I tweak this this will happen so there's manual where you really know the algorithm very well and there's three main automated ways to do that the first is called grid search where you just do you just like take a wide range of your hyperparameters and you just like every possible combination you exhaustive search it and you're like whatever gave me the vest best evaluation go with that um very timec consuming for something like this where it could take like 3 4 hours just to run one trial there is random search where you take a random smattering of those combinations and then you do some math to say oh this is kind of what it looks like I think if I kind of go between these right you can like interpolate the values to figure out where you might go and then there's beian optimization where it actually performs um it uses beian inference to say or uh yeah beian inference to figure out based on what I've seen I have an idea it basically creates a random probability of what what the output should be or could be and then as you take sample points you start to get a better idea probabilistically speaking of what that function looks like and so you can say oh from here go to this point and try another trial go to this point and you basically start getting towards the optimum set of hyper parameters um what most ml Engineers will do they'll start with random search because it's not exhaustive and to narrow the scope and then they'll do beijan within like a smaller range sure all right thank you everybody um come chat with me about RL [Applause] stuff