How AI Decodes Brain Signals for Movement & Recovery

[00:00:00]so with that I want to begin by introducing a wonderful lineup of speakers that we have on these topics our first speaker is Dr Karen moxin Dr moxin is a professor of biomedical engineering and neurological surgery she's Affiliated as a faculty at the center for Neuroscience in the center for neuro engineering and Medicine doctor Dr moxin conducts groundbreaking research and neuro engineering developing computational approaches to study the encoding of sensory and motor information an important focus of her work is the impact of neural injury on the representation of information in the brain early in her career Dr moxin contributed to the first demonstration of a closed-loop real-time brain machine interface system in the rap model that was quickly translated to non-human primates and more recently to humans with neurological disorders today Dr moxin will talk more about the relationship between artificial intelligence machine learning and brain health Dr moxin [Applause] set up for what I hope to talk about today so in my lab one of the major projects it's fine that we are interested in is to restore function after spinal cord injury and um today I'm going to talk to you about how we use machine learning to help us understand how the brain encodes information about movement so why do we want to understand how the brain encodes information well there's two major reason reasons one is just knowing how our brains function can help us to unlock its potential and in addition understanding how it works could help us to restore function after injury and disease and I hope to show you some examples of that today so of course over the last 20 years there has been major advances in spinal cord injury and a lot of this has resulted from team-based science where we bring together neuroscientists engineers and clinicians to work on these problems together so for example we now have methods to do spinal stimulation where people who have paraplegia from a spinal cord injury the stimulation allows the person to make stereotypic Locomotion with their limbs and actually begin to move their limbs in a walking type of fashion in addition we have advanced neural Rehabilitation where robotic devices can go onto the limbs and again help people to make that stereotypic Locomotion that you need for walking and of course people are also working on stem cell approaches where we want to bridge the gap between above and below the level of the injury to try to repair and restore function in my lab as Dr Hank said we do something a little bit different we do brain machine interfaces so as Dr Hanks mentioned I happen to be very fortunate to work on the first demonstration of a brain machine interface and it kind of works like this we train an animal to press a lever with his paw and as he presses the lever down a robot arm goes out and gets a drop of water as he releases the lever the arm comes back and the animal gets a drop of water he's thirsty he wants more water so he keeps pressing the lever the arm goes out he gets water and brings it back and he drinks it he keeps doing this activity over and over again and while he's doing it we're recording neural signals from his brain of the activity of tens of neurons 40 50 neurons and neurons have this interesting characteristic where they are fairly silenced and then all of a sudden they have what we usually call a spike or an action potential that says they're conveying information about what's going on and when we take the activity of any one neuron we can it's basically a bunch of with no activity and then a one every time there's activity we can lay this out in time and I put a tick mark every time the neuron fires an action potential and if I look at any single neuron it really doesn't tell me very much about what the animal is doing but if I take a large numbers of neurons and I average the activity together I can get a pretty good population function that kind of looks like nothing pressing the lever down and releasing the lever so now we play a trick and no longer does this lever moving down control the robot arm but now we switch it so this neural activity controls the robot arm and if the animal does a good job thinking about wanting to press the lever and release it now the robot arm goes out gets a drop of water and comes back because of us translating this neural activity to the movement of the robot arm so I just want to point out that in this case we're using a rodent model in these early studies because we we have shown in many cases that rats are a very good indicator of whether therapies will translate to humans and yes this therapy has been translated to humans if you go to YouTube You'll see a lot of videos of different people who have gotten this technology and the impact it's had on their lives so in this case here's a woman she's quadriplegic she has no movement below her neck and yes she's implanted with microelectrodes into her brain she has a connector just like the animal has in her head and when she wants to use the system they plug her in over here off camera there's a pile of computers processing this information and then the activity goes to a robot arm and she thinks that she wants the robot arm to pick up her water bottle and bring it to her mouth so that she can drink and the impact on people is enormous all right um imagine someone who for 10 or 20 years has had to ask someone to help them do the smallest things every single time they wanted everything and for the first time this woman can do it for herself and you'll see a lot of testimonials of these people um and the impact that it's had on them but what we would like to do is restore the function of One's Own limbs rather than just being able to control an external device so to accomplish that we're working towards something or we've been working on something called a closed-loop brain machine interface and in this case we're going to do something similar so instead of using the forelimb here we're going to use the hind limb we're going to train the rat to plus a lever if he presses the lever for the proper duration he's going to get a water reward and we're going to record the neural activity and we're going to build a decoder that tells us when the animal pressed the lever and for how long then because we want to study the impact of spinal cord injury the animal is going to receive a spinal cord injury and then we're going to instead of taking the information and just using it to control the robotic device we're going to estimate when the animal wanted to move and for how long and then we're going to use that as a command to control stimulation that goes back into the spinal cord and the animal can now move its own limb so we're storing the function of the animal's own movement so we call this closed loop uh brain machine interface and it's basically using functional electrical stimulation to restore the function of One's Own limbs and again this has been translated to humans so this is an example of a subject who we're using non-invasive recordings in this case EEG signals that are from the scalp we take those signals out process them through a computer and then use them uh to use the signals to control functional electrical stimulation so the patient can move the thing to notice here is that this person is in a harness and they need a walker and there are several different Labs working on this but it's true in every case there's no postural stability here so postural stability is absolutely necessary for any kind of movement if I start to move forward I'm constantly making sure that I don't fall over right and this is all this is doing for me is making stereotypic movements of the lower limbs is not able to actually control my posture so posture stability is compromised not only in spinal cord injury but in many other neurological disorders including traumatic brain injury Parkinson's disease stroke and of course even just natural aging we lose our balance more so if we can understand more about posture and maybe perhaps create a postural BMI we could have something that would really allow people to get up and move on their own so in this case if you could imagine we get some sensory information about the fact that we're falling we acquire that signal from the brain we do some processing on it we generate a control signal for stimulation back into the spinal cord and we extend our legs so that we don't fall this is the goal so I hope through the rest of my talk that I can convince you of the advantages of using this frame machine interface approach and basically I hope to show that this goes beyond traditional correlative analysis that I'll talk about more as I go through it enables a quantitative metric of the amount of information that the neurons can represent about the activity the person's trying to do and again I'll try to explain this as I go through my talk and it also reveals how the neurons work together to convey this information because we don't want to just say oh a neuron behaves like this and another neuron behaves like that we want to know how the whole brain is doing this in a way so that we can explain the computations that the brain is performing during motor control so of course there are enormous challenges not the least of which is the information about posture is broadly distributed in the brain so this is a cartoon and I'm obviously not going to go through it in detail but there's all kinds of places in the brain where postural information is encoded there's interactions with information about skilled movements planning and intention other programs and command signals the response generally involves the whole body right if I'm walking and I get a little perturbation I have to extend my limb I'm going to put my hand out so I don't fall I have to stiffen my trunk to make sure I stand up it's not just a simple moving of my arm or or limb and we don't really understand how the brain encodes postural information and these are the kinds of things that we're trying going to try to get at so one we need more data because data is King two we need better decoding algorithms because this is now a very complex process that we're trying to understand and to accomplish this we're going to need some help from computer scientists so we've had neuroscientists engineers and clinicians all working together and now we're going to reach out and we're going to grab a bunch of computer scientists and we're going to bring them over and we're all going to sit down and see if we can't solve these kinds of problems together so we have been making enormous technological advances so this is an example of a new Electro technology and instead of just having a few tens of neurons that we can record from there are thousands of recording sites here so now all of a sudden instead of recording from one location in the brain we're recording from visual cortex motor cortex soundness right I'm a whole bunch of different brain regions each one of these rows represents a single neuron and each tick mark represents when that neuron fired an action potential now this right now is only available in rodents because we tend to start with rodents and get Technologies working there but someday we should be able to get this kind of information from humans so now we're getting into big data and we have so many neurons what do we do we want to know how do these neurons work together because we can't possibly study this many neurons over time for all the activities that they do so here we're going to bring in some of these machine learning tools that Dr Hanks was talking about so we're going to talk about this idea of latent variables and at the end of Dr Hank's talk he talked about how there's all these pieces and if you just look at them correctly they all come into focus and that's kind of what we mean by a latent variable taking all of this information and bringing them into Focus so the idea is generally we have a lot of variables that we observe provide some system that we're trying to study no one can really pick out those variables perfectly so there's always some error in our ability to observe them and we want to use machine learning tools to combine them in some sort of latent variable that gives us Insight we could never have gotten by looking at each one of these individually and to get some sort of analogy about what I'm talking about think about an orchestra and if you sat down and you listened to the oboe player and then you listen to the percussionist and then you listen to the trombone player you'd have a hard time understanding what this piece was supposed to be about but if you put it all together you get some image of what's Happening that goes beyond the score of the conductor which is also listing them all out independently and if you're a sports guy imagine looking at each player one at a time and trying to figure out who won the game right it would be very hard you have to see them all together so it's a it's a latent concept of what's going on in our case we're trying to study the brain we're recording the activity of thousands of neurons we can't do it perfectly we're going to make some mistakes when we record these neurons and we're going to use these machine learning tools to try to identify these latent variables that describe the computations that the brain's doing when someone's trying to control their posture so we're going to identify latent states that ex that explains how the brain encodes information and we think about these as identifying the computations that the brain is performing so we're doing a postural BMI so we need a postural task we're going to go back to the rodent because these are early days and we're going to put the rodent on a platform and we're going to tilt the platform with a little bit of a perturbation and we're going to have four different types of tilts we either tilt them in the clockwise direction or we tell them in the counterclause wise direction we either tilt slowly or we tilt fast and we record the neural activity and as I alluded to before the neurons do fire um in a correlative manner with the behavior of what's going on so we can look at some set of neurons we call them excited neurons that um increase their activity depending upon the speed of the Tilt so if I tilt fast they have a very big response and if I tilt slowly they have a smaller response and then we get these other neurons that we call the mixed neurons and they seem to fire in One Direction sorry they can increase their firing rate if they can tilt it in One Direction and they tend to decrease their firing rate if the Tilt happens in the other direction so okay no this is nice we can say the brain is definitely encoding for the speed of the Tilt and the brain is definitely encoding for the direction of the tilt but is that really understanding how the brain is encoding in the information but we also have these other neurons and I'll just talk about one of them here so for example here's a neuron that has that is inhibited regardless of the direction in which I tilt it's always inhibited so we think for a long time what could this mean what could this be about well if you think about what it takes to maintain your balance if you get tilted in One Direction or the other if I get tilted in this direction I have to extend my limb here and I have to flex my limb here so I can balance myself if I get tilted in the other direction I have to extend my limb here and flex it on this side and the last thing you want to do is extend and flex the same neuron at the same time because that's like a charlie horse and it's very painful and it's absolutely not functional so the brain has to be very careful never to send a signal that's going to extend and flex the same limit the same muscle at the same time so these things have to be kept separate and we imagine that perhaps maybe that's one thing this inhibitory component is doing is they're being active when and inhibiting other populations of neurons when this side needs to flex it's inhibiting the extensors and when this side needs to flex it's inhibiting the sensors on this side but we don't really know so we're going to go back and we're going to try to understand are there any latent states that can help to explain how the brain is encoding balance control so the thing about latent variables are they're not any direct measure of anything so we have to kind of plot them out and see what happens so this is an example of what these latent variables look like the dots in the middle here or the animals standing on the platform before it tilts as the Tilt happens each one of these lines is a separate trial where the animal was tilted and you can see that the information curves around when the dot dotted lines start it means that the animal is being tilted back so this is the animal being tilted out and then this is the animal being tilted back all of the Tilt in One Direction are on this side of the plot and all of the tilts in the other direction end up on another side of the plot so there's a clear split between tilt in One Direction until it's in another Direction and the other thing you'll notice is that this is the slow and the fast tilt together here they kind of lay on top of each other so it doesn't seem to be a lot of difference between a slow and a fast tilt in this direction and there doesn't seem to be a lot of difference between the slow and fast tilt in the other direction so what can these latent brain States because this is now what I'm plotting here tell us about the computation that the brain is performing during these postural adjustments well we've got the animal tilting out and back we have this trajectory that starts in the center of this latent space as the animal gets tilted out they curve around to the right if they're a right tilt or they curve around this way if they're a tilt in the other direction and then they kind of follow a circle and end up back where you started from as the Tilt comes back so the first thing we notice is I know as I mentioned before is that latent variables for tilt in the same direction overlap independent of speed what this is actually telling us is that the computations for different magnitudes of tilt so if I push you a little bit or I push you a lot the computation in the brain is the same because you need to recover from that tilt the computations just run through the the brain faster for faster tilts which makes sense because if I'm tilted slowly I need to slowly give that information to the muscles to stop myself from falling over if I get perturbed fast I better send that signal down and have those muscles work much faster second thing we'll notice is that the latent variables for tilts in opposite directions never overlap so the tilts in One Direction or on this side the tilts in this direction are on the other side and they don't overlap and we think that this has to do with the computations that require extension inflection of one set of muscles is separated from the computations when I need to then flex and extend those muscles and this is one way in which the brain could separate out activity to ensure that you never co-contract like centers and flexors in the same limb so this is all very nice but can we actually decode the tilt obviously I wouldn't be here if we couldn't so here's an example of in solid black we have the true position of the platform and in colored we have the prediction of the position of the platform from our latent variables and you can see that as we tilt in One Direction or tilt in the other direction or we do a slow tilt or we do a fast tilt we are able to predict the position of the platform very well from these latent variables so not only do they give us insight into how the brain might be performing computations to maintain balance we can also use them as a decoder to send to let's say a spinal stimulator to write to help the patient maintain their balance during a perturbation so just to summarize this part of the talk we have different neuron types that seem to encode for the speed of the Tilt some encode for the direction of the Tilt and some team to gate information so that we don't have co-contractions we're able to identify latent subspaces that separate out tilts in one direction from tilts in another Direction and notice that the time at which the act the computation is performed is dependent upon the speed at which I need that information and finally we can use this information from these latent variables to decode the perturbation that the person has been subjected to so all very nice but does any of this remain true after spinal cord injury because we do want to understand how the brain works but then we want to take that knowledge and use it to create devices that are going to help people who have neurological injury or disease so normally we're here we have our brain we have sensory inputs that come from proprioception or sensory information and that goes up to the brain the brain does its computation and it sends motor output down to our muscles to make us move you have a complete spinal cord injury there is no information flowing from your brain down into your limbs and there's no information coming up from your sensory inputs to your brain mostly we generally have incomplete spinal cord injuries where there's some information going back and forth but it's very limited so can we still see these same computations in the brain and can we use them to decode the perturbation that the person is being subjected to so again we're going to have the animal go back on the platform we're going to record from his motor cortex and we in this case we did two groups of animals we did animals that received therapy and animals that did not receive any therapy and this therapy was just strict um treadmill therapy for them after the spinal cord injury so um before spinal cord injury there are most of the neurons encode some information about the Tilt but there's always a few neurons that are carrying no information they're doing something else if I don't give the animal any therapy more and more neurons over time stop participating in the activity and there's less and less information about what is happening to the animal during the tilting if I give the animal therapy then this loss is greatly attenuated so that there isn't even a significant difference in the number of neurons that are encoding late this is nine weeks after injury compared to early which is one week after injury so we have a lot of confidence that this information is going to be there then we looked at the activity of the neurons and we see things very similar to what we saw prior to injury despite the fact that we're not really getting the sunscreen information the same way we got it before and despite the fact that there's really no motor commands going out and basically we see neurons that this is pre-injury the ones that fire um with the magnitude of the activity changes depending upon the severity of the Tilt we have ones that go um fire in one direction for one types of tilt in the other direction for other types of tilts and we do see neurons that are inhibited but not as many as we saw prior to spinal cord injury then we wanted to know are these latent variables similar post-spinal cord injury as they are pre-spinal cord injury so we generate this learning algorithm we regenerate these latent variables and we plot them in latent variable space and here you see that the dotted line is pre-sci and the solid line is post-sei these are tilts in One Direction these are tilts in the other direction and we have a pretty good lineup between pre and post SCI so it appears that the computations that the brain is performing after spinal cord injury is really not that different than the computations they were performing before now the really Telltale sign is can the brain continue to encode for the angle of the tilting of the platform after spinal cord injury and the answer is pretty much yes so again we have the same situation where the solid black line is the true position of the platform the colored lines predict the position of the platform this is what I showed you early pre-spinal cord injury we can match the true movement of the platform with the brain predicted movement of the platform this is one week after injury five weeks after injury and so far we've gone nine weeks after injury and we can still predict the position of the platform so it's likely that we can use this signal as a control signal for spinal stimulation to keep the person righted so what is this future Direction look like so um I'm an engineer and I think in latent space so if you think about it I showed you at the beginning there were sort of these dots in the middle when the animal was on the platform and so this is kind of a balanced State and we're here and there's no problem and there's no issues I get a perturbation and I'm going to get out of balance but it needs to be somewhat controlled and hopefully with some kind of stimulation we can push it back into a controlled State and stay inside of a controlled or equilibrium state if we go out here into the red all bets are off and you're going to fall over so the idea of this control system would be to stay in the green as much as possible as we move out of this green latent variable State we would stimulate and move us back into this green state so we could maintain our balance um so our goal now is to build a control system that continuously decodes the brain State regarding balance and transmits the appropriate information to a spinal stimulator to be able to maintain that balance so now when we get the people up and walking they can do it without the additional AIDS of Walkers or harnesses or something like that and so lastly I just want to thank all the people that worked on this I had three great graduate students over the last few years Greg Dice and Andrew Kumar and Gary Blumenthal that did pretty much 90 of the work on this and of course our collaborators Dr xiaodon Kong in the department of mechanical and aerospace engineering so much and I'm really happy to answer any kind of question so please feel free to um you have a question probably somebody else in the room has the question too so just raise your hand and we'll bring microphones around to anyone who has questions I see a hand up up front hi uh thank you very interesting so uh you kind of grouped you started looking at the Step at stereotypical step as you mentioned is kind of a shortcut to understanding that the dynamic is there also an effort to look with finer Fidelity into the like individual firing of muscle fibers kind of a more detailed look do you think there's any benefit to that are there significant that's a great great question um and and this is this is a question both not only on the motor output but if you imagine on the sensory input as well where do you interface with the system so I have a little bit of a Prejudice and I like to go as far away from the end effector as possible because I want to give that system the ability to do most of the work I think personally that it would be much harder for me to say oh this muscle at this time then that muscle then this muscle then the other muscle the spinal cord is actually a complete intelligent system and so I can actually stimulate it's kind of hard to explain but it has a structure to it so that if I stimulate it will actually make a reasonable kind of movement and we have found that we can link together different stimuli and make more complex movements and these things tend to be additive which is always nice for us because then we're kind of living in a linear system world and so it's easier to stimulate into the spinal cord and let it and the muscles work together than it is for me to try to say oh this muscle then that muscle and get all that timing right I might end up being wrong in the long run but this is kind of how we think about it now so that's a great question any other questions thanks uh I'm not a neuroscience so forgive any uh misstatements here but I'm uh as as the two of you were speaking uh Dr Hanks as well I was thinking about and making analogies to my niece who has a genetic disorder cdkl5 which has left her subject to um uh seizures that have to be controlled pharmaceutically uh non-verbal non-ambulatory and uh my sister and her husband spend tens of thousands of dollars every year sending her for physical therapy to try to train her brain to you know stand upright or to to try to take a step and they're just at that point so this is a rather fanciful question rather futuristic but thinking about uh Dr Hank's discussion about the inputs coming from the uh from the robot vacuum the cleaner and my niece who it is not clear is generating any sort of inputs from her brain to her muscles and then like trying to marry that with what you were describing yeah is there any is there any hope for bringing those two together yeah so I I that again that's another really great question um there's a lot in there to think about and one thing that many many people are working on in my lab is moving towards this as well because we did actually spend some time and we still work on epilepsy with if you could imagine a closed loop epilepsy system where we record the activity as the seizure prior to the seizure coming on we're looking for pre-seizure activity and we can stimulate to prevent that seizure from happening there is lots of ways this is now neuromodulation is taking off so for Parkinson's Disease many of you may know you take drugs in the beginning and eventually the drugs don't work so well anymore so you get a Parkinson's stimulator and then that helps control your Tremors and things like that so they're now thinking about neuromodulation for depression they're thinking about neuromodulation for obesity for the complexity of of the problem you're talking about I don't know when we might get there but there's just an enormous amount of work going on just modulating directly into the brain to try to take this firings or misactivity and push them from a I call them the bad State back into the good State yours is a very complex problem but there are things moving in that direction one last question what does neuromodulation oh that's a fabulous question and I'm so sorry for using that word so I was talking about stimulating into the spinal cord so this is stimulating with electrical currents into the brain thank you very much thank you