Optimizing Brain-Machine Interfaces: Control & Learning

[00:00:00]you [Music] good afternoon thank you for coming to this talk we have the pleasure to have a male's body from University of Washington this afternoon to give a talk about her work on brain computer interfaces Amy to her PhD from Berkeley has several years postdoctoral shippin York University and now she's a system professor and a director in the Department of Electrical and Computer Computer Engineering University of Washington alright thank you very much so my research is really interested in brain machine interfaces which the idea is very simple that we can directly interface your central nervous system with the external world so technology allows us to make measurements from your brain and that allows us to get information out but we also have the ability to do stimulation where we can drive the activity neurons in your brain with things like electrical stimulation or optical stimulation and so this opens up a lot of exciting avenues for basic neuroscience studies but also opens up ways to think about treating neurological disorders so if you have someone who's paralyzed for instance because of a spinal cord injury they're paralyzed because the brain can't talk to their muscles anymore but their brain is still entirely intact and healthy so if we could use sensors to record their thoughts as they're thinking about moving we could send those signals to a computer for instance to allow them to control a cursor again or type and so this you know there's a lot of different flavors of brain machine interfaces you probably are very familiar with things like cochlear implants retinal prosthesis my particular area of research is on this the motor brain machine interfaces the idea of restoring movement to people who are paralyzed due to things like spinal cord injury and stroke and so this idea has been around for quite a long time and we've made some really exciting strides there's clinicals demonstrations in human patients multiple sites now one from Brown University and another from University of Pittsburgh where they've had been able have great success with paralyzed patients who have a variety of different injuries like ALS or spinal cord injury and they implant sensors into their motor cortex and they can then record those signals and allow them to control a robotic arm to do really basic tasks of daily living like reaching out and grabbing a cup to take a drink but there's a really long way to go before these proof of concepts are going to be real viable clinical therapies so just some examples of tangible challenges we face one is that the performance is really far from what we can do with our natural limbs none of these people are going to be playing the piano anytime soon with these brain machine interfaces doing complex tasks that require very high-dimensional dexterous movements for instance also the performance is relatively limited in terms of time there's a lot of variability from day to day as subjects try to use these devices and also typically performance sometimes declines due to issues in the sensor biology interface as well and there's also actually a lot of variability in individual outcomes so you'll within certain parts of the community of BMI researchers they have a term for this called BMI or BCI illiteracy there's a subset of the population on the order of 20 to 30% that actually really struggle to ever use these interfaces successfully and it's unclear why that might be so these are just some examples of kind of concrete specific challenges but more broadly the issue is that we have kind of very little principled or mechanistic understanding of how these interfaces work we this complicated system that we've created by interfacing the brain with engineered systems we still kind of don't understand how that works at a more mechanistic level and so that leads to kind of this problem of no design principles so if you gave me two patients one with a spinal cord injury one with a stroke they likely need very different interfaces to because they have different disease diseases underlying their paralysis but I don't have any sort of good way to principally design interfaces customized to those particular people and so one of the reasons that these design principles have been really elusive is because brain-machine interfaces are actually a lot more complicated than we originally thought so typically people have always thought about this field kind of as a decoding challenge we have neural activities in our brain that represents movements that we're trying to make and so if we can build the right decoder we could extract the perfect control signal to send to the cursor or the robot that we're trying to move problem-solve right but this is actually just a subset of what's happening in a brain machine interface once we have that control signal that we've decoded it goes to a particular actuator that has certain mechanics and properties of how it moves and then the subject has feedback about how it moves and they can use that to update their strategies so over the last several decades brain machine interface research has made kind of two critical observations one is that the neural encoding how neural activity relates to movements changes between brain machine interface control and arm movement control so even if I have a decoder that could perfectly predict your arm movements and it doesn't mean that I can give you a brain machine interface that's going to give you great performance the second is that neural encoding actually can change with practice and that subjects learn and change their strategies of how they use a brain machine interface over time and so these two observations initially were somewhat head-scratching from the field but actually if you think about the system kind of make a lot of sense one is that we are not in a brain machine interface where I'm say asking you to control a computer cursor you're controlling something fundamentally different than your arm right a cursor is massless it's frictionless it has very fundamentally different properties than your limb so it makes sense that the brain might change its encoding strategy and control strategy to move a cursor versus your arm also the adaptation and learning that we see is obvious because this is a closed-loop system subjects have feedback and brains are sort of optimized and designed to learn that's what our brains do all the time and so these observations kind of point to the fact that rather than thinking about BMI as purely a decoding problem we need to think about it as designing a closed-loop control system where we have an adaptive element like the brain that can learn to control this new system and so this is kind of the perspective that fuels all of my research thinking about kind of how to optimize this new closed-loop control system that we've created in brain machine interfaces I approach this from a couple different directions one is kind of from an engineering perspective given this new view sort of rethinking some of the design choices that we make as engineers so we can kind of treat the brain as some black box that we know is adapting and learning in some way it has some encoding model we don't understand and let's think about how we can design all the other elements of the BMI loop to deal with this black box the other is motivated more from a neuroscience perspective where we try to think about opening up that black box and actually studying some of the neural mechanisms of learning and control that are happening in the brain the goal of course is that these two research directions are going to inform each other as we learn more about the brain its encoding models control strategies we can better tune our models as engineers to design the performance of the system and so on also some of my work involves technology development because we're still kind of in a relatively new stage of neuroscience and so we also are doing a lot of work to develop new tools to try to interface with the brain and make a lot of different measurements that will inform some of these studies so today I'm mostly going to tell you about you know the first thread of research Mooka focused from an engineering perspective of kind of how this closed-loop perspective informs our design choices of different elephant elements in the motor brain machine interface so all of the work I'm going to tell you about uses kind of common motor brain machine interface paradigm in research I work with animal models so we work with monkeys and we do invasive recordings in their motor and premotor cortex and so these parts of the brain are very heavily involved in controlling your arm and hand movements and also have really strong projections to your spinal cord we put in sensors into there into the monkeys brain and you can think of this as kind of just a bunch of wires we're recording from very small exposed tips on the ends of those wires and that allows us to record the activity of individual neurons within the brain and these neurons fire action potentials and so most of the signals that we're dealing with are sort of these discrete events they're sort of all or nothing or neuron is either firing or not firing and so these are discrete sort of Poisson processes are the signals we use a decoding typically really simple decoding algorithms things like linear filters something like the common filter for example and so what the common filter is doing is taking the firing rate so we take this poisson process and convert it into a rate and use that to predict the common filter to map that rate of neuron firing into kinematics of a cursor on a screen and we're typically using just a relatively small number of measurements something like twenty to thirty neurons the animals are then controlling the position of a cursor that's moving around in two dimensions on a computer screen so position and velocity in this case are the kinematics for decoding and the subject has visual feedback of that cursor moving around and so that closes the control loop one other thing to note that sort of I think often fascinating to think about is that these subjects are controlling this cursor without actually making over our movements so these are able bodied animals they're not we don't have disease models but they're actually not moving at all they're just thinking and changing their neural patterns of neural activity and that drives the cursor around yep firing rates and going to kinematics is that are you doing that because that's the way the muscle action activation works so we use firing rate basically because the current understanding of neuroscience is that likely neurons are kind of communicating via a rate model that rate is kind of the language of neurons and then kinematics is in some ways a convenience choice if you are I could give you know a long lecture about how we still actually don't understand much about how motor cortex the the sort of representation scheme of motor cortex is still really unclear whether it's controlling muscles or kinematics is really hard to tease apart in part because muscles and kinematic variables are heavily heavily correlated so we don't actually know very much about the exact representation what I can say is that we can AB we can predict kinematics quite well from patterns of neural activity yes yeah a position and velocity yes so the your follow is firing rates too rate of movement and position yeah all right so throughout this talk basically we're going to walk through kind of different elements of this loop and kind of visit certain choices that we make as engineers and think about how that influences brain machine interface control so the first question we're going to talk about is sort of the broad idea of how the loop itself the properties of this control loop are actually going to influence control in the system so one obvious question when you're thinking or property of a control root loop is the rate at which it operates so in brain machine interfaces typically rates have been chosen kind of based on decoding perspectives right if you want to decode information you probably will get better and better decoding outcomes if you sort of average neural activity over longer periods of time and so but from a control perspective the rate at which someone is allowed to control occur obviously going to influence their performance and so what we did was design experiments though we could actually test very specifically control and feedback rates within the loop and how they influence performance and do this in a way that's separate from the rate of decoding so there's a lot of different rates that happen in a brain machine interface if we think about the forward path where we're going from neural activity to cursor movement there's the rate at which we bin our spikes since we want to go from these discrete Poisson processes to a rate we have to have some bin that we estimate that rate over we also then once we have our firing rates decode the position at a certain rate and then finally we have the rate at which decoded positions are pushed to move the cursor so this last step is actually really what defines the control rate but typically people have kind of lumped all of these together and but even though these rates are actually associated with decoding on because we're using a virtual system on the feedback path the mapping from cursor movement back to subject feedback is just purely set by the screen update rate so how often were pushing the updated movement of the cursor to the screen to show it to the subject so what we wanted to do was actually isolate this control rate from control and feedback rate from sort of this rate of decoding and so we did that by using a rate independent decoder that uses a point process filter and I won't go into kind of the math and the details but basically the idea is that rather than basically we're having we have a decoder that operates sort of better fits the plus on statistics of firing firing of neurons and that allows us to decode neural activity sort of on a spike by spike basis so it operates on zeros and ones so we can now do our decoding on a very fast rate in our case we were using sort of five millisecond rates of decoding and that means that now that we have a very fast decoding rate we can subsample our control rates and do those manipulations independently of the decoder yes yeah yeah so we have a yeah a model is fit for each neuron and then they're all combined together to get the ultimate estimate of the position and velocity yeah and so with this sort of this point process filter where we can separate out the decoding rate from the control and feedback rates now we can do manipulations of each so in a first context we have a fast feedback and fast control rate scenario where we're decoding on a spike by spike basis every 5 milliseconds and every time we decode we immediately push that to move the cursor and we also immediately push that to the display to update the subjects so that they can see that that updated cursor position on the other end of the spectrum you could imagine doing a slow control and slow feedback decoder we're now the decoding is still the same we're still decoding on a spike by spike basis but we subsample the pushing that decoded position into moving the cursor and same thing we're only showing the subject the cursor when it's actually been moved in the middle we can separate feedback and control rates by doing a fast control and slow feedback condition we're now we're decoding and controlling the cursor at a very fast rate but we're subsampling and only showing the subject on a smaller slower time scale when we've actually updated the feedback about the cursors position note that the inverse of this where you'd be doing slow control but fast feedback is sort of an unmeaning 'full test because you just be pushing updates to the cursor right so this together kind of isolates all of the feedback and controls great conditions for aficionados that are curious in our case fast for control means 200 Hertz so we're using five hundred or five millisecond bins slow is ten Hertz which is actually a really common rate that's used in a lot of brain machine interfaces on the feedback side fast was actually just set by our monitor refresh rate we didn't use ultra fast monitors so we just had sixty Hertz and then slow feedback is again ten Hertz so now what I'm gonna show you is the performance of animal and animal on a task in success rate so the trials per minute that he's getting correct for some of these different comparisons of these different types of decoders and over here is just the the chain the relative change so improvement in performance that you get by increasing either control or feedback rates so first if we compare the slow slow to fast slow condition we're now isolating the effect of increasing the control rate and so here what we see is that by increasing the control rate we get an improvement in performance so higher success successes per minute and this is actually very interesting because the subject doesn't actually have feedback about the fact that he's controlling the cursor faster right yet he still gets improvements in performance so is initially slightly confusing but if you look at the motor control literature of how we control our movements there's actually a lot of evidence that we have feed forward control mechanisms that we have certain amount of modeling of how our movements happen that are completely independent of feedback and so we can control our movements in a feed-forward way and this suggests that the same type of thing is happening in brain machine interfaces and by allowing the subject to control faster we're getting improved performance even though he can't see the cursor moving that fast so next if we control compare the fast slow to fast fast condition now we're isolating the effect of increasing the feedback rate and so here again we see an improvement in success rates and this points to the fact that feedback also is really important in control which is consistent with the fact that feedback control is very prevalent in the way that we move and is also going to be true and brain-machine interfaces and then finally yeah yes this is a this is a center out task so a trial is moving the cursor to the center then moving out to a peripheral target and we don't move him back to the center so he has to also it's sort of an out and back task so he asked to there's so for yeah in for for trials in a minute you know it would mean eight out and back yeah yeah so they're going quite quickly this is this performance is comparable to what animal the rate at which animals do this task with their arms and so then finally if we compare the slow slow to fast fast we see an even larger improvement in performance between by combining the effect of control rate and feedback rate and so this suggests that you know these two improvements and performance are actually somewhat independent of each other so both feedback and control rate contribute to improve performance and so finally I wanted to mention that by combining this point process filter that allows us to do fast feedback and fast control we were actually able to beat the state-of-the-art performance in brain machine interfaces so in blue what I'm showing you is the this same metric success rate for this point process filter with fast feedback in control and an orange is the sort of previous state of the art of a common filter that's operating at ten Hertz the feedback rate is still faster but the control rate is ten Hertz and so you can see that we get about a thirty percent thirty to twenty to thirty percent improvement across two different subjects and this is just an example showing you the trajectories of the center out movements and you can see that there's quite a remarkable improvement in terms of the straightness of the trajectories and so obviously this improvement in performance is nice but I think what I particularly like about this is that we were able to figure out where it came from right so we know that this 30% is a combination of improving the feedback rate improving the control rate and actually some statistical details of how the point process filter by modeling the Poisson statistics of the neurons we also also actually get sort of an improvement in performance from there yeah so I do think that the individual variability across subjects is actually very interesting we didn't in monkey experiments are also often sort of underpowered in terms of the total number of subjects but I think it's a really interesting question of whether you know the degree to which subjects rely on feedback versus control feed-forward forward control might actually vary across subjects right and so that might contribute to some of these differences great all right so in this example I hope I've shown you that you know by not just focusing on decoding but thinking also about the control in the system and sort of the rate and how that influences performance and we get significant benefits but all of that work that I was showing you focused on where we already had a trained decoder and I kind of glossed over how we trained that decoder at all right and so as I mentioned before the encoding model typically changes between a subject controlling their arms versus controlling a brain machine interface and that leads to a bit of a problem of how do we actually train this decoder in the first place so one possibility if you have an unknown encoding scheme would be to have an adaptive decoding scheme so that you could gradually over time learn the proper parameters of the decoder kind of Institute as the subject is using the interface so this idea is closely I call closed loop to code pation and simply if you have subject using an interface and closed-loop you can look at the errors and mistakes that they make and use that to update the decoder parameters that they're you that you're using for control and this idea has been used in quite a bit in brain machine interfaces for a wide variety of different challenge to address a lot of different challenges I've particularly focused on this in the context of kind of reliably and robustly learning this decoding algorithm when you have no AA priori idea what the decoder parameter should be so I want to sort of start from any random initial decoder parameters and get to stable final conversions converge to a stable solution no matter where I start from and this becomes actually an interesting problem in part because the subjects actually trying to learn at the same time as the decoder is trying to learn so you now end up in a situation where the decoder is trying to learn what the subjects doing and the subjects trying to learn what the decoder is doing and you have to make sure that these two learners interact synergistically and so there's a lot of so the details of how you actually do this adaptive decoding become really important one of the most important things we found ended up being the time scale so one of the most important things is that the decoder has to learn faster than the subject otherwise the subject starts chasing the decoder and so sorry this the decoder starts chasing the subject if if you're not learning faster than the subject and so one of the algorithms that we first came up with to do this we call smooth batch it's actually really simple and from a kind of adaptive decoding perspective is suboptimal in a lot of ways but it proves to be really powerful in this context of brain machine interfaces so if you have a subject that a monkey that's trying to use a brain machine interface in closed loop we can do a supervised technique where if he's trying to get this orange cursor to the blue target and let's say that his actual velocity is going off in the wrong direction since I know what the target is where he's trying to go I can for his actual intention and used that as a training signal and so now I can look at the activity in his brain and what I think he's trying to do and use that to create a new update of what my decoder parameter should be and in this case because we want to update the decoder quickly we do this with just sort of a small batch of data about one to two minutes and then we push that to update the decoder but because this is using a very limited amount of data we don't want to just purely rely on this one batch we want to kind of gradually evolve this over time with sort of a sliding window average and so what we're doing is sort of slowly bootstrapping the performance over time and this has a couple advantages one it allows the decoder to sort of update quickly and it avoids things like subject frustration but also because we're doing the smoothing average we're very robust to sort of noisy data so let's say that a subject gets distracted for a portion of a batch and you now have a really noisy bad estimate by doing this sliding average we can kind of smooth a lot of those things out so here what I'm showing you now is an example of what this adaptive scheme looks like in practice so here's again this rate metric events per minute when black is successes so when the subjects getting the trial right and in blue is a reach timeout which is just an example of an error and so when we start with a random decoder he actually can't even make mistakes the cursor is just sitting off in the corner of the screen not doing anything at all and so eventually as we start adapting he actually starts being able to make a couple mistakes he occasionally gets a trial right and then he starts actually being able to succeed in the task and performance starts to Plateau importantly if we stop adapting the performance remains high so this suggests that we've sort of converged to a stable solution we found some decoder that the subject can use readily and input really importantly this algorithm is really robust and reliable so we trained it across you know 56 sessions with a variety of different ways of creating this initial random guess all of them being completely agnostic movement and consistent with a paralyzed patient and so in all cases because we're not really using much information our initial performance is terrible but we get to the same final performance in all cases so we're independent our final performance is independent of where we start and we can also get there relatively quickly so we hit maximum performance on average of about 20 minutes or so even from any random initial seed yep and once he's trained on that yes so they are trained with the task they know the task rules separately from learning how to do the brain machine interface control so we train them on the rules of the task first with like arm movements for instance yes versions research versions of bananas are definitely involved in the training yeah and so one you know once we sort of figured out some of the smooth batch was sort of a starting algorithm and once we figured out kind of the general rules and how this worked we then went back to the drawing board a little bit and refined this even further so based on the observation that the rate of decoder adaptation mattered we then went back and designed some new algorithms that work to update the decoder parameters really rapidly so on every iteration of the decoder you can actually use something things like gradient descent and other strategies to update on a really rapid time scale and this pushes our convergence down quite a bit so we can now go from terrible initial performance to maximum performance in the order of just six minutes or so and also we also started thinking about the best ways to estimate a subjects intention so you can also start thinking about making optimal feedback control models of how subjects are trying to control the cursor to actually get a more principled estimation of their in tents and this allowed us to improve the success rate of so how quickly subjects are moving the cursor and without sacrificing any accuracy so they're still making the same level of errors but they're moving much faster alright so to quickly summarize this you can use adaptive decoding to kind of deal with this issue of the fact that we don't necessarily know ahead of time what the brains control strategy might be and this it's important given that the subject might be trying to learn the rate that the decoder adapts is going to be really important and that we can do this really rapidly and we can robustly as well but now we have an issue of how do we actually maintain this performance long term all of those examples I showed you was for one example session where I trained that decoder on that day and then I tested it on that day how do we start thinking about making a device that someone could come back to day after day and reliably use yeah in that case it was a common filter yes so because not not too much an in part because of the fact that there's very little kind of correlation between how well a model predicts neural relationships between movement and neural activity doesn't really translate into improved performance and so we haven't focused on alternative algorithms one of the reasons I like the common filter is because it's very interpretable from kind of a control perspective it's very clear what signals are being sent and the mappings between neurons and and control signals but but there's there's a wide space of algorithms that people have used I don't think that the interesting thing is that really complicated networks like even recurrent neural networks the people have used have not shown you know leaps and bounds improvements in closed loop performance in part because of this issue that it's not actually a decoding problem yeah right so if we're trying to think about maintaining performance one of I will propose that one of the ways that we might want to do this is start leveraging the learning and plasticity within the brain thinking about that element of it that this adaptive decoder is also dealing with an adaptive brain so one of the reasons that we have a challenge with consistent performance across days is because we have measurement variability so if we're putting these little wires into the brain to try to record activity of individual neurons any movement between the wires and the brain is going to shift what signals you're actually picking up right neurons are teeny tiny so even micro motions will change your signals and so we have this issue that we often have variability across days so typically the strategy would be that we just retrain our decoder everyday if I'm not recording the same neurons why should I assume that I should use yesterday's decoder I should just start over again from scratch and so this would avoid kind of declines in performance over time and in fact you can see that you know if you do this where we have this is the average performance on tasks across days where every day I've retrained the decoder you can see the performance is generally quite high but you also see that there's a lot of variability right you would be pretty frustrated if every day that you picked up your iPhone or whatever phone you have your typing was worse or better depending on the day and so this regular retraining sort of avoids declines in performance but it doesn't actually help with variability at all and so one possibility is that what we're doing is kind of disrupting the subjects ability to learn on longer timescales and so you know this would be in some ways equivalent to me asking you to learn how to play tennis where every day I gave you a different racquet there was slightly different weights or different sizes and you can imagine that because we're constantly changing the decoder on the subject in several ways and so one possibility is that we need kind of decoding strategies that are compatible with long term learning phenomena so a strategy that we came up with we call sort of co-adaptation and it's again a really simple idea that we have an initial decoder we then do our decoder adaptation to try to boost up performance initially and then we keep it fixed and so we try to record as best we can stable signals and we keep the decoder the same and this will allow plasticity and learning on the side of the subject over time but if we have a shift and measurement noise sort of leads to a reduction in performance we can go back so rather than starting back at the drawing board we should just go back and make slight tweaks and adjustments to this so if I've lost a neuron in my ensemble that was contributing to the decoder I can swap in a new one but keep all the rest of the neurons that we think are the same still in the ensemble and so we can retain performance and kind of make just gradual shifts over time and the hope is that the brain can actually track some of these slow subtle changes over time and so here I'm going to show you a video of what it looks like when we do this and so this is day one with the performance of the initial random seed so the subject is trying to get that white cursor to the Green Center target you can see that he's really struggling to do that now this is the same day but I've run my smooth batch decoder adaptation algorithm and so now he's actually able to do the task but he's still somewhat sloppy you can see in particular that he sort of really struggles to actually kind of precisely get to the target and stop it's one of his biggest problems now this is day three where we've left the decoder the same the only difference here is that he's had time to practice and improve so now you can see the performance is still sloppy but much improved and in particularly sort of more precisely getting to the target and stopping now this is day 13 where we've done gradual small tweaks to the decoder over time and he's had time to practice and you can see that performance is quite good where we have very straight and quick trajectories between the center and the tar on this performance again is pretty close to what they can do with their own arms alright so this is just a graph quantifying the performance that you saw in that video so percent correct over different days so we start off with terrible performance we boost it up using dakota adaptation and then we can continue to improve from there overtime and all of these little blue dots are times when measurement variability sort of led to reduced performance but we were able to do this sort of small tweaks to the decoder overtime and maintain this consistent performance and this also worked in the second monkey one of the interesting things that we found when we did this is evidence to suggest that by incorporating brain learning we could actually get better performance than we could with just purely machine learning alone so here I'm showing you an example where on day one we sort of fully maximize performance as much as we possibly could on the first day so he got up to a hundred percent correct on day one and he was able to maintain performance over time by keeping the same fixed decoder and then occasionally tweaking it when we needed to if you look though at the success rate what you see is that over time he's getting better and better and making these movements faster and faster he's also if you look at error he's actually reducing his error as well and so this suggests that incorporating brain learning and long term learning might not just reduce would both reduce variability across days but actually also give us better final performance if we allow the subject to learn and kind of master this skill of controlling the interface if we now look at what's happening in the brain it's interesting to think about you know what is it that the subjects actually learning when they're controlling these interfaces and sort of the short story is that they're kind of learning which neurons are contributing to the cursor movement and how to recruit those really precisely and so what I mean an example of what I mean by that is here I'm showing you an example of one neuron on a particular day this is what we call a direction tuning curve so this is the direction the subjects trying to move the cursor and this is the firing rate of a neuron that neuron at the time he's trying to initialize a movement to go to that target and we fit this with a cosine model which is a very common model that we see in motor cortex neurons and you can see that you know the neuron sort of fires most four movements in one particular direction and then opposites four movements sort of 180 degrees opposite it fires the least if we look now at what this neuron does across days as the subject is mastering this task the encode the preferred direction you know where this neuron fires most is still the same target but it's actually firing more and more kin more generally speaking so the subject is learning that this neuron is important and making it fire more often this is just another example of actually a neuron with an even more extreme case where initially this neuron had no consistent relationship to movement at all but over time with practice the subject learned that this neuron is important for controlling the cursor and actually started making that neuron fire we also see evidence that they're sort of faster and tighter temporal recruitment of neural activity as well and these patterns become more and more stable over time and this is really consistent with a lot of evidence from the neuroscience literature that this is kind of a hallmark of learning a skill and so this points to the fact that you know by neural and decoder adaptation can kind of work together in a synergistic way and that brain learning might be really important for kind of creating robust interfaces and giving subjects kind of the opportunity to master this skill of controlling the interface and importantly that kind of what they seem to be learning in this skill is really which neural activity to recruit and when yes both so the and I can maybe we can come back to this question because I've got I think a slide in a couple a couple of slides that will maybe hopefully sort of ice point out I think what's happening in this in the big picture of what has to be learned in a brain machine interface you can kind of divide it into two things and I think when you have decoder and in this scenario they're decoder is learning one thing and the subject is learning another thing is kind of how I think about it so let's come back to that so one one quick point I wanted to make is that some of our next steps for thinking about how to continue this research is this is all just controlling a two-dimensional cursor right and there's kind of obvious interesting questions to think about how we would scale this to higher dimensions so going from a cursor to something like our hands it has on the order twenty-seven degrees of freedom and this in part requires a lot of work that we're doing to try to you know think about how to really track and study high dimensional movements and in terms of motor learning neuroscience we actually know very very little about how people learn in high dimensions so there's sort of interesting challenges both from the engineering perspective of how to design algorithms that are optimal and scale optimally but then also kind of really open questions of how subjects are even learning at high dimensions all right so in the last little bit I want to talk about kind of go back to this question of you know what is it that the subjects are learning and kind of point to some of the new work that we're thinking about in terms of how to optimize this learning on the subject side so in this sort of Co adaptive paradigm one of the things I started realizing is that the subject actually in many ways is the bottleneck of this performance and I'll hopefully try to explain what I mean by that in a second so if we think about what's happening in brain machine interface where we're going from neural signals over to cursor movement what we're doing is placing our sensors in some randomly selected part of brain and we have you know thousands of neurons within your motor cortex and we're arbitrarily picking some small subset of them to control the cursor and then though the firing of those neurons is going through some mapping of the decoder right in a linear filter this is just a simple linear mapping that you know compresses the dimensionality from the number of neurons you have into the number of control dimensions so there's you can sort of break this big learning down and to kind of two things one is modulation learning what I call modulation learning and that means that the subject has to figure out how to make those particular neurons the neurons I've highlighted in purple fire reliably if those neurons never fire the cursor never moves right and so and if they never fire reliably the cursor would never move and given that we're kind of you know play arbitrarily placing our sensors in some random part of the brain it's in some ways not obvious how the subjects are figuring that out the second is a mapping problem that given a particular pattern of neural activity that you're able to evoke how does that relate to the cursor activity and so I think of closed loop decoder adaptation the decoder learning is learning and simplifying the mapping problem right it's taking a given pattern of neural activity that a subject has produced and create simplifying and learning the right mapping for a pattern of neural activity that a subject produces but how is modulation learning happening at all right we don't actually know so what I kind of going back to the observations we made previously the fact that what happened the changes in neural activity that are happening are neurons firing more suggests that the subject is in some way solving this problem of figuring out which neurons are contributing to the cursor and making them fire more so that's what they're solving they're not dealing with the mapping problem because we're helping them with that via the decoder adaptation they're figuring out which in the world of my billions of neurons and my brain do I actually need to make fire and so going back to this kind of issue I talked about at the beginning of BCI illiteracy right that some subjects can never really learn to control these interfaces maybe part of the reason why is that they are really struggling with this part of the problem and this part of the problem is the bottleneck right because we can have all of the smart machine learning we want but if the subject can't produce the neural activity reliably it doesn't matter and so that has led me to sort of really think about more about in detail about this problem and what it what it means and so one of the ways to think about this and kind of how we might optimize it from an engineering perspective is kind of rethinking how we actually select our signals so like I mentioned before we're just putting our sensors somewhere in the brain it's based on the fact that motor cortex controls movements but we're not sort of more principally selecting these signals in any way and given the fact that BMI is not a decoding problem again it sort of opens up the possibility of you know how do we actually select these signals and in addition to where you put your sensors there's also the question of what measurement you actually make so there's a lot of different ways that you can measure neural activity right we in all of the data I've shown you so far putting electrodes into the brain and measuring activity of individual neurons on those same electrodes we actually also get other signals the people often call local field potentials it's sort of low frequency activity that reflects some activity of a lot of neurons in the listening sphere of this electrode we can also do things like electric or topography which are less invasive where we put electrodes on the surface of the brain and we got similar signals where we're kind of summing up the activity of a larger group of neurons and you can you know continue to do this on a variety of different scales you can not put it on the surface of the brain you can put it on the surface of the dura you can put it under the scalp you could put it over the scalp there yet there's there's a you can do this sort of field potential signal you can get from electrodes kind of placed any distance from the brain right so EEG is just a further zoomed out version of this signal and generally speaking in broad terms the difference between something like EEG and electrical or topography is kind of the listening sphere right so you're getting because you're further and further away you're picking up neural signals from a larger and larger exactly yeah so you're losing as you go from something like spiking activity to electric or topography out to EEG you're kind of spatially blurring your signals in addition to loss of amplitude right because the skull is basically an amazing low-pass filter that kind of cuts out all of your signal yeah so if typically so all of these different signals spikes local field potential acog EEG all of those measures have been used for brain machine interfaces successfully but it's kind of still unclear which is the best choice and if we they have different pros and cons so thinking about kind of the two extremes spiking activities is commonly chosen in large part because it's very closely correlated with behavior right we kind of understand in some sense you know what spiking of neurons means and it strongly relates to behavior but we have this issue of kind of poor longevity and in signal instability it's also obviously highly invasive on the other extreme something like electric or topography is typically thought we sort of still don't understand a lot about how it relates to behavior so kind of the encoding relationship between acog and movement is still an open question and it's but it's potentially longer-lasting more stable slightly less invasive and so again from a decoding perspective the choice is obvious if you talk to people to think about brain machine interfaces from a decoding perspective they say obviously spikes are the best and we should go with that or you know pick your signal based on what gives you the most decoding information about movement but you can again flip this question around and think about it from a control perspective and going back to this modulation question which of these signals is actually easier for subjects to learn to modulate and control control and why and I won't kind of go into the details but there's a lot of reasons from kind of a controls angle that it's not obvious whether spiking or electric or topography as an example would be easier for a subject to learn to control and so I'm interested in starting to do some of these experiments where we kind of specifically look at how subjects learn to modulate neural signals and how that depends on the selection of your signals both in terms of measurement modality also where you put your sensors and things like that so I won't those are sort of new experiments that we're going to be starting in my lab but I wanted to touch on the fact that doing these types of experiments actually required developing new technology and so we had to develop a way that we could actually make some of these different types of measurements all within the same part of the brain right so if we want to do a direct head-to-head comparison between spiking and acog signals when you'd have the same part of the brain make within the same subject make all of those measurements together an typical neural interface technology doesn't support that so typically people put in one type of sensor close everything up and that's it so what we did was develop an interface that allows us to that allows for modular and flexible recording with a variety of different measurement signals so that we could combine these different measurement modalities together and so the general idea of our implant is basically you can think of it as kind of an access port to the brain I won't go into the kind of the implant details but it's it gives us chronic subdural access so what we actually do is basically open up a window into the skull and the also the dura which is a covering over the brain so that we can both see the brain and have access so that we can put electrodes in to make measurements and we designed the implant in such a way that it's really minimal if basically the chronically implanted portion of the interface has is very agnostic to the measurement hardware that you're gonna make and so then that allows us to sort of stack things up so you can think of our implant is like implanting the green Lego board base and then you stack up Legos on top of the implant depending on what you want to make what measurements you want to make and so this enables us to do sort of very flexible recordings both electrical and optical since we have optical access to the tissue and we can also do I won't talk about it here but we can also do causal manipulation so we can do stimulation with both optic optogenetics and electrical activity or you can also think about doing silencing where you can inactivate neural activity so this is really exciting for as an enabling technology to start kind of teasing apart the details of neural circuits and then how they relate to kind of this modulation learning problem and brain machine interface so in there's a lot of there's several different optical techniques there one is in this case well one of the things that we've done is intrinsic imaging so it's you can think of it as fMRI but on a much smaller higher resolution scale so you're measuring changes in reflectance of blood vessels that change as oxygenation yeah exactly so you can look at sort of changes in oxygenation in the blood you can also in animal models obviously you can inject gcamp which is a an indicator where now neurons expression of fluorescence changes with calcium and so calcium is kind of the one of the things that changes when a neuron fires an action potential and so that allows you to pick up firing of neurons this is about two centimeters and so yeah so for those not super familiar with the brain this is the central sulcus sort of roughly in the middle of your head it divides kind of the sensory parts of the brain in the back from those sort of motor and more cognitive parts in the front so this is the motor cortex up here and this is the sensory cortex so in a monkey this size scale covers multiple brain areas and a human this obviously the a monkey brain is about ten times smaller than a human brain so this size scale probably is more like one brain area in a human all right so with this capability of this sort of flexible implant system it allowed us to design as an example an implant where we could do simultaneous electric quartic ography so recording from the surface of the brain in addition to penetrating sensors as well and so it's a little hard to see in this picture but this is now that same brain that you saw before but with an overlay of where we're making measurements so these sort of bigger holes are where penetrating electrodes go through and each of those tiny dots is an acog contact where we would be recording from so we have 244 Econ contacts and then 32 penetrating electrodes and so this allows us to now make a variety of different measurements at different spatial scales within the brain so we can look at sorted activity where we're looking at individual neurons we can also look at things like multi unit activities where this is kind of Groot larger groups of neurons these are again just action potentials but it's picking up action potentials from multiple neurons that you're recording on the same electrode we now have local field potentials which is again this measurement from within the cortex but sort of the summed low frequency activity of neurons around the electrode and then ACOG which is that same type of signal looking at groups of activity of groups of neurons but now recorded from the surface itself and so there's obviously kind of very interesting cick neuroscience questions about how these different spatial scales relate to each other but one of the you know I'm particularly interested in given these different spatial resolutions of measurements which one is actually going to be the best signal for a brain machine interface and why again motivated from this perspective of the subjects have to learn to modulate the neural signals and create kind of reliable patterns of neural activity and so it might be different depending on the way that you're measuring the neural activity for both so they're comparable yeah all right so to wrap up I hope I've sort of convinced you that thinking about brain machine interfaces not from a decoding perspective but from the perspective of creating this closed loop control system that the brain has to learn how to use is particularly useful for improving brain machine interface performance and so we can you know again kind of revisit a lot of the existing choices that people have made when we design these systems to try to improve the performance and that includes things like adaptive decoding incorporating Co adaptation where the decoder is adapting along with the subject other things like how you actually design your loop properties and even how you select the signals and this is going to be really critical I think for making robust interfaces not just robust in terms of kind of long term stability within a subject but also across subjects because this allows us to sort of think about how subjects are learning these interfaces and optimize the learning process so that it's robust and then you know importantly I think kind of insights into how subjects are controlling and learning brain-machine interfaces are going to be kind of really critical for creating these design principles that I alluded to before so with that I want to thank you for your attention of course and point out other folks that I've worked with so all of the loop manipulation decoder adaptation and co-optation work is from my PhD with Jose Carmina at UC Berkeley the multi scale neural implants the flexible implants that I talked about at the end is work done with Abidjan pesterin as part of my postdoc and then we have a variety of different funding sources oh yeah I got a women in science award from L'Oreal yeah not yes that's a great question so the short answer I guess is that it's complicated I think that there's my prediction is that there's a sweet spot of if you have too many electrodes at a certain points especially given that machine learning is always somewhat suboptimal you'll never get to the point where irrelevant signals contribute zero and so there will be a point at which adding signals actually adds a penalty cost but there is in general some some benefit up to a point I guess of adding neural signals yeah and it's not necessarily so there's actually quite a bit of basic science research in BMI particularly the things where people are looking at how subjects learn these interfaces where they use really small populations of neurons one or two or you know maybe like up to four or five and it's it's possible and the subjects learn pretty quickly yes so in the clinical trials there's definitely some interesting work where they try to use you know the subjects ability to talk to try to learn a little bit more about what's happening the interesting thing is that I think sort of similar to how if I asked you to explain to me how you learned to play tennis you would probably have a little you could tell me broadly what you were doing but at the nuts-and-bolts details of how you perfected that serve you'd really struggle to tell me and I think there's very similar phenomena that happened in brain machine interfaces where subjects will say vaguely that they started with a certain strategy for instance I thought about wiggling my tongue when I wanted to move up and then I thought about wiggling my foot when I wanted to move left but eventually that sort of fades very quickly and they eventually can't tell you what they're thinking about I think every subject tends to have a slightly different strategy and that actually relates back to you know if you look at like decoder parameters and other things like that you will also see that neural features that contribute to a decoder vary a lot across subjects as well so people pick kind of different neural signals to modulate depending on their own personal preference and part of what's really interesting and you know thinking about the learning the subjects are doing is we like to think that we're optimal in learning but we actually often get kind of stuck in local minima and so it's going to be really one of the interesting things to think about is kind of how do we optimize that learning process to you know help people find the best solution and not get stuck in some based on whatever initial strategy they tried you know that's going to bias where they end up in the space this whole feedback loop that it generalizes someone during in the scene example we had where you move your cursor it looked like the target locations were discrete yeah what would happen if you would rotate 22° yeah so we've done I didn't include it here but in for instance the point process decoder we also did tests where we gave subjects arbitrary positions to move to and even gave them obstacles so that they had to not use a straight line path between the two targets and the cursor control generalizes in that case generalization more broadly I think is a really interesting question so one example is that we're talking about very controlled lab settings where the subjects are only doing this one task right example of you know trying to grab you know this Mouse while I'm talking to you you know is another really interesting question of generalization that's an open one so I personally have not but people in the brain machine interfaces committee have there's a lot of questions about you know could you do this control kind of from any part of the brain and the interesting thing is that you can use something like visual cortex in in this case it was done in rodents so it was a slightly different paradigm the neural activity was controlling an auditory cursor since rats are not very visual animals and so they were listening to tones and controlling the tone of the cursor with their neural activity and so you can do it had been demonstrated that you could do that in motor cortex but you can also do that in visual cortex as well and so in general sort of the flexibility and plasticity of the brain is very allows this to work with a wide variety of neural signals which is actually really exciting because it points to the fact that you know let's say I have someone who's paralyzed because of a stroke they're paralyzed because they're motor cortex and their motor neural circuits are damaged and so it suggests that you could restore motor function you know or whatever function has been lost due to neural damage by repurposing another part of the brain right yeah go back to the separation of like control rates of feedback rate really curious and what kind of investigation is done to like that the discrepancy in having a better result because of the control rate when the feedback rate does not change like because does it still make it more accurate because your control rate is higher like I mean there are some fellow audio notes here so I would compared to like the concept of up sampling for filtering to make it more accurate as it's something like that yeah so we we haven't done I think to really properly answer the question we require some really carefully designed new experiments but based on the data that we have what and we haven't done those based on the data that we have what we think is happening is sort of going back to this idea of kind of feed forward control that's happening where you're kind of launching the cursor based on the existing knowledge that you have independent of other feedback that you're getting what they're improving on and where the improvement in performance comes is actually related to holding at the target so they are better able to kind of stop precisely at a particular point and we think it's sort of you know they're better able to kind of predicts or they're using the fast control rate allows them to kind of more precisely stop the cursor at a particular target individual what happens if you can't because I haven't done that experiment but I think that especially with these various with these highly spatially localized signals the generalization would probably be very poor right because each the heterogeneity of kind of how neurons relate to movement is really high whether that's true and you know sort of more spatially average you know less spatially precise signals you might get more generalization because the you're not trying to map one's object's brain you know in fine resolution to another but but yeah it's a great question I think it's there has been some work trying to kind of apply things like ideas from deep learning for instance into brain machine interfaces to deal with some of this both robustness over time but you could think about using it for subjects as well where you kind of have a large library and based on you know pass decoders you've seen you could predict what the best decoder is for a new subject that you haven't observed before but that hasn't been done it is because you have a very fine position Electro's I think that's a big part of it they will have age in general in in principle yes I think the the question is whether all subjects use the same strategies right and so that's where the generalization is less clear and you would probably need to do some amount of adaptation to customize the decoder to a particular subject yeah I think so but it it's it's an open question but I don't think you know the literature doesn't speak to it yet but that's my assumption how to use the BMI so you have a good adaptation system you put the cap on subject which doesn't know have never ever working do we still will need learning and how long you'd be expected so I do think that each subject is gonna have to learn how to control the interface because it is such a new thing for each particular subject in terms of how long it will take the one of the things that I kind of glossed over is I showed these example learning curves of that were on the order of two weeks right for one example decoder but that's that one learning curve is actually part of a much larger learning curve so in addition to the subject learning a specific decoder they're actually also learning this bigger picture tasks of how to control the interface itself so it's sort of learning how to learn in some ways you can think about it and that timescale I would say is on the order of a month in these particular cases whether that would be the same for you know a human subject especially where you could in principle tell them more about the rules of how things work I don't know yes but I mean I think that the part of the nice thing about the decoder adaptation is that the goal is that we can even during that kind of two-week period we still had an operational interface right that he could use to perform the task and so the goal is to sort of assist and in some cases actually in brain machine interfaces they will use this even like assistive paradigm where you have kind of a you know robotic assistant or you know automated controller that participates along with the subject and you gradually fade that out over time and so that is one strategy to kind of you know make sure that you have an operational interface even while this kind of longer learning process is happening colleagues please more questions would it help if various started trying to do some sort of Maki and a see if the subject moves the hand from one point to another point C Beach neurons fire and then lady literals along those paths or some sample from those electrons those right so that in general no because controlling well wait sorry can you say your question again I want to make sure I'm not sending from point A to point B team yeah so when we so all of the examples of smooth patch for instance that I showed you where the initial performance was terrible the way that we trained that decoder was by having the subject watch a cursor move through trajectories and so you see patterns of neural activity during that context but it doesn't relate necessarily to what the neurons do when they're controlling the brain machine interface yeah there's sort of a very distinct mapping between those two they're kind of very different contexts that said it actually does depend a little bit on I think how the subjects are trained so one really interesting thing is that some people will in these cases of able-bodied animals right some people will have a brain machine interface tasks and they allow the subject to keep moving and what often happens is the monkey keeps moving their arm while they're also controlling the brain machine brain machine interface and then over time the monkey realizes oh it's not actually my arm that's controlling this I don't want to move anymore because why exert that energy and in those cases often people actually see that neural activity is similar between our movement and brain machine interface control but in our case what we were doing is asking the subject from the very beginning to not move their arm and so that potentially leads them to different strategies to control the interface right so going back to you know what they're doing maybe you know the monkey is thinking about wiggling his foot versus you know something else during the task as opposed to thinking about it from the perspective of moving their arm and so you know the strategy that the subject comes in with and how they approach the problem is going to ultimately influence how this works and I think going back to the goal of robustness that's why we kind of need to have a system that allows for all possible strategies right because everyone's going to approach this a little bit differently it's an interesting question I think that it's hard to it's hard to quantify that really carefully there's actually a broader issue in the field that there's not a lot of standard metrics and so there's enough variability between like similar studies that have been done in people and my work that I can't do a head-to-head comparison to quantify it carefully I would say generally speaking the monkeys are probably a little bit better some of that might be due to the fact that they have a lot of motivation where the subjects are and they're not they're not paralyzed but you know I guess subject motivation varies a lot more from person to person depending on a lot of factors whereas the monkeys you know because we're controlling their motivation by incentivizing them to do the tasks and so I think that's an additional factor that's a great question so the BCI illiteracy term actually comes mostly from the non invasive community so it's in part I think because they non invasive BCI's have been done in a much larger patient population so they can actually properly kind of estimate the percentages and that's the context in which it's seen it's possible that you know some of the details of how they're doing the EEG recordings might influence it but within a given study there will typically be some portion you know where they're controlling broadly speaking for how they're making the measurements there still is a patient a population of people that can't use it often that's attributed to the fact that or people often attribute it to the fact that a EG is just a really broad signal and maybe it's just not a useful signal but I think it actually you know goes back to this issue of how the subjects are learning the interfaces and I think we'll you know if we start doing invasive BCI interfaces we'll start to see people that are you know BCI illiterate quote-unquote even with these more spatially resolved signals but yes it probably is related to like how we're picking the signals as far as the degree of freedom freedom that you would have it kind of more natural like would a more 3d version of this maybe just result and better because it's kind of more natural for people and for my lazy um yeah I think that's definitely possible one great study that points to this is that someone kind of went back through a lot of the existing BMI literature and kind of looked at what was working well and what didn't work well in terms of decoders but from a control perspective and what they found is that when you have a cursor that kind of is being controlled in a way that obeys reasonable physics where position is the integral of velocity and you know these sorts of things that performance is better and so that points to kind of this idea of you know having kind of realistic physics and other you know properties of the interface are gonna be really important for performance I haven't thought about that in the context of like 2d versus 3d but I think it makes a lot of sense that sort of the the nature of the interface and kind of how similar it is to control schemes that the brain has seen before will probably influence performance yeah but then books are actually the only 3d creatures human yeah do you think that might be ways to speed up this feedback loop by about not going through a complex task you give like the brain its success right so like there's stimulation you know feedback yeah people people have started doing those experiments where you give feedback not just through natural sensory systems but by you know stimulating neurons in the brain that's it's a really interesting question and I think we don't yet know whether it would necessarily speed up learning but we do know that that type of signal can be used to inform learning and to drive control in these systems so there are examples of brain machine interfaces where neural activity is being used to control a device and then the only feedback the subject gets is neural stimulation and that works so well questions shall we think [Applause]