Machine Learning for BCI: Challenges & Best Practices (Tutorial)

[00:00:00]um so welcome everybody uh this is uh the first inaugural seminar series within the next gen uh initiative of the BCI Society uh my name is David valerani I am a postat masser and I'm also the chair of the student and postal committee within the BCI society and it's my great pleasure today to welcome our first Speaker uh Christian HF uh talking about my learning for BCI um so Dr herf is an assistant professor in the school for mental health and Neuroscience at Mast University and uh he's uh leading the invasive BCI research line uh and his research interest lays in the application of machine learning technology which will be the subject today uh to neurophysiological data for brain computer interfaces and Neuroscience research and before passing the uh uh stage virtual stage to to Christian here um I just want to to say a couple of things about the uh the series so BCI Society is pushing to uh creating and organizing new initiatives virtual initiatives uh throughout the year and this next Generation uh is a a seminar series for both Technical and career development um um uh talks and panels and discussions open to uh to everybody uh for uh trainees and more advanced as well uh BCI researchers uh if you uh are interested in uh presenting or uh in helping out with the BCI Society feel free to uh reach out through our website bcis society.org um and if you are Tweeting or using social media within this uh presentation uh remember to use the hashtag that is at the bottom right BCI Thursdays uh and uh a final note if you have uh so the presentation will last about 30 minutes and then we'll have like questions uh at the end of it feel free to use the uh button at the bottom of your screen for submitting your questions or type them in the chat whatever is easier for you and we'll answer as many questions as possible at the end so with that without further Ado I leave the stage to Christian thank you very much thank you very much for the really kind introduction um I feel very honored to give this inaugural um session in the BCI Thursdays for the next generation and I will be talking about machine learning for BCI today and I must admit it was very hard for me to prepare this uh presentation I have given a lot of lectures about machine learning for BCI but usually that's 12 times 90 minutes over an entire semester so boiling that down to 30 minutes for a group with very diverse backgrounds with very different stages of their career was actually quite a challenge so what I try to do here is introduce the main Concepts give you um some of the lingo we use in machine learning and then just pointing you in the right directions because in 30 minutes I believe there's not much more uh that is possible if after this discussion uh and presentation you still want to talk about something feel free to reach out um either via Twitter or via my email address which you can see down here so I think machine learning is an elementary part of BCI we all know that a BCI measures some sort of brain activity and what brain activity we consider really doesn't make a difference for BC we then do the necessary signal processing and extract meaningful features and um all of these parts will be um spoken about in one of the later lectures now in the decoding stage is where the machine learning magic happens it translates those meaningful feature vectors into an output that can be used in the application for our patient in the end so in this decoding stage is really where machine learning plays its role in BCI in machine learning in general we try to categorize problems into two um different um problem types the first one is classification in classification we predict one class from a set of classes from the input vectors to give you examples we could predict is it um is the person imagining to move the left hand or imagining to move the right hand in a motor imagery task one of the best known BCI paradigms so the two classes the set of classes is left and right and our classifier predicts is the patient thinking of leftand imagery or right hand imagery in an example with more classes from a very recent paper by the brain gate group the patient was imagining to write a letter um and the classifier was then saying which out of all of the letters that could be written the person was trying to write down now in another two class example we have the very popular p300 spell on the right so the p300 spell can spell out 36 different letters as you can see here 6X six but the classification is actually a two class problem it just classifies is the patient focusing on this row or column or are they not focusing on it so just yes or no whenever I do a classification problem I need to know how good we're doing and in classification the easiest evaluation criteria is the accuracy the accur accuracy just tells us how often was I correct out of the total number of decisions I made now if I have a large number of classes like for example in the handwriting decoding from the previous slide just the number just the plain accuracy isn't very meaningful so and this is also true if we have very unbalanced classes for example in the p300 spell we have a lot more misses than we have targets so it is often very helpful in classification to look at the confusion Matrix in the confusion Matrix we see what the classifier predicted here on the x axis versus what the reference what was originally intended here on the y axis this is often done with colors and in an Ideal World we would have high numbers only on the diagonal of course especially in bcis we often make mistakes and we can then see where the classifier makes mistakes for example in this um study uh from my colleagues and me we see that I and J are very often not recognized as themselves and that they classify instead things it's a silence or an R This was um a speech decoding task from brain signals the second category of machine learning problems we faced um in bcis is regression in regression we predict a continuous variable for each input Vector so instead of saying uh left or right for example we can predict how far to the left or how far to the right a robotic arm shall move or shall even move in three-dimensional space as you can see in this study by Adelman on the right another example for a regression task is one of our studies again where we predicted continuous spectrograms of speech from invasive brain data so instead of two or three or a number of classes we predict continuous values in regression tasks again we need to evaluate how well we did we again do this by comparing what was intended to what was decoded you can see this uh in this example from G puts group on the top you see the target cursor positions that needed to be decoded uh in EG here and then the decoding of several decoding algorithms calman filter and uncentered kman filter and so forth we can then compare the two trajectories so the real one and the predicted one for example using a standard correlation here is a piercon correlation or a mean Square squared Arrow between the target between the Target and the prediction by just comparing them sample wise now especially in machine learning I think it's super important to uh in in BCI it's super important to understand that we don't always need to do machine learning we can also build knowledge-based systems and I'm giving you two examples here how we can do Fantastics bcis without needing machine learning at all for example from the literature we know that when we're moving our right hand or imagining to move our right hand we will have a strong decrease in beta power this is is called uh the movement related desynchronization and you can see it here this is frequency on the x- axis in between 30 13 and 30 Hertz we see a a much lower um energy in the frequency then during rest we know also this must happen on the left motor cortex because um the limbs are controlled contralaterally so we could just built a knowledge-based system saying when on this electrode the power is below a certain level It Must Be Imagined left hand movement and if it's above this threshold there is no imagine Movement we do the same for an electrode on the other brain side and we have a motor imagery BCI and this will work very well I mean it might sound super simple but in reality these approaches often perform astonishingly well of course such a knowledge-based system is also possible for regression from the 80s we know that um neurons in the motor cortex exhibit a a so-called cosine tuning so the firing of certain neurons depends on the cosine of the direction from their preferred Direction so if we figure this out for a certain neuron we can just say okay these neurons fires this much so it must mean we are at 100° of the preferred Direction so the participant or in this case monkey wanted to move the Cur the cursor that way so by simply including knowledge about Neuroscience we can build very strong decoders and don't need to rely on machine learning but what is machine learning then the early definition of machine learning I could find is by Arthur Samuel from the late 50s he worked on teaching um computers to play Checkers which you see on the bottom here he didn't succeed um his programs were still worse than human players but Checkers was one of the first games that was solved by AIS a little later and he defined machine learning as the field of study that gives computers the ability to learn without being explicitly programmed now the explicitly programming is exactly what we did in the previous slide so the knowledge based systems are not machine learning but what is it then the AR Arthur Samuel's definition doesn't tell us that and for that I would like to jump to a bit more modern definition of machine learning by Tom Mitchell um from the late 90s 40 years later and he said a well post learning problem is a computer program is set to learn from experience e with respect to some task T and some performance measure P if its performance on T as measured by P improves with experience e now what does that mean the experience e is the training data in our machine learning task so we have to have some sort of data on which we train our algorithm our task T is one of the is your machine learning task in the BCI for example the motor imagery I showed on the first slide or it could be the decoding of the handwritten letters I showed later or maybe the speech decoding from one of my studies the performance measure P then would be for a classification task could be accuracy for example or for a regression task a simple piercing correlation so if the performance gets better with training data as opposed to without training data in one of our task we're talking about machine learning how does that happen in the normal machine Learning Community what's usually done in machine learning is that people work in open loop data so they have some sort of data set that they previously collected or downloaded from the internet the BCI competition 4 for example um is very popular for this and then they simulate to have new unseen data if we have very big data sets we can just take a part uh and use it as test data but usually what we do is we simulate to have unseen data and we do this with cross validation so what we do is we take some part of the data and say now this part is my training data and the little bit remaining is the testing data I train my machine learner on this evaluate the performance p on this and then say okay it's this and this good and it eratively now I repeat this process in The Next Step this is a test data and all the remainder is a training data and I repeat this until each bit of data in the data set was used as testing data exactly once I can then average over all these iterations and say my machine learning algorithm is this and this good in discriminating left and right hand motor imagery from the neural data this is super important um and has advanced the field a lot but this is not what we need to do in brain computer interfacing what we do in brain computer interfacing is we have a training phase in which we tell our participant you see the participant here what they have to do so in BC usually we can't start right right away we need to collect the training data first so in the training phase we tell the participants what to do for example here I tell the participant now imagining moving your left hand now imagine moving your right hand imagine moving your right hand again and in the end move your left hand I now have parallel training data I have the data which we usually call X and I have the corresponding label y so I know this data corresponds to right this to left this to left and this to right and by using this parallel data for which I know what the real label was I can train the machine learning model we will get into the models a bit later after the model is trained we can now get to the actual BCI phase to the testing phase where the participant can control the BCI and for each bit of new data we feed the data into the machine learning algor Alm and the algorithm says what it thinks is in the data so we feed this bit of data in and it says that must be right hand and again right hand right hand right hand and we see this classifier only predicts right hand so it might have not been the best classifier possible what we also see here in this layout is a few of the peculiarities of bcis we have have to collect usually have to collect the training data right before using it so the longer the more training data we collect usually the better the decoding model model gets but that also means the longer the participant or patient has to sit there before they can actually do the testing this also shows that this training phase of our classifier is in between the training and the testing phase phase so the patient again has to wait if my machine learning model now takes 7 and a half days on 25 gpus to train the patient is going to be very demotivated to ever use a BCI again so these are things we need to keep in mind that are different from normal BCI from normal machine learning use in BCI so I've summarized some some of the pecul peculiarities of bcis here for all of you who have worked with neural Data before you know that the signal to noise ratio in all neural recordings is horrible even if you have the best invasive recordings the signal to noise ratio is horrible and much worse than say in an image for image classification or in speech for speech classification at the same time we often have very high dimensionality so for example a standard EEG recording has 32 channels of very fast EEG signals one of the thing things that makes BC really special is the variability the data we use in our recordings is changing over time so when I record EEG data for 10 hours the data is going to look very different at the beginning to the end so I always have to find this balance because the more data I collect the better but also the more data I collect the more different the data is going to be uh at the end compared to to the beginning this variability is even larger between sessions so if I put on an e hat today and put it on again tomorrow the data is going to look very different and it's often very very difficult to use data from one day to the next and what is even more different is the data between different participants in speech recognition it's no problem to train a model on a lot of different speakers in BCI this is extremely um difficult and to illustrate this uh I brought you a picture on the right hand side of two brains I 3D printed one of the brains is mine the other one I printed for one of my students and you can see how vast L different they look well in this one I didn't remove the support structures yet um in this one the i i peeled them off but still in the gyri and Sala you can see how very different these brains are so you can already tell that signals measured on this brain will vary greatly from signals me measured on that brain let alone sensor position skin conductivity thickness of the skull and and so fors because the variability is so large this results in the fact that in BCI research we mostly work with tiny data so um all of the world is about big data and collecting more and more images to train but we usually face problems in the tiny data domain because we have to find this balance between training time um and variability we often have only a very few number of samples for each class and we also have the limitation that our algorithms in training and decoding need to be fast coming back to this slide again if I have an hour of training phase where I collect the necessary data and then my machine learning takes two and a half days um to train the patient data is going to look very different but the patient is also going to be really annoyed by you Sim similarly if the decoder takes 5 minutes to decode every sample of data and I want the decision every second I'm going to have a very large delay very quickly so this is H another uh thing that's special in BCI compared to many other BCI um machine learning domains I would like to now go into some of the models we can use for BCI uh and some of the uh special things that are important to keep in mind in BCI because we have these particularities it's often great to use very simple models as opposed to Super complex models and a method that is been used very regularly um even in current literature is template matching in template matching what I do is I save the training data and for every new sample of data I want to classify or regress I just compare that new sample to my training data um with some sort of metric uh we can use whatever metric you prefer and what is good for your um for your problem and then just compare it to all the training data and pick the one is that is most similar I'm going to show you an example from the Fantastic handwriting decoding paper I have referred to so often already so the feature space is projected into 2D space here so imagine now we have a new sample we want to classify and say it lands here in the feature space What template matching does now it Compares this new sample to all previous sample and then it says okay this one is most similar closest in feature space to the Z so the participant must have drawn a z here we can improve template matching methods for example by averaging over all Z's and O's and D's and so forth to make a more robust template but the method stays the same and as you can see in these three examples I picked from 2018 2090 and 2020 this method is still used in very high-profile Publications so it sounds simple but it's very very powerful in BCI research we can easily extend this method by not only looking at the one nearest neighbor by looking at the K nearest neighbors and taking the majority class of these so now again imagine our new sample we want to classify is here and we look at the five nearest neighbors so then the five nearest neighbor for this points are this one this one this one this one and this one so the classifier will say I've got two times e in the neighborhood and three times Z so this new sample must be a z this usually is a bit more robust than just looking at the one nearest neighbor I know this sounds trivial but given all the limitations we have in BCI these template matching methods are often the most powerful ones we can use if we don't want to use such simple methods we can use linear models which is still um very robust so the idea in linear models for classification is to find a separating hyperplane between the different classes going to illustrate what I mean again imagine we have a two-dimensional feature space because everything else is hard to visualize in BCI of course our feature spaces are usually much larger say we have um beta power for each of 32 electrodes we have a 32 dimensional feature space okay coming back to this example we have the two dimensional feature space and we have the classes green which is right hand imagery uh and red which is left-and imagery and then we F try to find a hyper plane between the two that says everything on the one hand uh is right hand imagery everything on the other hand is leftand imagery so the classification is then just multiplying uh our feature Vector X so you see all the data here x um with all of the M features with a weight weight vector Theta uh and add the threshold value B and then if it's positive it's on the one side if it's negative it's on the other side and if it's if it's zero we're exactly on um the separating hyper plane we then get the Y value for um for the classification and can say okay this was left or right hand imagery how do we Define this hyperplane in the linear discrimin perent analysis for example we find this by calculating the means for both classes and the covariance matrices and it's important to keep in mind and I want to point this out here that the covariance Matrix has the dimensionality of the feature space squared so if I have 30 features from my 32 32 features from a 32 electrodes The covariance Matrix already is 32 times 32 large now imagine how many samples in the training phase I have to collect to estimate such a large Matrix reliably this is really important to keep in mind linear models for classification of course can be used for more than two classes we can do this by training a lot of One Versus one classifiers so say we have three classes left hand right hand and foot we can do left versus foot left versus right and right versus foot and then I can um majority vote which one was chosen the most often an alternative approach is one versus many so I have left foot versus the other two right foot versus the other two um uh right hand with the other two and foot with the other two so um of course there's other ways to find the separating hyperplane but uh I like the linear discriminant analysis is a lot it's a personal favorite and when I start a new BCI problem I often times start with a linear discriminant analysis in regression we can also use a simple linear model so uh again in this example we predict the Y values from the X values so to have a two-dimensional problem but of course we can predict um a multidimensional out output from a multi-dimensional input using these weight beta head and if we optimize this least squares approach uh this linear regression using ordinary least squares we can get this weight Vector very easily so we transpose our entire input data multiply it with itself take the inverse of that multiply it with the transpose of the data again times the labels we know from the training data and this gives us the weight Vector again why do I put this formula here just to see how complex this gets um so again we say we have 32 features so we're multiplying a 32 times number of samples we collected Matrix times itself and then we have to inverse it so we have very little data that we collected calculate a very um unreliable Matrix from that and then we have to inverse it I mean this just screams numerical um instability so you can already see where I'm getting with this BCI because of the tiny data is hard for machine learning the whole world is crazy about deep learning and deep learning has revolutionized the machine learning um this discipline in many many domains before machine learning we've never been uh before deep learning we've never been as good in image classification speech recognition and a lot of other domains a deep neural network works by feeding the input feature Vector through a number of different nonlinear layers these can be convolutional or recurrent layers there's a lot of different ways through a number of layers with a lot of different connections between them and in the end either put out some linear layer for a regression or a soft Max layer for probabilities for each individual class as I said this is fantastic for a lot of machine learning problems for example in image recognition but these networks have millions and I'm not exaggerating here they have millions of free parameters that need to be trained now what we had here so far were examples with say thir 30 or 60 or maybe 100 samples in the training data so as you can imagine training these millions of parameters with so small data often gets very unreliable so large improvements through deep learning in BCI have yet to come and this is often because we can only pull the data between different patients so difficulty the problem is when I Define my models on um the data I can get more and more complex with more and more um free parameters in the models what can happen is overfitting overfitting is when we fit our classifier too closely or even exactly to a particular set of training data there is a danger that it may not generalize afterwards to new data H here is an example um very illustrative example from one of the books I'm going to recommend later where you can see a classification problem between Salmons and seab bath so the two fish based on their width and the lightness of their skin and you can see that with a simple linear hyperplane we would be able to discriminate this very nicely but here they show a perfectly fitted complex hyperplane but which would very likely not be reliable I mean this point clearly would be a cellon but this super complex uh separating plane would are classified as a cbase because it's fitted too closely to these outliers what can we do against this overfitting in such problems we um generally meet something called the curs of dimensionality and that is not a pirate movie what the curse of dimensionality is is that the complexity of our problem grows exponentially with the feature space dimensionality so for each feature I add the dimension uh the complexity actually uh increases um um exponentially to rescue us from this we can use regularization for the linear discriminant analysis as I said my favorite uh classifier we can use shrinkage and in shrinkage we just interpolate the co variance matrices we calculate with the identity Matrix so that means we look less at the covariances and look more at the variances on the diagonal of the covariance and um the reason why I like this so much is that the optimal parameter for this trink cage interpolation was def finded in the paper by Lis and wolf in 2004 and the paper is called Honey I Shrunk the sample covariance Matrix which I think was a stroke of Genius we can also use regularization in regression tasks you saw the regression tasks so what we usually try to minimize is this cost function um so the squared error between the actual y value and the V yv value we get from multiplying x with our weights Vector beta by adding an additional cost term which could be um an L1 Norm of the weights vector or an L2 Norm of the weights Vector we imply certain specifications on the weights Vector which lead um to a lower dimensionality which also makes the estimator more robust when little data is available as I mentioned it's super difficult in BCI problems to use data from different patients or even different sessions for a new patient or a new session but of course you I'm sure you agree that it would be much better if we could use all of the previously recorded data for our new participant ideally even without having to use a training uh a training session and just put on the measurement and start with BCI immediately this in this case we maybe only need to adapt our model slightly to the new user and are ready to go if we would be able to use this then deep learning of course would be much more feasible because then we could just train on all of the patients that have been recorded previously um um take all the time to train the Deep net and be okay um the research into this area is called transfer learning and has a particular importance in BCI I think and uh if this is of importance for your machine learning problems I would like to point you to this great paper uh by more go ventres group as a last point I want to make before we get to this the summary is that I think especially in a technology which is at its early at its infancy like BCI it's incredibly important to understand what the machine learning model does don't use machine learning as a blackbox model I pour my data in and some output comes out and I work with that try to understand what your model does and we can achieve this by interpre interpreting the machine learning models so I think it's important to make sure to understand what the decoding is based on is the decoding physiologically plausible so for example I brought you two visualizations of common spatial patterns for left hand and right hand imagery and you can see imagine immediately this is the activity it relies on for right-and imagery this is the activity it relies on for left-and imagery and this makes total sense the um the area fits the contralateral laterality fits this is exactly how you want it to look now if you had the same problem and the decoding would rely um exclusively on an electrode here then you would very likely be decoding eye movement and we really need to make sure that um to know what our classifier is based on for all interpretation of linear models I refer you to this fantastic P paper by Stefan Hower at Al there is some very early stage investigations into interpretation of deep learning models for brain computer interfaces this study is by Miguel angre and myself um and this study is by Robert Cher Meister from T bals group um who did a very large scale EEG study with deep learning as well if you're more interested in and interpretation of deep learning model I refer you to this paper from nature Communications by lapushin from the buring group which uh very nicely looks at the state of the art of interpretation of deep learning model so I I think I'm okay in time so to conclude machine learning allows us to automatically learn multivariate patterns from data this is not always the best sometimes it's better to just um build a decoder ourself based on the Neuroscience knowledge we have whenever we decide to use machine learning it's important to pick the model depending on the task at hand we need to keep the training data size in mind we need to take the training time in mind and we need to take the decoding time in mind there is no one model that's always the best in machine learning this is called the no free lunch theorem so there's nothing that works best for everything I also want to really give for you on the way that the interpretation of the model ensures biological feasibility and is also our opportunity to generate scientific insights in the field of bcis we don't want to only get an accuracy and a uh and a model that works we also want to know why as scientists so I think by interpret by interpreting the models we can really generate scientific insight and as I'm most often asked uh about literature recommendations I point you to my two favorite books uh in machine learning which are pattern classification by dudah heart and star and a bit more mathematical elements of statistical learning by Hasty at Al um if I wasn't using this background you would see both of them standing right there behind me I'm a huge fan uh and with that I hope um I didn't talk too much um and I'm I I thank you so much for the possibility to present today and I'm really eager to hear your questions thank you very much um I'm not sure how David plan this I'm just going to go through the question and answers one by one now um the first question we received was can you please suggest some online courses that merge between ml uh machine learning and Neuroscience a fantastic question um I think there is a bit of a gap here um there is not one Perfect online course for neuroscience and ml there is really fantastic um Neuroscience signal processing classes a lot of really good papers um but the machine learning BCI online course I I don't know one maybe somebody else can recommend something uh okay the next one is just a thank you to me that's great um are there any applications for non-regression machine learning in BCI um that's a good question um actually in practice the classification um plays a much larger role for bcis in real life than the regression for example um all the wheelchair control you've seen all the spellers that are inuse by patients are based on classification so um classifying whe left or right hand imagery or classifying what letter in the p300 spell the user was focusing on um so yes there's a lot of application for classification in um uh in the um in the machine learning in the BCI literature do you use python or Matlab in making the model this question I mean I use something but the that's not the point um as the choice of the model your your application really dictates what you should use I mean you can use almost anything um you will get good LDA implementations in almost old programming languages so what is important is to pick something that's easy to work with something that's easy to interact with the application you want to control something it's easy to get the data into and maybe something that's easy for you to implement it in I mean most important is that it's correct I personally and um my colleagues we use Python but that's because it's convenient for us um we have a very nice interface to the D data with python um the machine learning libraries for python are very strong but I I believe you can as easily implement it in matlb or B BCI 2000 so that's C++ um can you talk about differences between offline model performance and online BCI control for example does the best offline model always work well online our models inspired biology oh it went somewhere I don't know what happened I I I can read the question again can you talk about differences between offline model performance and online BCI control for example does the best offline model always work well online are models inspired by biology easier to learn this is I mean this is a wonderful question uh I love it and the answer is offline models most often don't work best or online because the problem is what I do with offline models is I can optimize so it might be not um overfitting because I do something wrong but it might be manual overfitting because I try so many different models and then in the end something works but that's also overfitted um so in my experience comparing offline results and online results is next to Impossible It's always important to first show that in offline data it's it's possible so there is enough information in the data to decode this but then translating this to an online BCI control is a huge step and um it gets so much more difficult because I have to set everything in advance I have to really I optimize later on I really have to set the parameters in advance this is very similar to um to what we see in the rest of science with pre-registered studies if I collect the data first and then decide what statistics I'm going to use I'll find something but if I pre-register which methodology I'm going to use um it's much much more difficult to find something so um no the the offline model don't always work best it's very hard to compare them and biology models I mean biology models are robust and robustness is one of the key things we need for online uh control so often times biology inspired models definitely work better online I I really this is a great question um so the next question is uh uh would you like to would like to hear your comments on artifacts removal algorithms on bcis do you use any of them what's the current Trend um I think data cleaning data progam processing and artifact removal algorithm are key to BCI performance it's extremely important we do at least some sort of filtering in all of our bcis also in the online bcis uh at least getting rid of noisy channels um trying to filter for example 60 HZ harmonics 50 HZ harmonics um absolutely crucial I didn't talk about it here because for me this is signal processing and feature extraction um I think there will be a lecture about EG signal processing later in the Next Generation Series so that's why I didn't talk about this but it's crucial and no way around it I would say what's your prediction how the BCI field will progress in the coming years um it's a huge question it's a huge question it's also um for me this question is largely independent of the machine learning Factor so um I mean David and I could talk about this for hours um so I'm going to answer it only for the machine learning part and for the machine learning part I think we will see more advances in the transfer learning that data of different sessions or different particip ANS can be pulled and more complex models can then robustly be used so um that we can make use of the large data sets we have collected in the past uh and that's how much I'm going to say about this and uh would you recommend neurom match Academy for computational neuroscience and neurotech extra sources for BCI and Signal processing um yeah I mean I can always say learn as much as you can neuromag Academy is fantastic it's huge it is a bit more Neuroscience focused though but as I said on one of the early slides the more you know about what's going on the better uh neurotech X is a fantastic Community a lot of discussion going on there they have great initiatives so absolutely yeah I can recommend them and are there any application for non regression machine learning in BCI uh I think I answered this one okay uh on motor major decoding is common special pattern classification algorithm and linear discriminant analysis regression algorithm no linear discriminant analysis is a classification algorithm and common spatial pattern then is a pre-processing method to uh it's a it's a filter basically to maximize the variance between cloud yeah they actually used most of the times in combination with each other CS exactly so you pre-process with common spatial or filterbank common spatial patterns and then you classify the variance the feature of a common spatial pattern is then the variance um into the LDA um to so okay let me go to question do you envision adaptive classifiers that don't require long calibration sessions as a likely direction for future bcis um I don't Envision them because they are already used um especially in the invasive literature they use a lot of adaptive classifiers they um adapt over time that start with some random noise and then adapt more but instead of starting with random noise they could be pre-trained on other patients or from the day before uh and I think this trend will go more into other fields of BCI um so it is already in use I think there's even there's another Next Generation talk about adaptive invasive bcis by Amy Osborne right yes yes so that's that's will be a fantastic lecture she's great um um and uh yes adep is classifiers for the world and does LDA ultimately identify the features electrod channels that correspond to the classes like right or left hand the LDA finds a hyperplane between in the feature space um between the two classes um so that is a backward model so modeling from the feature space the um the variable left or right hand it's not directly clear um so to interpret this you need to do a bit of stuff um as is described in the Stefan Haler paper fantastic read I read it once a week a great paper um so yes what it identifies is a normal Vector of the separating hyperplane between the channels or features that discriminate yes okay are there any time serious analysis uh techniques that we can borrow from Fields like econometric well I wouldn't say we borrow them from econometrics um I would say econometric borrows them from the same sources as us um but there are a lot of techniques that are boast that are used in BO for example the always powerful calman filter or uncentered kman fils are hugely popular in econometrics but are also very very commonly used in bcis so I wouldn't say we borrow it from them but I would say we borrow it from the same sources uh could you perhaps describe your own methodology in choosing models and model complexities do you choose one model and stick to it according to the nature of the data or do you go through a sequence of models and choose one after evaluating its performance you said you always with LDA right as a um this is actually this is a fantastic question because I also make the the the mistake of trying this model and that model um yes if I have a classification problem I always follow the idea to start simple because in my experience if I get a result with a simple model there might be the opportunity to improve upon the simple model but if I don't find any result with a simple model it's very unlikely that I will find something with a more complex model um I also tried to not overtune and in my personal research I try to always stick to something that makes sense from the Neuroscience background so for example in a motor imagery classification problem I could extract all the feature bands from zero to 100 Hertz in two Herz spins and look at all of them first of all the complexity will explode as I described and second secondly I don't expect to see much useful information about 50 hertz in non-invasive EG for example so I can throw that out I'm also not expecting to see anything useful below three Herz for aote imagery tasks so I can also just focus on the 13 through 30 Herz we know works from the literature and this way I much less prone to overfitting so a healthy combination of Neuroscience background and playing around but again as I said um to Jen Collin's fantastic question comparing online and offline data is really difficult so then making the choice which one you want to use um in an online BCI is always a very difficult one and um so especially in such a diverse field where industry and Academia have to work together what's your point of view on how maybe more people from industry data science machine learning should also work in labs and Academia I think it would be fantastic to have more people work in BCI but currently there is not a lot of money in BCI right I mean there is no applications we can sell so why would a data science company have somebody work in BCI um so what I think what we need to do is we need to show that BCI has its application that it is useful for the patients um and that we can really profit from this research otherwise it's a nice little exercise for tiny tiny data machine learning but will get Dundon at some point so what I think we need to do to get more people into it is proving that it's an important field um I mean there is a lot of research doing exactly that going on um but I think this is what needs to be done to involve more people but let me add to that that there have been like a number of you know companies and applications that have been shown in the past couple of years is very recent extension uh with all major actors also investing in VC and you know the the market of wearables like increasing a lot in the past few years so uh I think there is a great potential for industry to pick up uh BCI applications and make up new applications that we are maybe not uh looking into in research um so how about feature selection and dimensionality reduction do you simply use regularization or do you recommend using feature selection method such as filter or wrapper feature selection is super important I use it most of the time but again feature selection for me is more pre-processing or processing of the data and um and also feature selection is very dependent on the measurement modality you're going to deal with if you have a UTA array in the hand knob of the Moto cortex you don't do need to do much feature selection everything is going to be relevant if you have an EEG cap and look at a p300s um you will need only very few features so yes feature selection is absolutely crucial and I use it whenever necessary yes and what is your opinion on the current state of data sharing inside the scientific Community um I think it has improved a lot um when I started my PhD um in January 2012 there were almost no papers that made their code uh the code and the data available uh there were a few open data sets like the everpresent BCI competition um but since then this has changed a lot and some of the major journals now request you to um make the data available I try to make the data available as much as possible our most recent preprint has all of the speech brain data available as as well I think it has improved a lot but there's still papers saying data is available upon reasonable request and then you write to them hey um this is a reasonable request can I have your data and you never hear back from them um so I think there is still a bit to go uh but it has improved a lot and you you talk about deep learning applications and one question here is is what reinforcement learning is a good tool for developing online bcis uh using the feedback from users as a reward uh in in that to improve the model most probably developed as an offline one firstly so it's about yeah using reinforcement learning in b um I think reinforcement learning can be used in BCI I personally haven't read a lot of paper where it's used but I think it has great potential yeah and I'm sorry I didn't speak about reinforcement learning which is an important part uh of machine learning but it hasn't doesn't have a huge place in BCI so far uh is there any BCI toolbox do you recommend to run our pipelines online as a real experiment not only offline only with eg Cog or intracortical data set I think it would be helpful as a student yes uh there is a lot of great middleware or back ends to run these experiments personally we use the lab streaming layer which also has a nice series of lectures with it uh which shows you how to handle the data um how to decode the data and it's also it works in a lot of different programming environments so you can build uh code in C++ in Python in madlab there's a Unity interface there's um something to build Android apps with it um so this is our back end lab streaming layer and that can also be used with um cheap EG headsets U like the open BCI headset but can also work with state-of-the-art very expensive um EG headsets like from gtag or all of the others do you see that error related potentials can be used as a valuable feedback for Improvement of the classification ah yes definitely error related potentials are great and and there is some studies out there um changing the classifier output if they detect an Erp an error related potential um so yeah fantastic it's really cool uh do you think it is the noise the main problem to the code 3D trajectories from EG and what strategies do you think are need to work in the future like State space modeling or uh yes I think we need more State space models I'm a I'm a great fan of State space models I mean trajectory decoding from EG is very difficult because um it's a very intricate process and the EG does just does give does just doesn't give us the necessary resolution so um I I think we will see improvements in sensor technology in the future um I don't think this is a purely an algorithm problem that if we have better algorithms we certainly can decode perfect trajectories from EEG I think this is also just not in the data yeah uh what do you think about the growing interest in ranian Geometry classifiers for an invasive bcis um fantastic it's really really interesting work um it has a few very clever approaches so great and so in terms of computational time will it be wise to compromise accuracy because you know like in I mean I think a take on this is that in bcis you don't only want accuracy but you also want the system especially an online system to respond in a timely manner it's not that the output is produced like two hours later um I think this is exactly the type of balance we as BCI researchers have to find we're not only optimizing on the data in terms of accuracy there is machine learning scientists who are better at that than we are but what we need to find is exactly this balance what do we need for for our application how long can the patient wait uh how long is the perfect window to make this decision so um there's a lot more to BCA to BCI than just accuracy and often yes our application requires us to go down in accuracy for the sake of speed for example yes why are the brain signals at different time not the same even if we are in the same mood or emotion um well if I only knew exactly the the brain is such a complex um system that we just don't understand everything that is going on but we know that the slightest changes um uh already change the measured activity a lot I mean the more invasive I go the more stable the signals are over time so for example in the brain gate um trials with those implanted uter arrays the signals are relatively stable between say today and tomorrow uh in EG that is much less so but this is not only a question of the brain this also depends on how much you've sweated how clean your hair is um the general tension of the muscles in your body um so there's just so many factors influencing it um it's really hard to say because if we knew we could just uh backward project it into the same space and it would be solved but sadly we don't um so in terms of cross validation um so we use cross validation to estimate the performance on an unseen set of data however kfold cross validation still might be inflated and there might be a risk of overfitting better approach is the nested cross validation which sets aside a part of the data completely until the end and then test the performance on that uh do you recommend that approach to have a non-inflated estimation of performance because as you said you already don't have many samples and sting a side a part of data completely might not be the right thing yes um I mean because of time limitations I simplified this part a lot a nested cross validation is really what you want to do so in each fold um you you have the training data make another cross validation to choose which model you would use and then put use that on the real test data for this fold but it gets more and more complex and at some stage we need to ask ourself how useful is this for the application I have in mind in the end so what happen often happens in my studies and I'm being brutally honest here now um is that we design a super intricate super complicated algorithm on the offline data and it works really nicely with nested CR cross validations for parameter optimization and so forth and then when we make the decision to go um online and try it with a patient we use an LDA again um um so um I urge you to keep that in mind what we do this for um because we really need it to be robust um and fast in the end but yes for offline data analysis the nested cross validation is the right approach so final question uh so about the accuracy what is the best value for BCI applications for example accuracy greater than 70 80% this question is actually a lot harder to answer than you would think because first of all I mentioned that um accuracy can get a very tricky number to interpret if you have many classes or unbalanced classes uh there is a paper an older paper by um Andrea kuas group who say above 70% the user feels comfortable and doesn't feel like it's wrong all the time for me this is oversimplification I think it totally depends on what you classifying and totally depends on the use case for example if I can classify between left and rightand motor imagery I think we would be very frustrated for two classes if we above 70% we would just be super annoyed if it's um uh if it's wrong in almost a third of the cases however for the Fantastic handwriting BCI if that is correct with 30 classes 70% of the time so um in 30% of the classes we have to then enter backspace again and try again I think everybody would be super happy so it really depends on your classification problem how many classes how easy is it um to redo this step how long does every trial take um I think we really have to figure that out um for our BCI independently and and also I guess what what are the consequences of a misclassification yes yeah very good point I okay so I think uh it's time to wrap up this uh so thank thank you very much for answering so many questions for the great presentation as well and for kicking of this next Generation seminar series from the BCI Society um thank you all so to all attendees uh that we had today for the great questions and for your attendance uh keep uh an eye on the BCI Society website for the next uh seminar series and the other events uh feel free to to join there as well and in the meantime thank you again uh Christian herf for a great presentation for being with us today and thank you for having me um it was a great opportunity uh and I really appreciate all the effort uh you put into to bring the community forward thank you thank you and have a good day everyone bye bye bye