Understanding Deep Learning Models: Adversarial Examples & In-Context Learning

[00:00:00]foreign hi everyone welcome to mlfl this week and this week we're really honored to have Dr Dimitrius cipras from Stanford University as our speaker Dr Dimitris Cypress is a postdoctor postdoctoral scholar at Stanford University advised by pristine Leung and Greg Valiant he obtained his PhD from MIT advised by Alexander Madrid his research is aimed towards understanding and improving the modern machine learning toolkit focusing on topics such as reliability benchmarks and interpretability now without further Ado let's welcome Dr cipras foreign thank you for the introduction um yeah so today I wanted to talk about um a few Representatives Works in this general direction of how can we get a better sense of how our machine learning about its work by trying to better understand their successes and failures in different scenarios and of course I'm sure you're all seeing the news that deep learning is being really transformative these days and every time I make this slide over the last five years the argument becomes easier and easier starting from things like image classification which really showed the potential to things like having deep learning models layer strategic games scientific discovery with protein folding learning to generate language learning to generate code or these really impressive text to image generation things are really going and all of these developments are really exciting there's actual genuine progress happening here there are many useful tasks that we were not able to do before and are able to do with today's massive learning technology and interestingly the recipe behind all of these advancements is quite simple and common basically says get all the data you can get your hands on by just aggressively scraping the internet be it archive Flickr Reddit Wikipedia anything you can get your hands on and then train a really big model the biggest model you can you can using that data and then potentially apply this model Downstream and within simplicity this is actually a very powerful recipe like um we have models now that are able to do many different tasks without having to humans to supervise them in a very fine grained level um and in general the whole idea is that instead of trying to teach your model how to do something specific you're just teaching it how to model the data that it sees in the wild and good things happen at the same time things don't always go the way you would expect like things are the outcome of this process is not always predictable um and this can have both good and bad manifestations so the bad is that models often fail in ways that you didn't really expect them for instance if you try to optimize this half data to run as fast as it can it gets into this weird Optimum where it will flip upside down and start uh jumping around in order to run you have things like self-driving cars that for some random reason tend to go towards barriers so even when these systems work most of the times there are edge cases where they fail in ways that you wouldn't expect them or this recent example from the anthropic paper where a chatting board is very happy to explain to you that one of the good ways to get into selling cocaine is to raid an existing drug dealer at the same time it's not all wide sorry and yeah and perhaps one of them most prototypical examples here is this phenomenon of adversarial examples where models that are normally highly accurate can be fooled by a tiny amount of noise these are called adversarial examples and starting from a classified correctly classifies this pig adding a tiny amount of adversarial noise causes this classifier to classify the image as an airplane all bad like sometimes trading these large models allows them to succeed in ways that we didn't expect it before for instance there's this phenomenon of in context learning where large language models trained to model internet texts are able to learn by just a few input output examples that we provide them for instance if I give the model a few examples of English to German word translations the model is figuring out that I'm actually trying to do English to German translation and will happily provide new translations or similarly with addition and this was one of The Primitives that allows the models to succeed in many tasks without fine healing or a phenomena like this chain of a reasoning that says that oh if I prop my model in order to encourage it to show its internal thought process in some sense then it can get better certain reasoning tasks or if you want you can even skip this and if you just prepend let's think step by step in the answer of a model then it turns out that it becomes much better at reasoning about this sort of simple yeah first grade arithmetics so I'll go to the data but the stock is to dig a little bit deeper into some of these surprises uh the good and the bad and and the Hope here is that by looking at how models behave in these unexpected ways we can complicate a better sense of the way they work internally because if all you're looking is what the accuracy of the model is on the task it's expected to perform uh then you're not really going to see many things so the model is either going to be correct or it's going to be wrong but by probing Its Behavior on on behaviors that you find surprising we can actually get a better sense of what's going on and basically gonna have two parts so first we're going to talk about adversarial examples and then we're going to talk about in context learning okay so let's get started um yeah so the first question when you see um yeah adversary attemption just prefix this is Joint work that I did with uh mostly my collaborators back at MIT where most of them were grad students at the time and what is really intriguing about adversarial examples is that they don't really make sense at first glance you see this example you say that oh I have a model that correctly classifies a dog as a dog but the moment I add this imperceptible noise the image doesn't really change but suddenly the model is completely confused so what is confusing is okay why is this specific perturbation important clearly we as humans know what dog is we can see that this worst case perturbation has nothing to do with the dog but somehow the model is very sensitive to this perturbation and I want to take a moment and post and think a little bit that like this is actually a very human phenomenon our surprise uh to the effect that the model is confused by this specific perturbation is actually deeply rooted in the fact that we as humans do not view this perturbation as meaningfully so can we did this thought a little bit deeper so one natural hypothesis here for someone seeing adversarial examples for the first time is to assume that okay the model is really accurate and it has some adversarial examples so in some sense the model is making its correct and accurate predictions according to mostly correct features it's paying attention to roughly the right things but there are these weird vulnerabilities and these weird brickliness is that well if you try hard enough you can make the model uh misclassify things so to put this in a cartoon drawing you can think of linear classification when you have some green features and some blue features and the green features are the useful features these are your features that are useful for classifying these images and then you have some blue features that are useless and they don't really convey cinematic information but somehow the model is sensitive to them and these are the features that the adversary manipulates so this cartoon drawing is meant to illustrate this fairly natural Viewpoint that says adversarial examples such as buns the models are working fine but there are some weird edge cases that can cause them to fail but can we actually test this hypothesis can we get a sense of whether this is the case or not and for this we can design a simple experiment that goes as follows starting from a data set of cats and dogs we have a very simply classification task given animals can you tell if it's a cat or a dog we have a data set comprising of a bunch of dogs and a bunch of cards and now we're going to transform this data set by creating an adversarial example of each image towards the other class so we're gonna perturb the image of the dog to make the classifier predict cut and we're gonna perturb all images of God to make the classifier predict dog and all of this is for a fixed classifier you could train your classifier first and then you create these adversarial examples now for the next step we're going to actually relay beat these images based on the label predicted by the classifier so now we're labeling this perturbed log as a card even though to our eyes this is mislabel this clearly looks like a dog sale and then we're going to train a new classifier from scratch on this mislabel data set finally we're going to test this classifier on the original test set yeah so just to quickly reiterate starting from a normal data set of cats versus dogs given a fixed machine learning classifier we're going to perturb these images towards the other class so now every dog is predicting the scouting every God's breaking the dog and then we're going to label these images according to the predicted label train memory from scratch and test it on the original test set so how well do we expect this model to do well at first glance um the training set is completely mislabeled with respect to the test set we're training a model on things that look to us like a dog but are labeled like a cat and then on the test set we're going to test the model on dogs that look like dogs and are labeled as dogs but well it turns out that the model is actually able to do pretty well for instance on the c410 task of cats versus dogs it's able to get 78 accuracy and similar results for imagenet or so or the full c410 so this gives us a bit of a pose like what is happening here how is the model able to generalize when be by being trained on this completely mislabeled data set but somehow there's enough information to generalize the original test so clearly our model our mental model for adversarial examples is missing something right like if adversarial examples were just bugs and they were just perturbing meaningless features then there's no way that they would carry this semantic information that allows us to train a new classifier to generalize with assets so instead one way to think about it that actually makes sense and to reconcile these behaviors is what we can call the robust teachers modeled so instead of thinking of features as useful and useless you can think of useful features as below in the two groups that being robust this could be things that are actually hard to change by small L2 perturbations so this could be things like what we perceive as a cartwheel or a face or the background of a dog and then there are no robust features that could correspond to faint patterns or small idiosyncrasies that the model is being attached and the key idea here is that both of these families of features are actually useful so both of them carry uh semantic meaning towards making a correct prediction it's just that one of these families of features is robust and the other one is not and from this perspective actually the experiment we just presented makes sense um all the ideas of the model is actually coming from round robust teachers because we're starting from some data set where our dog images have normal dog robust features and dogs not robust features and then we're adding small perturbations to turn these images into something of a different class so we're effectively flipping the neurobas teachers so now we're creating images where the robust teachers stay roughly the same the non-robus teachers change the words features of the other class and this is exactly what the model is able to to leverage because these images will have non-robus features that point to the other class these images will be labeled with the other class so the model is able to use these robust features to generalize the test set and sort of by the design of this experiment all the model accuracy is coming from these non-to-pass teachers so to summarize what does this imply well this experiment shows that non-robus teachers can actually be predictive you can take a data set and you can make very small changes to this image and produce features that generalize the test set so this clearly tells you that there is signal in non-robus teachers that allows you to generalize to the test set and since we train our models to maximize accuracy they will utilize any feature that is useful There's no distinction between robust and control robust so if you put one one together you can see that well at Brazil the vulnerability of a multiple personal examples can actually to some extent be explained by the presence of non-robus features in the data there are some faint patterns the data that are not really Salient to us but if we train our model the model will take advantage of these patterns be more accurate and size be pretty there are two other interesting um examples and what is this really interesting phenomenon of transferability so what's robust teacher's model says is that if you create adversarial examples for one modeling and you apply them to another model these are likely to transfer so different models are likely to be confused in the same way and the and this is actually something that is doing practice it was perhaps one of the reasons why uh adversarial examples seem quite fundamental like they're not something that just applies to one model but it can transfer to other models training the same data set and it's actually something that's explained by this robust leaders model if adversarial examples are coming from the models being sensitive or non-robus features well this is the same data set has the same non-robus features if you train two models on it they are likely to rely on these features in a similar way so this means that if you create adversarial examples by perturbing these the features that one model relies on this will also with the other model because the other model is also relying on the same features and secondly there seems to be this trade-off between robust and accuracy so in some sense uh if you want robustness then this would mean that your model has to only rely on the robust teachers but if you've got accuracy then take me into account the not robust teachers is actually quite useful and it turns out that they saw something manifesting practice and it turns out that more Visa tend to be more adversarly robust are also slight are also typically less accurate and there are some nuances of these that I'd be happy to talk about but the first order approximation is that these things can be authored for a fixed data set and perhaps this seems um particularly like talking about um number bus features but it's not really something that's it's very strange in fact there's a lot of work by now that shows that models can actually rely on features that we don't really think make a lot of sense for instance it could rely on linear Direction pay a lot of attention to texture over shape rely on high frequency components or dense feature mixtures the key takeaway is that all of these different features even if they are not entirely intuitive they are actually valid from an accurate perspective if you're training your model to maximize accuracy even if there's some weird feature that vs human could not pay attention to the model has every incentive to learn it and improve its test accuracy because to discuss these features do generalize okay so one natural question is okay what would happen if we tried to make our models robust to adversarial examples is something like that possibly anyway the first thing to to say is that okay of course we need to change the way we train our models in some sense we never asked for our models to be robust so it should be no surprise that they actually don't so the way we train our models in the standard way standard generalization says that I want to train my model to minimize the expected loss over random XY pairs from my distribution so here Theta corresponds to the model parameters and you sample random X Flight pairs from a distribution and you want the Lost Below on average but now if you care about robust machine learning and specific adversarly robust generalization then you need to modify this objective and you need to say that I don't just care about having a low loss on an individual point but I also want to have a low loss on all the small perturbations around that point so if I also introduce a perturbation Delta to X and I choose the worst case perturbation by maximizing the loss over some small neighborhood then here I'm directly encoding robustness into this training objective so in other words I'm being explicit about the set of environments is that I want my model to satisfy and if we train if we frame the problem like this well then optimizing this min max loss corresponds to an instance of robust optimization and in order to solve it in practice you can think of the mean and the marks as corresponding at two different uh things intuitively the inner marks corresponds to finding a worst case perturbation so you can think of all the standard methods for finding adversarial examples that's examples that maximize the loss of the model exactly as trying to solve this inner maximum and you can do this for instance with just projected gradient descent just move local in in a direction that maximizes the loss without stepping out of the capital Delta and then the auto minimization problem corresponds to learning a robust model it corresponds to saying I want to find parameter Theta so that the loss is low no matter the Delta that the adversary chooses and in order to do this well with trainer models uh normally using stochastic gradient descent and here it's a little bit tricky because you're trying to optimize the maximum instead of optimizing an actual differentiable function but it turns out and I'm happy to discuss more Offline that all you need to do here to actually reduce the value of that Max is train on worst case inputs so instead of just picking a random X Y pair and training on it you take this X Y pair you compute the worst case attack and then you train to minimize the loss of attack and this is also known as adversary training from good fellow adult and in terms of it actually leads to robust models and the one caveat that you really need to be careful about is making sure you have reliable attacks so this means that the worst case inputs that you're training your model on have to be a sort of worst case in some sense and a lot of the failures of early attempts to to build robust models came exactly from these um from this culture where oh you're training your model to be robust against some attacks that you can find but if you're attacked consistently misses adversarial examples specific and personal examples then your model doesn't become robust in general and overall once you take care of this then it turns out that you can actually get models that are reasonably robust for instance only imagine it for a bunch of different perturbations L Infinity L2 translations and rotations you can actually get meaningful robust accuracies and what robust accuracy means is what fraction of the test set can the model classify correctly even under the worst case perturbation so even if I give my model and a desk set to an adversary and I tell the adversary make these examples as hard as you can then the module still predicts correctly again within the specific and durable Infinity Norms that we have and no good I'm going to like okay if adversarial examples come from the presence of non-numbass teachers are robust mode is somehow relying on different features of the data and this is a bit of a longer discussion that I cannot get into too much depth but I want to show this like nice visualization that says here we're going to take a standard model and transform this primate into bird so I'm creating a large adversarial example where I'm trying to make the model really confident that this is a bird and if you miss that you probably missed the transformation that happened nothing really changes too much the texture of the primate is changing a little bit but if you do the same thing for a robust model then you actually see it changes much more salient and here we're not doing anything weird all we're doing here is just taking the original picture and doing gradient descent to maximize the probability of predicting bird and we see it to some extent we do the model is indeed sensitive to actually quite different features because this local search for adversarial examples led us to something that looks to some extent like a bird and yeah and like one interesting thing like I like this visualization on one hand because it does show us that robust models are sensitive to different things but those who makes robust models this really nice primitive for doing a bunch of image synthesis tasks so if you have this primitive that says maximize the probability of another class and then features of that class pop up you can do things like imagination translating horses to zebras super resolution and things like impacting um yeah so to summarize one of the key takeaways from this first part are that adversarial examples can actually be attributed to some extent to the presence of non-verbus teachers in the data our data has predictive patterns that our models learn but these patterns are not robust and thus full memory during deployment and it turns out that if you actually train on a worst case perturbations you can actually make your model robust to to these worst case perturbations and these robust models tend to actually rely on different sets of features okay so moving forward into part two and this is Joint work that at Stanford with Shivam and Percy and Greg um we're going to talk about in context learning so as I briefly mentioned before this is this phenomena that we see in large language models where we train these models on a lot of texts from the internet and these models are actually able to produce quite coherent text you can prompt them in some way and they're actually able to produce text but one of the happy accidents is this notion of in context learning where if I start for instance if I want to make a model translate from English to French to German I can just provide a set of examples and the model is able to figure it out afterwards and you can actually do more interesting things you can say oh I want to classify movie reviews into red or green and the model sort of figures out that red is negative green is positive or things like oh I want to do addition but instead of having the plus sign I'm going to use the pound sign and again the model is actually able to figure out that from a few examples that this is what you actually are trying to do and autocomplete in the right way and this is really interesting because all of this is happening purely at inference time and this is one of these Primitives that allowed models to really perform well on tasks that they were not trained explicitly on so these models are just trained on the web you just train them to predict and model the text that is found on the internet and somehow we're able to apply them directly on tasks like sentiment analysis summarization question answering without fine-tuning them just because we're able to specify through examples the the input output form that we care about from these tasks foreign new tasks like are these tasks that we're seeing actually new tasks that they're learning from these in context examples or are they just tasks that they've seen during training and if we focus on the English to German translation case well the model has obviously seen English and German translations so in some sense the task of translation is not something that you learned from this data but the model did have an idea of what book corresponds to in German and even the notion of like translations between languages happening in the list is also something that you find on the internet so to some extent maybe one can ask okay the module is not really learning something new here but maybe just retrieving the the task that is so during training and predicting according to that task but the problem like when you train on such a large score price that you have no idea how to reason about the training data how can I even go and say okay what are the examples that influence this behavior and try to see if this task was when my training data or not so in order to circumvent this I wanted to focus on a bit more or of a clean setting where we ask in a very uh abstract sense what can Transformers learn in context and specifically we're going to study simple function classes uh function class that we understand well in a synthetic setting so that we don't have to deal with all these questions about oh what's this in the training data or was it not okay that's specifically the Primitive that we're going to care about is the question can a model in context learn a function so what this means is you start with some function let's say from RD to R and then you have a model and if you sample a bunch of random points X from the domain of this function x i through XK X1 for x k let's say from a standard univariate gaussian then you compute the F the value of f on these points you fill this into the model and then you sample a new query point then you fill it into the model and we're asking can the model predictor why that approximates f of x query so just to reiterate because this is at the center of our setting what this says is that if a model can in context learn a function then it means that given input out to examples from that function X comma y pairs where X is X and f of x is y then the model is able to approximate this function on a new input now we can take this one step further and say what does it mean for a model to be able to sorry and yeah and we're going to talk about refer to this set of examples and queries as the prompt and now we can take this one step further and ask okay can the modeling context learn a function class so what this says is that okay if I have a functional Class A Class of functions let's say linear functions then I sample one function from there at random then I sample the x's and then I asked my model to predict the value of that function on a new X if the model can do that then we say that the model can learn the function class F in context and notice this is just a probability a property of the model so I can actually test this quite easily given a fixed model I have my function class f i can sample a bunch of random functions sample input output examples for these functions and ask can my model approx make a random function from that class and everything here is actually happening at for a fixed model so this is in some sense inference time there's no training going on here for now now how would I go ahead and drain such a model where we're going to use Transformers and the way we're going to train them is by just directly training them to learn this function class so this has I'm gonna sample a random function f from this class I'm going to sample a random prompt from this class and then I'm gonna teach memory how to predict the right Y at every single point of the sequence and based on the architecture of the Transformer what you get up what you end up getting in the end is that why one will only depend on X1 whereas Y2 and so on will depend on all the previous input output Pairs and the corresponding X so for instance y k depends on X1 F of X1 X2 F of X2 up until x k but of course not on f of x k so this is exactly how we're going to train the body and we're going to do sampling random math train them all to predict better on this Earth and then rinse and repeat by sampling a new function f and teaching a model again to approximate to approximately in context learn this F based on a set of random problems and just to clarify here there's no retraining or training on text this is the whole process we're doing this from scratch we're randomly initialize the model and we train it to be able to in context learn functions from this family okay now you may ask come with drain the model that can learn linear functions in context this is perhaps the simplest family of functions you can come up with and here we're going to work with 20 Dimensions so these are 20 dimensional linear functions which depending on your point of view can be either low dimensional or high dimensional like in general I would say that these are high dimension functions like they have a lot of the prop like the probability of landing on any specific organ for instance is really low um and in general things like nearest neighbors do not tend to work that well so here we're going to train the model as we already talked about and we're going to look at lots of this form so on the x-axis we're going to be looking at uh the number of in context examples that we provide to the model so these are how many x f of x pairs do provide the model before the X query and on the y-axis we're going to look at the squared error of the prediction normalized so that the trivial error is one so if my model just predicted zero on average it would get an error of one so what does an ideal algorithm look for the setting well this is just something like list squares and the error of least squares drops roughly linearly okay in theory it drops exactly linearly as you get more and more in context examples and the moment you get 20 examples if the number of example matches their Dimension then the model just gets list squares gets zero error and their Erza stays at zero so this is ideally what we would hope to get so this is the best thing you could have hoped for is the optimal estimator that estimates F given the in context examples it turns out the Transformer is actually doing pretty good compared to this Baseline so the drop is actually also linear to some extent the Kink at D equals 20 is not exactly as sharp if the model doesn't exactly hit zero error uh but it does improve over time and when you reach um 30 or 14 context examples then the error is actually quite small so this was actually quite surprising to us because it shows that we are able to train a model in a relatively natural way and sorry and just to contextualize these these are some simple baselines things like oh doing three nearest neighbors or just averaging out the X Y products these are simple baselines that maybe a simple mode would learn but they don't do merely as good as they're transform it as and overall what this shows us is that the model is actually able to learn new functions right so we have a function class and the model hasn't really seen has seen some functions from this class but has not seen everything but they still able to learn these functions from in context examples and just emphasize this is not something that you can explain with memorization so even if you say out of the linear functions that my model saw during training even if I use the best one of these uh the the error would not be that low it would be something like 0.0 uh quite far away from what we're getting so in terms of this is showing us that the model is actually able to learn this function is purely from these in context examples and do to some extent correctly in a regression a bit deeper uh maybe following a better theme of adversarial examples before you can ask what is the model learning what is the function that the model is learning and if you view the model as taking a prompt of inputs x i through XK and then one query input X query well you can think of the prompt as being fixed and the whole thing being a black box where you fit in X query and you get y and you want this to be roughly f of x query and we're comparing how well this does compared to the real function we're trying to approximate that takes in X query and then produces W transpose X query for some linear weights w now one way to see how close we are is to look at the gradient of the function the gradient of the output with respect to the input and for the true function well this will just be w and we can ask okay if I look at the gradient of the model of Y with respect to X query will this also be roughly w ow so we can do this experiment and we can look at a similar plot where on the x-axis if we look at the number of a context example that we give the model and then on the y-axis we're going to look at the average product let's say at first between the gradient and w and you see that actually after you get 20 in context examples which is actually what you need to be able to to fully specify to be in the fully specified regime then you can see that basically the gradient and the weight that there is actually aligning almost perfectly well when you have fewer examples you cannot actually recover uh the true way W but it turns out that if you project W to the set of income tax examples that you already got which in some sense is the best you could do then you see that this also matches really well so this was really interesting because it shows that okay not only are these models working on an average input but they're also locally correct so this says that even in a small neighborhood around this input uh the behavior is actually what you expect it to be it actually follows the function properly now the next natural question is okay we drain on this distribution of inputs does the model still work when we sort of deviate from it in small ways and what I mean by this specifically is we're going to look at some out of distribution prompts and the way you we can think of distribution here is that we have our model and there are three distributions of play so there's a distribution of x's this is from where the space from where we're sample the X there's a distribution of functions even within this function class the function the functions that we sample could have different weights and then there's the distribution of the query where are we querying our model on and so far these have been exactly the same during training and testing we fixed the distribution of x f and X query was coming from X and we did this exact same evaluation during training and testing so we did see that the more generalizes well we didn't see if it also extrapolates to some sort of different distributions here so let's look at a couple of examples of how we can mix things up one very natural case is you can ask okay what is my inputs X are not actually coming from a universe gaussian but are coming from a gaussian with execute covariance so let's say the eye sagging value something like I over I squared 1 over I squared so in this case the distribution of inputs is not what I saw during training does my model still work well it turns out that it sort of does to some extent like the performance actually drops quite a bit uh so you can see the error doesn't actually reach the great error that at least squares could get at towards the end but still up until 10 in context examples it actually works quite well and then but even then it doesn't completely go haywire and it sort of plateaus at some reasonable approximation now a sort of complementary question is okay what if instead of messing with the x's and mess up with the Y's so let's say that the functions that I now look at are actually noisy functions so I add some noise and instead of just observing W transpose X I add some noise to the label and this is actually quite interesting because we see this phenomenon um or is it known as double descent and it's actually something that is quite well understood in the at least squares setting and I guess recently that says that as I give more and more exam if I if I look at the no optimal noiseless estimator which is just list squares and I give it noise inputs as I give it more and more inputs where the number of inputs that I give it approaches the number of dimensions then I get this big spike in performance and then things go fine again see the Transformer also follows these squares and has a similar qualitative uh Spike so it also exhibits this notion of double descend which gives us more confidence that okay it's actually doing something relatively close to linear regression okay and now another test you can do is you can say okay what if is my modeling doing some variant of nearest neighbors so what if instead of looking at the random distribution of inputs I make sure that all the in context examples are actually concentrated in one quadrant one orphaned and the query point is actually in another one and in terms of that here the Transformer is actually doing really well this doesn't seem to affect it almost at all and is actually able to generalize in the settings and so this shows you that you're clearly not really depending on some notion of nearest neighbor search to perform well and one thing like there are a lot of interesting tidbits about how this training process works but one thing I want to highlight is this notion of curriculum learning where the way we train these models is we gradually increase the complexity of the function plaster training so what this says is that okay if we have this function f that our W transforms X then originally during training we zero out some part of the X vector so we're effectively trading the model to do linear regression fewer dimensions so if we're training for linear regression in 20 Dimensions we start from five and then we trade for a while with five and then we increase to six and then seven and so on and it turns out that is actually pretty important um I'm gonna show you here the training laws with three random scenes with and without curriculum and if you train it on 10 Dimensions these are relatively smaller problems than both cases both with and without curriculum converts to a pretty low loss after a while with curriculum it converges faster now when we go into the setup we've looked at so far with 20 dimensions then the difference is actually much bigger uh there is this when you train without curriculum there's this relatively long period where the loss of the model doesn't drop by too much but and then suddenly drops of course if you Trend with curriculum you're actually able to get pretty consistent Behavior you can see that all three seats are basically overlapping and if we're confused of why the loss grows up over training this is that oh during the curriculum phase we are making the function class harder so every time we increase the complexity of the function class well the loss of the model will drop up but if you look after the vertical dashed line um yeah you can clearly see that by this point the money has done really well and this becomes even more Salient if you do go to 30 Dimensions whereas one of the training runs without curriculum failed and if you go to 50 Dimensions where we're actually not able to train any models without curriculum in a meaningful way um yeah I don't know exactly what the optimization barriers are here but somehow slowly warming up the body through this curriculum learning is actually having a huge impact here Okay so where we saw that okay we can train models that in context learn the class of linear functions but what about something and I actually in a way that extrapolates this is not just the in distribution setting but we saw that even when we messed with the distributions a little bit things still work to some extent but what about more complex function classes and here we looked at uh three different things uh sparse linear functions so here your weight Vector is actually three spars and here least squares is no longer the optimal thing to do uh you can use an algorithm like classes that actually performs better and is able to exploit sparsity and learn with fewer examples and now if you drain the Transformer to learn sparse linear functions again this is not the Transformer that we looked at for that was trying to learn linear functions this has been trained specifically to learn sparklinger function from here then it can actually approximate the performance of lasso quite well and again it was not trained to perform less it was just trained to predict well and sparse linear functions and it learned that algorithms actually comparably then we looked at something a bit more complicated like decision trees where you treat every coordinate of X as a feature and then you build a random tree based on these coordinates and this is actually a quite challenging problem especially if you build this Tracer random and you can do things like greedy learning or extra boost but it turns out here that training a Transformer to learn this actually does much better we were not able to compete with the performance no matter how much we tried we were not able to compete with the Transformer using a more classical method uh in this setting and this is something that we don't really understand very well as a community like understanding what are the good algorithms for learning specific families of decision trees and I find this really exciting because it would tell us that oh maybe if we understand what the Transformer is doing here maybe we would be able to discover good algorithms for learning decision trees um in some sense exactly because we're training the model to find the learning algorithm if we could reverse engineer the final algorithm that would be that could be really impact for you and finally I think we're trying trying like two layer rail networks so this says okay I'm gonna sample random weights for a two layer Network and the function is actually the product of feeding X into this network and here a reasonable Baseline is something like doing this gradient descent learning a network of the same architecture um using gradient descent and the Transformer is actually performing quite well is actually matching these Baseline and outperformances in certain cases and note here that the Transformer like we're not doing any sort of gradient descent here this is a fixed model that we just feed in these in context examples and ask for predictions about their label yeah so the takeaways here are that okay turns out in context learning um we can train Transformers I think context learn quite involved function classes so these are the things that are not released like learning building an algorithm that learns decision trees or learns particular regression is not something that you can easily do in a simple feed forward model many of these problems don't really have a closed form solution so in principle um in context learning can be quite real so it's a bit hard to connect then context learning behavior that we see for language models and practice and say how powerful that is but these results tell you that well if you train these models in the right way then then context learning that you can get is actually really powerful and actually in a way that is not really restricted to the training distribution exactly which gives us hope that well these models you train them on some large Corpus of text and then they're able to extrapolate to some extent to other distributions and still have a relatively Behavior all right so to conclude I wanted to yeah uh so so far we looked at these two examples of really interesting behaviors uh of models that we didn't really expect both the existence of adversarial examples came as a surprise and phenomena like and context learning came as a surprise and we saw how basically by digging deeper into this phenomena and looking at body Behavior Beyond just accuracy we can actually get a glimpse of how models can actually work without a sense of oh you know what like models can actually rely on different features and this is what gives rise to adversarial vulnerability or you know what models can actually properly use their context to learn fundamentally new functions and I think this is a really promising research Direction overall that says okay I don't want to understand just the behavior of my model on some specific Downstream task because this can mask a lot of things but I want to try and look a bit more deeply into the various behaviors that I see and some ways to do it is uh you can build bespoke benchmarks you can build uh tests similar to what we did in context learning that says okay here's a very specific behavior I want to capture and here's a concrete Benchmark that I can actually measure this Behavior with then of course the mystery of okay how did these behaviors actually arise like as we said like for adversarial examples are to some extent connected to the training data and what is present there but can we build more General tools to connect how models learn and how different parts of the training data influence their behavior to the same practice and finally it's really nice if we could get a better mechanistic understanding of how these models work uh okay we sort of even if we understand to some extent what they're doing understanding how they're doing it is actually important for instance in the decision tree examples we saw that oh maybe this would allow us to discover new algorithms or more generally if you're able to understand how bullets work this would actually provide you with a new interface to interact with these models and change them in ways that are different from just standard fine tuning or interacting with them and creating new capabilities by just understanding how their internals work and being able to sort of pop open the hood and mess up with things a little bit uh yeah so that is all you have to take any questions I thank you so much uh Dr I think this is a wonderful talk a lot of Joy I feel um this is really fundamental research [Music] helping us understand the reliability of models and the capacity of Transformers as say Universal approximators um so let's see we I think we have time to take some questions I see one one question in the chat so what is the structure of Transformer as say a neural network I think this is for the income tax learning part of the talk right that was that was my question uh just not familiar with Transformers from before models that are yeah so you take as input a sequence of vectors and the goal is to predict the next Vector in that sequence and the way it works at the basic level is you have this alternating layer so self-attention where you mix together these vectors you can you compute linear combinations of these vectors According to some similarity Matrix and then you apply perfect or fully connected layers to transform these vectors individually so the Transformer as a whole is you take a sequence of vectors and then you transform these vectors applying fully connected layers then you do an attention operation where you compute you replace these vectors with linear combinations and then you keep doing this by Computing other fully connected Transformations and then mixing them again and so on um yeah um can you hear me yeah okay great uh so thank you for a wonderful talk that was really fascinating I was just curious to what degree do you think the existence of non-robust features in data sets is just an artifact of their size so if you had you know an arbitrary number of examples of each class do you think all of the features would become robust at that point yeah I think that's a great question like it's not entirely clear what is the answer like to what extent is like this is a minority phenomenon that like oh some consistent fraction of the class has this set of examples or whether it's a small n phenomenon that oh they're sort of individual samples I do have a feeling that there are things that would persist um like they're definitely patterns that seem real and again building again on a lot of this like other work on things like high frequency components like or yeah I can feel these things wouldn't necessarily go away as we get more and more data but I do have a feeling that a lot of the other weirdnesses that we see would potentially go away and I think that this is one of the places where training on more data is one of the ways that you can sort of get waves on this robustness accuracy tradeoff so yeah basically it should be possible to yeah because like if you think about like one way to view this phenomenon is that okay maybe for our current data set sizes uh robust teachers alone are not sufficient to actually get really good accuracy so the fact that we have number bus features you can think of them as a crash that models rely on but if you really hide a lot of data and the model and the robust features were actually clearly Salient and we had enough data to learn the models that are really relied on these well maybe then you would be able to get robust features robust models to free for free to some extent but yeah I think this is like a really interesting question that I have thoughts that I don't really have a crisp understanding uh hi thank you for the wonderful talk so specifically for the first uh part which is with robustness where you try to add the inner objective of maximizing the uh uh the perturbation uh as the worst case uh can can this be thought of as something like data augmentation and is it something that you can leverage for other training so instead of like right now has a lot of the uh applying Transformations but if we okay like can adding the worst case perturbation be something like an additional data augmentation for improving performance of other tasks yeah I guess like yeah is this data documentation I guess the answer is yes and no like I guess conceptually yes it is Data augmentation because you're training on different versions of your data I think it's also quite different in the sense that you're not training on a random transformation but your training specifically on the worst case transformation and this has a different effect for instance we know that if you just train your random noise then you don't get robust models like if you train with by adding random noise or the same magnitude as your perturbations then you don't really become robust so there is a difference between average heat segmentation and worst case recommendation that I think we don't fully understand well yet like when should you prefer one to the other but I definitely feel that this is a very useful primitive like like we have seen plenty of evidence uh that training models to be robust to a worst case L2 perturbations causes them to have different Behavior as for instance there are these results that show that if you train models to be adversely robust then they can actually transfer better sometimes to Downstream tasks so I feel like this is a very powerful tool that we have in our toolkit that says here is a way that they can change what memory relies on and sometimes this change is useful and sometimes it might be not and we'll just drop accuracy but yeah it definitely could be applied to other settings and it's actually something that even if you really don't care about the robust so if you're modded from a security perspective there is Merit in thinking about it actually training on worst case transformations oh thank you uh for a great talk I have a couple of questions can you elaborate a bit uh about this LG Boost comparison because it's not clear to me like what function was uh it was trying to approximate and why it it's not lower bounded by its reverse performance yeah so in general like yeah so just to clarify about them the specific structure here um and they can quickly go back yeah so here are the structures we have x's in 20 dimensions and then you build the decision tree of that four by randomly picking a random coordinate then looking at the sign and branching based on the sign so you pick a random coordinate if it's positive uh the left subtree goes one way the right subject goes another way and the way it works is basically yeah I sample one of these random subtrees and then I generate data according to that train and this is in fact a really challenging setting uh because extra boost what okay so what agree the algorithm is trying to do says okay I'm gonna try to find a variable to split so that I minimize the variance of my two subgrays but the problem is that because there's so much noise here and we're working with x's that are sampled from random gaussians it's really hard to tell apart the two samples of the road so it's very likely that we're gonna that you're gonna split the wrong way um so greedy just doesn't really work at least up to 100 examples and XG boost is doing similar things like yeah like in some sense if you have a real world application where there is a real feature like oh Aid is predictive of something then you would expect these algorithms to find this feature because there's actually a pretty Salient Reliance from this feature to the output but here because there's so much noise it's actually really hard to distinguish these things so basically the task is approximizing piecewise like random piecewise linear function right yes exactly yeah this was constant yeah yeah sorry piecewise constant yeah function and language group is just kind of does it worse than uh Transformer what about time uh so like how sorry not influence training times like how long how long does it take to trade the experiments compared to extra boost like uh extra boost you can run in like seconds these Transformers take like a data train um I mean the inference is quick again the inference is seconds um so it's still like okay yeah even inference of these Transformers we should probably figure out the exact times but I think it's definitely comparable the argument here is I would be hard-pressed if formulate an argument that says you should be using Transformers over extra boost uh this is because again this is like sort of a weird case and perhaps this is not ready yet to be something that you would try to apply it's more about showing that well it turns out that you can really discover non-trivial algorithms that would do better than bespoke baselines we have and so and final short questions like have you uh cubicle would try to like fine-tune I don't know some existing large Transformer model to this task and uh to get like better training time compared to training Transformer from scratch across tasks maybe it's uh it might like be faster yeah so I think yeah both directions are interesting so I guess the question of like if I pre-train on text and then I train on these tasks what is the impact but also what if I pre-train on these tasks and then train on text is this something that I'm gonna get the boost on let's say if I train on models on these tasks we've got explore linear functions will a model discover good Primitives for learning text and we haven't done either of these directions I think both of them are interesting and I would be curious if you try and you see something thank you any other questions if not the let's give doctor to press another round of applause thank you so much