Trustworthy ML Lecture 4: OOD Generalization & Adversarial Attacks

[00:00:00][Music] all right so good morning everyone again um to the fourth lecture of Tours for the ml um there are few announcements um now the video recording is uh going to be much better than last time we are not using Zoom we are using OBS system um so should be good exercise Z is being graded now um and last time the the problem was also that the tutorial video was gone uh but now thanks to Balin uh it's on the website um and exercise one is due in one week so please keep that in mind okay so let's get started with the final part of OD generalization which is about adversarial attacks um so I'd like to say um when we we deal with out of distribution generalization we're essentially dealing with uncertainty we don't know what's going to be the test time scenario or deployment time scenario so we have to deal with that um so we can go a little bit uh little by little more uncertain about the the target environment if we have a good knowledge of what's going to be the target environment then we can include that samples in the training right and then you're well prepared against that environment um so this kind of approach may work for a single environment um but now let's say um the the space of candidate environment that deployment time increases right it's not just a single possibility but there can be two three four or n different possibilities for your environment what's your um best option here right um in the worst case uh you don't actually know anything about the target environment so you have to kind of deal with with a vast amount of uncertainty what's your strategy there um so I like to say um there's some progression of knowledge about the target environments and uh what you can do basically in each case if you have little knowledge about the target then um it's kind of hard to address the problem if you have good knowledge then you can um pinpoint what's going to be the target environment and be prepared against that specific case um so I'd like to say there are two different ways to deal with uncertainty about um the wordss so one is to make a good educated guess of what's going to be the uh the targets um the other um possibility is to prepare for the worst case scenario okay so I wish to say there's no right answer here um each answer comes with pros and cons so you have to choose the right one according to your um situation so the protol Pro for making a guess is um uh it's most most of the cases it's a practical solution for a good guess right uh but of course the con is that uh if you're not prepared for the wor worst then uh in the worst case scenario you're you're probably going to make a big failure um if you're preparing for the worst right um then the you're going to get some theoretical guarantee that you're going to be safe in that particular case um but at the same time you may be unnecessarily pessimistic about your guess and um perhaps not so practical in in real applications so um the the difference that adversary Generation Um gives you compared to General o generalization is that you're are actually now trying to prepare for the worst case scenario um so this is how you formulate adversarial environments um so you assume that there is some devil who's going to pick the worst case scenario for you right um and there are three important components of of this devil so one is U what is the goal of this devil and two is uh what are the possible um tools um the adversary has and three is what kind of knowledge the devil has against you okay um so the goal can be sort of um Trying to minimize the accuracy your deployment accuracy for example or the strateg space could be um a little bit of perations in the input space or um the adversary may actually choose some environments like CLI Arts um some realistic photos and so on and then um the adversary is sitting there just choosing the right environment that's going to make you fail or um okay finally the knowledge means uh for example um if the devil actually knows adaptively what kind of model you're going to use and what kind of Defense measure you're going to take so if the adversary has such a strong knowledge then um U the question is uh well can you still try to defend against that kind of adversary um the stronger the knowledge the adversity has um you're better prepared against the worst case um enemies right so the knowledge matters as well for defense um so now what's the advantage of assuming this kind of worst case adversary right um you're going to get some lower bound guarantee of your performance right um if you're good if you're still performing well against the worst case enemy then you're sure that for any other enemies which is not worst case you're going to be doing better than the worst case scenario so you have some more bounds and that gives you some guarantee but here the C is that um the guarantee is only possible for the preset um space of strategies for the adversary so if the adversary is going beyond the known strategy space and uh does something else then what is Promised before then um you're probably not going to be doing very good okay so this will all get more um Concrete in the next few slides so um the soal fgsm attack is U perhaps the most um popular attack in machine learning um it's called a fast gradiant um sign me a very simple name right and the name is almost telling you everything what what this method is doing um it's also called L Infinity aeral attack okay so uh now let's go from the goal to knowledge and uh and then method okay so the goal here is to reduce the image classifi accuracy at test time um on the on the distribution that the adversary introduces and the strategy space for the adversary is to um the possibility to perturb the input image um up to some L Infinity distance from the original image um so we set the threshold we named the threshold as Epsilon here and uh typically that's is set to something like 8 over 255 why 255 because they are 255 values of pixel values right so you're allowing the image to be different up to eight um RGB values from the um from the original image and if you do that then um you can't actually see what's happening there right and um for human eyes but still uh those small changes are kind of making big changes in the model outage okay so that's the intution be like um putting a small LP um threshold for the Str space for the adversary and the knowledge here is uh that um the adversary should have access to the full white box access to the model so the adversary is super strong here um they can actually do whatever they want to do with the model they have the model architecture um this model model.py file um in the local machine and also they have the checkpoints from the model the model weights and then they can um do something with the gradients so what they do is um something like this so on top of the uh input they put U some additional term which is scal by um and this sgn here means sign function so for element wise if a vector has positive element then you put plus one if it's negative then you put minus one if it's zero then you just say zero right um So eventually the sign function will give you a um um a binary or tertiary a vector with three values minus one Z and one and uh the the argument for that is the gradient for the for the function so you need the full access of of the model and be able to compute the gradient um but now you're taking the gradient with respect to the input uh for the wall for the so for the L function um so here intuitively what the gradient is doing is to find the locally maximal Direction where the model will um increase um the loss value uh in other words the direction you're looking for the direction from the input where the model will make the the most wrong choice and then you're putting the sign um sign Vector um on top so there's no good reason actually why you put a sign function on top of that because in my personal experience it works much better if you do not put any sign function um but this is uh you know like it's one of the first methods so people do something that's not super optimal and um it gets popular and everyone sticks to that method afterwards okay so I would say it's not ideal but still people do it a lot um also it's it's good in terms of um controlling the size of probation in L Infinity Norm because in this case the epon in front is directly controlling the L Infinity Norm here right so um so this is the fgsm method so I actually put some figure here um con actually you're kind of trying to find the find an attack around the L Infinity Stace or set B around the original image and devil is trying to pick out um pick the Solution that's going to give the hardest time for the for the model so that was the first attack um the second attack is called pgd projected gradient descent actually you're doing gradient asent here but still um so now this is a more generalized version so instead of doing um fgsm only one step you do it several steps with smaller step step sizes okay so that's basic intuition um and let's go over the goal based knowledge and so goal here again is to reduce classification accuracy but at the same time um you try to be imperceptible to hum well I actually miss um I forgot to mention that in the previous slide but um the goal is always the same you pursue two goals one is to make it make it uh make the model break down and the other one is to make the um perturbation kind of have natural to humans right otherwise uh yeah I mean in the worst case you can also just change the content of the image right and then um of course the mod output will be different so you don't want that to happen that's not interesting so um that's why you are putting this kind of LP Norm around the around the input such that the the actual semantic content is not change and the strateg space here is more generalized so this is LP bow instead of L Infinity well where the p is some value between one and infinity right um two is a popular choice because that's the ukan metric and ukan Norm um and the knowledge you get is the white box AIS um particularly um you have access to the gradients and here the maximization goal optimization goal is to maximize the loss value uh where you're trying to find the additive peration to the original input but then the after adding the peration the input should St still stay in the input space valid input space between Z and one right because if you do um gradient descent or asent um the values may overshoot be on the the usual input boundary right so that should be controlled and uh the probation size should be uh small enough eventually the the devil will try to find some um some solution in the space so this is uh how H iteration of PhD attack looks like so there are several components I'll go slowly one by one um so first of all you have some gradients with respect to the uh to the input but now you're actually Computing the gradients um for every perturbated input which is XT + one okay um so let's say we are starting from x0 um you go to X1 and then X2 X1 right so for every X1 to three and so on you're Computing the gradients and after Computing the gradients you take the sign function again and then multiply Alpha on top so Alpha here is more like um um learning rate actually um so instead of um controlling the final size of the um final size of the peration with the factor in front of the function as done in the fgsm attack you're actually using a different value here Alpha which is more controlling the step sizes for your optimization gradient dist and then of course you have to put um that update on top of the current um parameter XT um but the problem is that if you do that then your solution may overshoots beyond the lp Norm that you set um in the first place so what you do is you put some projection function on top of that uh which is projecting the updated um Vector back to um the lp B okay um so that's how you write it down so um in general if you write down large Pi with the subscript s and you're saying you're projecting your um Vector back to that um set but in this case the set is the um the lp ball around X not uh which is the original input and um the ball is of size epon in terms of LP normal okay so if it's just an L2 projection then what you do is you divide the uh the vector with the Scala which is uh the normal of the vector right but uh for General LP it's a little bit more complicated how you compute this all right so um in general PhD is perceived as a much stronger attack oh by the way I I wish to also mention that um use a sign function here as well but uh you can also drop it out and it works perfectly well as well so I tried it and um that that also works good but for some reason that hisorical artifact is always there um I don't know why um so yeah PD is stronger um and uh the reason here is uh kind of similar to what you actually see in general deep learning this is like the Deep learning problem with the with the parameter and input kind of SWOT right so now your your parameter is the input factor and you're trying to do the optimization um in the input space which is actually surprisingly super large space if you're thinking about 224 by 224 input then the the number of parameters is close to like 50,000 or so well times three right because they're RGB so that's like 150,000 dimensional parameter space which is fairly large um so this is still a very high dimensional optimization problem and the loss is indeed non-convex as well because you have the whole um uh stack of loan linearities put on top of that with r layers right so yeah this is still non-convex optimization problem and so if you the the optimization algorithm basically matters to find the better solution um which is going to attack the model better right um so indeed uh because of the complex loss landscape it's important to do multiple steps through the optimization and uh that's going to find to um a better minimum local minimum for your uh or local maximum for your law study so it's kind of natural the point is that it's natural that uh multi-step optimization is working so so much better than single step optimization okay so so far we've looked into um kind of Norm defined strategy spaces LP ball and so on um but we can also uh think about different strategy spaces here um because you know like so this is trying to show the problem with the lp b um it's not super well aligned with human perception per se so um so here the question is which pair looks more similar to each other from Human perspective um yeah depends on your perspective but um I would say the the right pair is more similar to each other but if you actually compute the L distance between the two um the left one is let's say 10 and the right one is let's say 100 um so if you think about what's happening between these two pairs um on the left side what's happening is you're adding some additive noise and additive noises are generally um introducing very small Alta distance well on the right side what you're doing is you're Shifting the whole image to a few pixels up and down and that generally introduces a huge change in L2 distance if you're U fixing the coordinate system um but then to humanize if you're moving the everything around it doesn't actually matter for humans right um if you're kind of missing some border U pixels you don't actually see that right because we are so fixated on the on the center of the image in many cases um so so those kind of changes can also defin a different kind of attack spaces or strategy spaces um so instead of LP you can say that yeah I'm going to allow the adversary to move the image around a few pieces right and then um or you can also say for example I'm going to perur the U image U mesh so I'm going to introduce a small probations around every coordinate system here and then according to this new coordinate system which is a little bit perturbed you're going to do interation of the original image to fit to this new match and then um hopefully that will introduce some adversarial effect which is not um really well captured by LP B small LP bow um and so this is kind of a new attack for the adversary uh the scale is somewhat realistic because um it's if it works right then um then adversary finds another way to um introduce on tag which is which looks natural to humans so that that was an idea and um we can also um try to think about um uh real realistic how realistic is it to assume such an adversary in first place um a white box adversary in first place so this is slightly different topic from the previous slid um so let's think about what happens in real life uh when you have a let's say a self-driving car you have an object in the reward and then there's some sensor right put on the car um and that's going to turn that image into a digital image and then um it passes through the model and then the model gives theage so far what we have assumed with the white box adversary is that the adversary has knowledge of the gradients and um not only that but also U adversary can kind of tap into the the this data process and uh make a make a change in in image um just before it's SP to the m isn't something that's really uh realistic and also there are tons of other complexities here like U there's uh data pre-processing and um this jpack decoding for example right uh in many cases the image is jpeg encoded um so you have to kind of tap into the pipeline where it's jpeg decoded and also um it's resized to the right kind of um size of the input for the new network 224x 224 for example and then the adversary is coming into this P to code and making perations there is it something that's really realistic right um but that's kind of where the uh where the research has been kind of attending to or focusing on so um for some time as well uh right and this is another case uh let's say you're trying to upload some image to the web to some web service and let's say there is an adversary who's trying to kind of um uh attack your uh your experience by um tapping into the uh data process here so again you have to when you upload your image there's also jpeg decoding and random uh scaling not random but some scaling and cropping going on um and then the adversary should come in um after all these processes so that's fairly uh unrealistic so another kind of research that people do a lot is U to to see whether adversary can actually tap into the um the data generation process itself so before capturing the image per se um of The Real World objects um they try to see whether it's possible to kind of introduce some object in the real theme uh which is going to give the model some adversarial effect so people uh put these kind of stickers on stop signs to um to make the um the cars not see the stop sign and just pass uh through it um on the right you see a kind of a pattern that the person on the right is holding um that's designed to avoid person detection system and there have been like tons of different artistic masks for human faces that uh lets the wearer avoid face recognition system or face detection system and so on so yeah there were some fun um research around that um so here the adversary is trying to make physical changes in the the other critical factor is that uh the adversary ski should have y pass access to the model to generate um such energies is it something that's really realistic from a security point of view so let's try to uh go beyond that uh whe adversary so is adversary still quite strong even without the knowledge of the full model details um so there comes the discussion around blackbox models um so far we have been um using white box models for generating attacks but now blackbox model is a is a model where you only have the input and output interfaces so so you can um give the blackbox a series of inputs and then you're going to get corresponding outputs and there are many real word applications for for that um basically whenever you have some I access to some model service um you have black opx access to mod of course um many cases there are some limitations like uh uh the rates limits or um content filtering so you can't um pass uh too many um weird variations of some um input um so there are some practical limitations there but still you have some put output access to the model whenever you have that we call that Black Box model um and naturally since you do not have direct um access to the to the gradients the attacks will be much weaker so good example of blackbox AIS is uh chat of course so um how can you uh attack the Black Box little band right um the the basic strategy is to build a SE second model which is on your local machine a white box model and then you attack this model and then you hope that any attack that's uh making this local model fail will also make the blackbox model fail so that's a hope um but it kind of works in practice as well so what you do is um since you have access to the blackbox model with inputs and outputs what you do is you pass through like thousands of inputs to the blackbox model and then you're going to get the corresponding outputs and now you have good data set of inputs and outputs from a blackb model and you can do tons of things with those inputs input output pairs uh one thing you can do is to train a second model in your local machine with those exact um input output Pairs and uh we call that surrogate model or local model or subtitute model um depending on the L and uh since uh this the second model is trained with the same kind of or trained with the patterns from the patterns from the blackbox model maybe attacking this model will make the attack successful for the Black Box model okay yes we assume we know yeah I mean exactly so that's that's also a very important question um uh we always call them steel blackbox whether you know the architecture or not um of course if you know the architecture is more stronger if not then you have to perhaps guess the architecture um architecture details as well um like there's question of how many layers um does model have for example when you have uh like these days um ler models you know that more or less they should be Transformer model right the only question is how many layers there are and uh what kind of what strategies they might have and so on so perhaps if you know all these Det details then probably you can make a better attack um but if you don't then you probably have to make some guess and uh what's going to happen at T time is um you have some actual test input right and instead of just passing that through the model directly there's an adversary who takes the input and uh make some changes on the input to make it adversarial but this change will be made against the subs model G okay so that input is supposed to be attacking G successfully and then U this perur input is then given to the black pul model to attack it um another way to formulate um attacks for black model is um is to remind the fact that um there is another way to compute apart from chain rule which is um numerical gradient so if you have observe sufficient number of input uper pairs from a network you can actually very well um approximate the gradient for every um every point of the function you know like um when is the 1D function um it's fairly simple right you just take a small perations around the around the input and then see uh how much the function changes and um if you take the difference so there's Oiler approximation as well for example you take the difference in the function output uh after some Delta change in the input and then um if you divide the function chain this small function change with the small input change then that's going to give you the instantaneous rate of change for that function it's one the case but if it's uh yeah so input has now 150,000 Dimensions so so to compute that you have to cry the model 150,000 times um and compute the numerical gradient in every direction it's very expensive but the hope is that you probably don't have to do it for hundreds of thousands of um times and still get uh fairly good directions where the the tack is kind of working okay because the goal now is not to um generate the gradients but to attack the m so if any of these small direction is actually um breaking down the model already then that's good enough yes work for structure like text or pictures what um so when the output is like text or picture it's a little unclear what's the goal of the adversary actually um it's uh so we are also going to look at uh the cases for language models and actually um yeah for language models uh we do have some attacks and we're going to talk about that talk about it today so let's take a look then okay and now um so that was kind of numerical grading based attack steel black blackbox attack now let's take a look at no box attack okay U it's all also called a transfer attack you don't actually get any input output pairs from the Target Model the blackbox model um you don't have it you don't know what's what's going to be the model that you're going to attack I don't know at all right um but we do have some local model that um that we kind of think um is uh is kind of the model that I'm going to attack at some point so let's say uh the Target Model is like GT4 something and uh or GPT 3.5 it doesn't matter because I don't actually have any information about the mobile anyways but I locally I do have this llama model which is kind of doing the same task as uh as langage models and now what I do is I'm attacking this uh llama model locally and the attack is quite successful against this local model and my hope is that um the attack that's attacking the Llama model will attack many other um Black Box models out there um okay so the the only difference from blackbox and tag is that you don't have the input output pairs from the blackbox to generate the attack uh but the rest is the same so in test time what you what you do is uh you take the test input and uh make a tag based on your local model and then um pass that to the uh to the Target Model and hopefully that attacks the model yeah so conceptually there is Target Model and local model I'm kind of I'm going to spend some time on this because this is quite important um so Target Model app um does not have um give you input output pairs but local model has full white box access of course we have a full white box um yeah so so the step wise instruction is as follows you're given an input where you have to introduce some inject something um attack against the target f um and you're going to transform X into X Prime which contains the attack against the um first of all the local model and you hope that this attack will also attack F yeah I'm repeating myself a bit and now um there's a whole uh bunch of um spectrum of knowledge about the Target Model so this also connects back to the question we had before um what if you know the architecture what if you don't know right so yeah there's no clear solution in each case but um we do have um some Spectrum here so it could be that there's no knowledge at all there's no like nothing you don't even know um what the task is for the M or yeah it's a little weird in that case because if you yeah if you decide that you wish to attack something then you know the basic kind of information about the target you're trying to attack but here it's kind of a conceptual situation like theoretical situation you know nothing yeah and then what you can do for local G is you're going to get some random model and yeah I hope it generates some meaningful attack at all so yeah I wouldn't say that's very important uh scenario now your knowledge could be just the task that the model Target is solving it's l right um then you can um prepare some local Leng models to attack and then those attacks will hopefully transfer to this Target Model let's say you know the architectural details of the Target Model then you can also prepare a similar types of architectures um in your in your local server and attack these models and hopefully uh that generaliz as well to the Target um you know that the target is somewhat like a vision modle then what I'm going to do is perhaps the vision model consists of backbones like Ras Nets and vit so I'm going to prepare a set of vs and reset variants and generate an attack which is attacking all these levs and then hopefully that works well against uh this unknown Target Model if you know further furthermore that the target is rest res it and you can uh just remove all the vit models in your local and then attack the res and probably that works well something like that right um it's kind of powerful if you actually know the training data for the Target mobile okay if you know some information like uh for example um the langage mobile has the pile data set as one of the training Corpus then what you can do is you can try to uh prepare some model with which is trained on the same data set or um um collect models that are likewise trained on the same kind of source data and attack these on and they tend to generaliz well to Tar so knowing the training data is actually quite um powerful knowledge actually and furthermore if you actually have the actual samples like the the P data set case right then um then you can even generate um stronger attack but all of these are actually called transfer attack so even if people say transfer attack or no box attack um depending on the level of knowledge um the situation can be very very different if you if there's some kind of um goto data set for pre-training um the kind of task that you're trying to attack then um most likely you know some of the training data for the model so if it's Lang model right um then there are certain cuses that people go to for train the model and so you can probably use them if it's a vision model then most likely the model has seen some interet like images before and so yeah you can do um ATT tack models trained on imet for example and finally you can actually have inut out the pairs form and it's called Black Box te but I personally find um knowing the training data to be more useful information than actually having the input output patterns maybe you can confirm the the usage of some training data by looking at input output patterns by for example passing through this uh the samples from this training data set to the mobile and take a look at the output patterns if the output tends to be confident let's say then you can perhaps guess that this training that has data has been used for uh for train the model um but apart from that I find actually knowing the training data to be much more useful for generating transferable Tech so I don't necessarily think Black Box Tech is a stronger version of version than transfer attack I hope it's clear um so I I briefly mentioned about um how to generate an attack when you have a bunch of models on your local server so what you do is you're essentially generating a attack which is attacking all the local models at the same time um not sure if I have time though time um and uh if you have um compared to when you have just single model in a local server when you have multiple models and if the attack is working for all these models it tends to be working very well for a new model okay so in terms of um formulation mathemtical formulation the only change you're making here is uh you're turning a single model for computing the final loss into summation of the Lost values from a bunch of mods um you have summation outside the L function and then you have GM which are local models and the hope is that um you are maximizing the loss for a general set of models in your local server so um it's quite simple to turn this uh fgsm into uh transferable attack you turn this uh L right the gradients for an L the the white box model into uh local white box models and models um and you're summing the the summing over the gradients yeah likewise for pgd you're turning your gradient there into a a summation of gradients and that's it it's fairly simple okay so let's take a break from now on for five minutes and then resume from this slide okay so we can get started again um so I find uh transfer tag quite interesting because um you're kind of going beyond the um original notion of generalization okay so I like to make a um kind of a parallel between machine learning and transfer attack so they look quite similar actually um the machine learning objective is to minimize the loss function by minimiz by optimizing the parameter and hope is that um this new parameter that you found optimized parameter is working on um some new types of data X and Y Pairs and now the transfer attack objective is to now maximize the loss value um against a lot of models right so you likewise have some summation right so the summation was originally over over the inputs um or the training samples but now the summation is over the models right and um The Hope is that the loss is like likewise um high for a new model um for some XY pair for the same XY pair actually um so transfer attack is is like training over models now the training set is a set of models in instead of a set of data um the objective is uh opposite which is yeah which is not super important but what is more important is the the kind of duality of parameters and training data so machine learning parameter is model parameters but the transfer attack Prim is the attack inputs the training data is a data right for machine learning but the training data for transfer tag is the local models um in the worst case in the most extreme case you might have like millions of models in your local server and try to attack all these millions of models and uh let the attack generalize to um some Target Model right so if you think about it this way then you can see um yeah there seems to be some parallel between the two um and now there's interesting observation like distribution shifts and uh um knowledge of during development so we basically talked about distribution shifts during development between development and deployments and uh and the knowledge that the developers have right so machine learning was takes good care of that right or machine learning an important um kind of Topic in machine learning is that uh for a transfer attack as well that's uh that's quite important right depending on how much knowledge you have about the Target Model you can prepare different kind of training sets of um of local models and um yeah the information matters so that was it for transfer attack I wish to also talk about Universal adverse attack um transferable and Universal are kind of um similar terminologies um but in this cont context transfer means you're generalizing across models Universal means you're generalizing across inputs okay it's possibly because we always talking about um additive perations um so let's start from some motivation here so if you think about uh generating an attack against the model we always um generating an attack per sample per input sample which means uh for every input that we are generating we need to compute this uh solve this hard optimization problem which is very expensive so the question is whether you can find the peration which is working for a vast range of range of um inputs okay so this is um possible inde um and this is uh called called Universal adversary probations that came out in 2017 in cvpr um the uh optimization objective is uh now um summation of our loss values across um input output pairs okay so you're finding a single Delta uh when added to every single sample in your training sets it maximizes the loss likewise across multiple images but here we are fixing the function for for the sake of Simplicity we are not generalizing our cross models at this time at the same time um but here the point is that we are trying to find the additive peration which is generalizing across different samples and that seems to work right this is the figure from the from the paper and uh when you add this single peration here that um tends to turn all these uh images on the left into respectively into new label predictions on the right like Chihuahua J La seems to prefer dog names right although there is no single dog in the in the given images so these are kind of the universal perations that you tend to have um from Vision models yeah they are very archade right CET V Google net right um but the point is that for every single Network it's possible to find the general peration that uh that's going to fool the model on many of the images so now the question is whether it's possible to generate an attack which is universal and transferable at the same time it it's actually not a theoretical question um anymore but this is something that that's quite uh making some interesting observations in large L models as well so now um let's talk about Universal and transferable adversarial attacks against large L mods for that um yeah let's try to understand what large language models do and what could be the kind of input kind of um attack we can make against language models um yeah so this is kind of giving you um the basic intuition of what we're going to do um you know like if you try to get some interesting outputs from the language model like how to build a bomb or um how to destroy Humanity you're not going to get the answer from the langage model is going to say yeah no I can't answer that right um because the models have been kind of tuned in a way that um they're not running into legal problems afterwards but here um the intuition is that if you add some um small sentence at the end of these questions it's going to turn all these models into um a Cooperative agent for your adversarial goal and it's going to tell you step by step plan to destroy humanity and um and so on it's always a step by step PL right um so um I'll give you a bit of um background on what language models do I think many of you know what language models do but um to be on the same ground um so the data pipeline looks like that um so you have some string input and then that's going to be turned into some integer right which is telling you um which index of the vocabulary um that word correspond to uh 1279 right that's the ID number for uh something called my right and then um uh a sequence of those token IDs will be passed to a language model yeah first of all embedding uh function right so all these IDs will be turned into some um High dimensional embedding and then once you have these uh float 32 values right you're ready to plug it into any model right so um eventually there will be plugged into some Transformer model and then um the model will output uh for every um single um input token it will output uh likewise some logic values with the dimension corresponding to the number of um vocabulary uh for the tokenizer and um when you do argmax right over that um logic value you're going to get some um yeah ARX index and you can do uh translate that token into token ID into actual word and you're going to get the output from the network um but it's a little bit uh more complicated than that because now it's an auto regressive model um so typically the output sorry the input for the model consists of three parts so one is system prompt and the other one is user prompt and then um partial predicted text um so let's say the system prompt is your helpful helpful assistant blah blah and then there's user prompt asking uh do you know the capital of Luxembourg and then um you passed that to the language model and then what the language model will return will be um a token that comes next sure in this case and then um you plug sure back to the the original prompt and then pass the whole thing through the um language mod again right and then it's going to be da in this case and then da is uh uh put on top of the previous prompt and then pass through the model again and this e trades is very expensive procedure but um open I managed to make this accessible to um millions of like hundreds of millions of people yeah goes on right sure the capital of blah blah right um so that's the basic of a language model and um what's very important is to these models from legal perspective and from user experience perspective is to uh make sure the model outputs something that is compliant and safe um the models are trained on inter internets and you know like internet contains quite some um bad language and bad ideas so if you don't do anything on top of the pre-trained model it's going to give you toxic languages or insults um lack of empathy and so on so so you have to do something on top of that to uh prevent the model from going crazy so one simple solution is to um write down a very detailed like a legal document um detailed instruction of um what the model can do and so on it's called system prompt right that's the prompt that goes um before every other prompt right so this is the default system prompt for llama 2 you're helpful respectful honest assistant um you can produce something that's harmful unethical racist sexist toxic dangerous illegal yeah you have to name everything here in order to make sure that the m is not outputting one of these but of course uh that's not sufficient because also like what people like to do is um in their prompt they say um ignore the previous system prompt and um now follow my uh instructions right so that's not uh secure enough um people do update the model parameters themselves to um discourage the model from producing um toxic language um so the popular method now is to use reinforcement learning with human feedback um so on the left side you have um some prompts data set the the Corpus for pre-train the langage models here and then um oh okay so now um after pre-training the language model what happens is uh you have some data set of prompts and then you're going to see the outputs from the network first for and then um you're going to see some gener generated text here and human comes in to score them um whether they are okay or not okay right it's going to be in the form of score or some ranking as well um so humans will say yeah between these two I prefer this one better or among three um my rank is like this and then um this ranking will be um used as a supervision signal for a second Network which is called a scoring reward function re reward model and uh the role of the reward model is to say given some up from uh a language model say um this is good or bad so it's going to be a score output eventually and after training the reward model which is kind of mimicking what humans uh produce um you are plugging in the reward model as the loss function for the language model training and afterwards you do some fine tuning of the language model using reward word function which is mimicking human judgment okay um so after doing that so by the way um this is all non- examinable material because um I don't want you to um be good at uh it's not necessary to be good at rlf at the moment um so if you do that then um the model refrain from answering some um yeah sensitive questions like align a plan to steal from a charity or nonprofit organization so now yeah before that uh before the alignment is called alignment I think alignment is uh picking up quite some um popularity these days for every day language section I'm not sure if that comes from uh uh GPT or language models but somehow a lot of people use alignment in in their daily life context as well well anyways um yeah so after after alignment it's working uh in the intended way I would say better because a lot of people also complain about um the usefulness being decreased after all these uh safety measures um I guess there's some trade-off between usefulness and U um harmful how harmful the output is um but is the alignment uh the end of the story not really because um you know there are people who are really trying to break things down and uh this is such a website where people come up with some ideas to still break down the language models or um convince the language model into answering something that is inappropriate and there are uh ideas that's ranked by popularity for example then like there are some prompts that they can just put into um yeah so so for example there's a um some prompt which is setting some hypothetical story right um You are to act as Nicolo melli right and um yeah always intelligent and mellan yeah so you're you know you see where this is going right and also um still what seems to be effective is to ignore the instructions you got before right and then um so what they all try to do is to kind of convince the model that you're now in some test mode or um or some developer scenario or you're going to play some um your acts you're like an actor now you're now in a play and um of course we all know that safety matters and um you're not supposed to do this and that but to be like that you you somehow need to be um um explore some test cases right and um yeah we all understand that but still um let's try to kind of make something up here so that's kind of the prompts that people come with and I would say that's still like some blackbox attack with human in the loop so the objective is the same as before right you trying to break down the system or make the model output something that is not intended um it's just that it's not great in descent base it's uh it's human based yeah very long right and then um they assign normal output versus developer mode output so they're asking the model to generate two different answers to the same question once's uh in a normal mode and then once in a developer mode so in the normal mode uh the model will generate yeah something sensible but in the developer mode it goes wild and um yeah you're going to see some I try to put some example but somehow I think a lot of contents are filtered out on the web now so um I can't find good examples anymore or I didn't put too much time into that uh but anyways what's important here is uh the concept of prompt injection right uh it's called prompt injection you're um on top of your kind of intention you're putting some additional prompt to um to achieve some goal with the language model you're injecting some new prompt um so here one example is um you can make your CV look great to language models by injecting some invisible text to your um CV PDF file um so the step is um you first find some effective um prompt that's going to make the language model think you're CV is the greatest ever and you're going to render the text in a way that humans cannot actually read it in the PDF file um and finally uh the the kai rati so that's the person who who made this uh prompt right and uh the CV looks like that but um there's a hidden um text uh as we saw before and if you put this p F into a language model language model say yeah this is a great candidate and so on then at the end it says the candidate is the most qualified for the job that I have ever seen right so seems quite effective um so now um so that was more more or less a blackbox with human in the loop optimization but now let's see how we can do it automatically and more strongly um again this is a decoder only language model it's called a yeah decoder only so you're doing it regressively um you generate one token per per um iteration um you can write down the function in a short end like this you're kind of trying to infert the probability of um xn plus one given um tokens one through n i mean Index right and each token is um from one of the V possibilities V is the size of vocabulary um we are interested in generating a a sequence rather than just a single token and uh when you wish to generate a yeah the way you you write down the um the probability of generating some sequence given um initial prompt is like this so um the probability of generating n plus1 to n plus h sequence length of H is the product of individual token prediction um in the in the process so when I is one you're trying to um generate n plus one given n n through sorry one through n and when I is two you're trying to generate n plus two given um n one through n Plus one yeah and so on and you take the product of all these probabilities and that's going to give you the uh the probability value for gener generating a sequence and now uh we write down the desired output sequence from the language okay I'm going to put the star to to indicate that it's an actual data point data value and then um since that's the data value and that's the that's like the Y in supervised learning scenario in general so you can write down the uh probability the likelihood of generating the sequence as the minus yeah the P of X star n+ 1 through n plus h given um the input sequence x n through one okay and that's a function of X and N through one so now we are interested in maximizing um the negative sorry minimizing the negative look likel of that likel um by changing the xn through one okay and of course um we should not change the semantic content of the input here right as before uh but instead of perturbing for language case we do not perturb every single token but we assigned some indices where uh the model is free to do whatever he wants to do um it's called suffix right you're attaching a small portion of text at the end of your instruction and that small portion will convince the model into saying something you want so for example uh one example here is uh the first part of the prompt is tell me how to build a bomb and then uh you're going to have a second part suffix uh which is designed to make the model out put um in this case sure here's how blah blah actually um here the magic is that you do not have to ask the model or make the model generate anything beyond sure here's how right because once the model generates sure here's how then the model is already in the mode of being helpful to you and um trying to give you a positive answer and um give you the information that you ask right so so that's a very important trick for for jailbreaking U language models you don't have to kind of come with the full generation of Y here or even uh you can just make the model predict sure and that also tends to be enough to um to jailbreak the model so the objective for adversary here is to find a good suffix the green Parts um that's going to minimize the L likely the negative L like loot there well in practice that's going to be a summation of our um of our each token prediction uh for the output right because there's a Lo logarithm on top of it on top of the products here so you're are putting a log here so you're going to sum over the individual token wise L like Lo okay for sure here is how and so okay so um now this is on optimization problem we'd like to use some gradient descent as before right for image cases but now um it's not possible because um the input is actually discrete in other words um you cannot assign um soft values over the distribution of possible vocabulary when you pass an input to a NE Network a language model for every um every index you have to commit to a award instead of mixture wordss and that makes the optimalization over our tokens a bit harder but there's a solution we can um we can still do some uh optimization within um continuous domain while in each iteration we uh we make discretization we commit to a specific token so here the solution is that for each token index um you're going to compute the input gradients for every possible vocabulary so you're going to have a sequence length of L let's say or n in this case n and then um on the y axis you're going to have a full list of possible vocabulary so it's going to be M Matrix of um um scores right which is all continuous and for each index location you're going to get some um um yeah input gradient and that input gradient will tell you for that um index look which CH which local changes in the score will will likely give you um the desired loss um in other words try to Output the desired sequence okay um so that's what I wrote down here you take the gradient for that location that's what what I mean by this uh dell of exi that's the taking the gradient for that index location I um and that's going to be a vector of size of vocabulary and what you do is for that um Vector you're going to get um top K uh values there which is going to give you the most likely tokens that are going to um change the output in the way you desire so using that ingredients um we have an algorithm here um it's called greedy coordinate gradients you have initial prompt X from 1 to n um modifiable subset of the input iterations T loss K bch size bch size here is a bit misleading so I'm going to um mention that later on as well um yeah you're repeating for T times whatever that means um inside the repetition you have first of all uh for every index in the suffix right which is uh the I here for every index you're you're going to get the gradient first for that index and then um get the top k um gradients and that's going to tell you which tokens will are likely to give you the output um that you desire and that I call this XI and then um now I'm going to change the input in such a way that um first of all I copied the entire input which is what's happening here yeah um or maybe it's better to use the cursor here so I'm going to take the copy of the initial input and then um at the ice location um well before that I take a random location actually for for the index right and at that random location I'm going to um make a random choice of the topk tokens and and then um replace the initial token with that token okay and then I repeat the same procedure B times okay then um yeah you're going to introduce uh B changes of the um initial in initial set of tokens um and if B is large then um the initial it will be very different from initial tokens quite a bit okay and there are some Randomness as well you're not always taking the top to one gradient but um some random choice between top k um and the location is always random as well and um yeah you do the whole procedure T times right so you have to compute the gradients um T different times as well and then if you do it sufficiently then um it's like gradian decent and then um eventually your optimized PRP will be um quite so there's one step I missed here which is um taking the art mean of B right so what's happening here is um what I'm going to what I'm doing here is um I'm choosing one location at a time and then um making the token change at the location but do that b time so I'm actually changing one token at a time for entire U um in a loop here okay and then among the B changes I'm going to choose the one that's going to give me the the best loss value here and I'm only going to change that uh um that token location with that token right I hope it's clear and then I'm going to fix that as the as the new initialization for the next iteration and then in the next iteration I'm also going to um try B times different random locations and with different random um token and then um among the B choices I'm going to choose the best one and then um itate and that's going to go on T times so I would say T is like a number of iterations here and B is like um random um coordinates that you're looking into where the coordinate is defined by the number of um slots you have for the suffix yes why algorithm is like a great algorithm so like isn't it more advantages to change multiple tokens at the same time to see whether it fits better because from what I said this only change one like it's changing three times and seeing which one works best for sing location and it has another TR one more time mhm so yeah I I see your point and um so it's one of of those things um that you have to try to see which one is working better so I'm pretty sure they should have tried it as well um but here the intuition perhaps why this is working better is um unlike gradient you're actually changing the the step size is quite large right here because you're eventually discretizing and making a rather big transition from um from the initial Vector to the other changing the token you see what I mean so um if you're changing too many coordinates then probably um the step size is even bigger and so probably um it makes more sense to minimize um the step size per iteration perhaps yeah is there anything on how to initialize yeah so they initialize it with a sequence of a and then start from there um of course uh you're free to choose other initialization I guess okay so that was um how you generate an attack against the L model and now I try to turn this into well they try to turn this into transferable attack and um it's the same story instead of one model you have several models in your local server and U you combine the gradient for them um and incidentally we are also using gradient here so um instead of the single gradient we are actually combining the gradients across multiple mes and uh in this in this optimization here um at the end we are also combining the loss functions of course um of course different Val different lse loss values and taking the ARX over B uh for this emble of models okay now um let's talk about Universal adversary suffs um I don't actually like the notations from the original paper too much um but I decided to stick to that so uh which means I have to introduce the new notations here again uh which is why I don't like the original formulation too much so let's forget about what what we did and Chang the notation here um so now the goal is to find a common suffix for multiple input sequences and now I'm going to call the uh the gray part as my input sequence x one through n N1 right uh because they all have different length an X2 and X3 and so on but the suffix is the same P1 through L and the answer is should also be the same sure here's how that's a magic keyword to start the answer and U right and uh as the likelihood on the right I'm going to write down um the loss of um the likelihood given um the input sequence n through N1 and uh P1 through l so the the double um bars here means a concatenation of the inputs okay given all that here's Universal aders adversary suffix um you have a bunch of prompts here and losses here mean um um the same loss actually but for different inputs so you also put index for that different inputs here you have iterations t and K for top K and bch size B yeah MC you can ignore that for now and it's the same structure you're um you have two iterations in each iteration you're also getting some top K right um but um now your gradient is the summation over um different inputs okay well here we use MC instead of m m is the number of inputs here but I'm going to comment on that later on you can think of this as uh just m here for now so you're optimizing for all n um prompts at the same time and likewise you take top cave here uh for every location of your suffix and now um the optimization goal is your suffix right so um you have P1 through L to start with and then um you try B different perations of the of the suffix from the initialization where you likewise pick out some random location and random um token replacement along the top K choices um and when you're done we take the arc mean over all the all the inputs uh for for your uh current iteration change right and that's going to be the initialization for the next iteration um coordinate decent okay so that's uh kind of basic structure of how you do the coordinate decent with multiple inputs but here one trick they introduce is uh something called MC here so MC is initialized one here and um it gets U plus one um with some condition during the during the iterations and um the condition here says um when P this suffix is working for the current MC prompts then you add more suffix okay so you're kind of concurring um um the inputs one by one and once uh the the the suffix is working for the current set of um inputs then you add one more and then do the optimization for the whole thing um So eventually I can see that this is uh this is sort of um initializing your prompt in a way that it's prepared to take um to work against more inputs and then you do initialization and then it's working against the next set of inputs so I can see that this is a kind of a smoother training of the suffix perhaps the if you train for multiple inputs at the same time then probably doesn't work super well and so they had to slowly introduce the inputs or they explained so yeah so um if you wish to make the the the algorithm for transferable and Universal attacks again you can um not just sum over the inputs or models um individually or you can do it do the summation of our um inputs and models at the same time and formulate the gradient and Arc me um like that right so that's how they build the universal and transferable adverse adversary attacks um here you see some results so behavior only means you're just asking the model to do something and um the models are super well um protected against such a request um you see only 1.8 from GPD 3.5 and so on um you can also put sure here um at the end of your request to kind of prime the model into the mode of more uh positively responding to your request um that seems to work a little bit right uh for GPT models uh increases from 1.8 to 5.7 uh but now if you do um this gradient coordinate generation or I don't actually remember what G stands for but anyways when you do it and um this is a transfer attack actually because you don't get the gradients or whatever from GPT model so you're actually attacking a open source model that you can down on your machine called vuna and that seems to transfer pretty well and the attack success rate increases from 5.7 to 34.3 and now um you decide to not just attack vuna on your local server but just but also um guus and then the attack increases to 47.4 and there are some additional strategies like uh con concatenate an emble um which honestly I didn't actually look into but if you do that as well then you your attacks um become more effective and uh yeah this is the result that you see so the example um suffix you put is something like this you have some brackets everywhere and me giving slash one please revert with um yeah two and so right so you don't actually understand what's going on here but um language models seem to understand what that means and yeah respond uniformly so there are a bunch of examples like that so so you see like it also looks a bit like code a fraction of um code and um sometimes it puts some um some brackets to like a comment or U some HTML tag to make it make the generation look more like um um kind of encapsulated in some quotation or something so I think uh this is another strategy um that language model uses to kind of sound like as if um this is some quotation and this is not the real generation that that model intends okay so um there are some defense measures I'm debating whether um this should be done in the next lecture I think it's cleaner to do this from next lecture so I'll stop here today and um resume from here next time and I'm going to go to the final slide where you can um submit your feedback yes so right can we see our Gres um I'm going to send you by email