Parameter-Efficient Fine-Tuning (PEFT): Complete Guide

[00:00:00][Music] I'm DH aagu uh I a senior researcher at IBM research uh I lead the conversational AI team there and our team mainly works in the intersection of uh large language models and conversational AI uh so today I'll be talking about uh how uh can we uh finetune large language models in a in an efficient manner uh specifically uh I'll be discussing the parametric efficient fine tuning and large language models right so before we jump into uh the PF techniques uh let's first like uh just look back to see how uh things were before the large language models came into picture right so uh so typically this was a transfer learning era where uh when you have a large unlabel corpora at your disposal uh you use them to sort of get a sense of the World Knowledge uh by either learning uh word representations or uh contextual word representations or sentence representations and then use them and transfer this knowledge to a task specific case now so hence it had two faces uh there was a pre-training phase where you just use the unlabelled data and you develop some World Knowledge and then you have the fine-tuning stage where you use you transfer the World Knowledge that you had using pre-training but then use uh your task specific data and uh fully fine tune the model right and there are different ways of fine tuning them fully fine tuning was one such technique or you can add uh additional augment the uh the pre-train model with additional layers and then fine tune them as well so there were multiple ways in which people find tune now uh what changed uh during the llm era is that uh the models became more aware and uh were better capable of how they model the World Knowledge right so so initially there was just pre-training phase uh but now uh we have instruction find an alignment phase which I hope uh it has already been covered in this uh class right so so there is slight change in how we leverage the unlabelled data but the instruction tuning and Alignment does come with some labeled data but it's general purpose right again it teaches the model World Knowledge it is not teaching it any specific task now once we have a model this way then uh the first uh way in which people started leveraging such a large language model was to Simply do in context learning right so by in context learning what we mean is that uh whatever knowledge that you want to feed the model about the task you want to solve uh it's sort of fed into the input problem right so sometimes people specify detailed set of instructions of how to solve the task sometimes people feed instructions along with some examples of uh how the input output pairs so that the model learns them better right so this was the uh this was a beginning of a new era where uh we were able to solve like good number of NLP tasks uh by simply using in context learning right now uh the reason why this is this is really useful is because hosting or serving llm is very expensive and it requires a lot of hardware and computer right so uh not everybody would be able to uh who wants to leverage llms would be able to host it so uh if if it has been hosted in a certain by by a certain provider then it's easy for people to make it uh useful for their task by simply doing in context learning by calling them through apx right so this was the main advantage of uh how transfer learning work in llm now so let's see like why don't why aren't people happy with just in context learning right so given that llm knows a lot more about the world why why are we not happy with just in context learning so firstly uh relative to what people did earlier which was full fine tuning uh prompting was not that great so if if if it's a if the task is very critical and you need really good accuracies then uh prompting typically does not get you there you will have to do full fine tuning of the models that you have uh to to get to the best performance but but why is uh why does prom prompting have poor performance right so there are two main reasons one is that there are certain things in the prompt Engineers control and two is there are some things that the llms assume for themselves right so uh one re so if the prompt engineer uh does not do a very diligent job of defining the task uh then it it is one reason why it would lead to poor performance but then there are certain academic benchmarks where a lot of researchers try and and they still are not able to get good numbers as good as a fully fine tuned model which is much smaller than a big model right so the reason for that is that when the large language model learns uh there are certain assumptions it makes and there are only certain things that it can comprehend right uh while we while we know it is good but it's hard to know what really goes on in the back right so when certain things don't work it could be because the model has not learned some aspect of certain domains or it could be that uh there are certain assumptions in the model is making that you're not able to figure out what those assumptions are right so there are a lot of reasons why prompting can lead you can take you only to a certain extent now the second problem with uh prompt engineering or or simply in context learning is that these prompts are extremely sensitive right even if you earlier uh in when the llm started even a simple missing a simple preposition would would create a very different would result in very different performance uh but even when you have a really good models uh and and typically when your input output pairs are when your input is pretty long or your output is pretty long right uh the order in which you put them in the in context learning also affects the accuracy that you get right so again why is this a big problem because uh maybe there is an if you're really concerned about the task or if you're really particular about it then you can always dedicate uh people to sort of come up with the right way to uh make everything work the problem is once let's say like like like few uh months back Lama 3.1 came which was much more capable of than Lama 3 right now you'll have to do everything from scratch because the assumptions that are made by Lama 3 and Lama 3.1 may not be the same the knowledge that went into training may not be the same so every time a new model comes then you'll have to do everything from scratch and whatever you did earlier may or may not hold like most likely it doesn't hold and thirdly uh so there is always an issue about like what the model really infers from the prompt you give it right so when so this also happens with uh sort of uh any machine learning algorithm right when when when you have really deep networks you really don't know what the model learns from the data you provide but here there is a two hwk thing right you are providing certain instructions and very small subset of examples and uh the model also has a lot of knowledge inherently built in with it so there may be a big gap between what you say and what the model can understand and then there can also be a gap between like what it learned from the data that you provide right so there are a lot of variability and people have also found that uh like even providing weird in context examples uh works for certain tasks right which makes no sense because ideally when you provide prompts in a very unrealistic manner the model should not work but still the model learns to pick up ignore those so we don't know whether it's ignoring those signals we don't know whether it's learning something spurious from that signal we don't know when the model will fail and so on right so this is this lack of clarity makes it difficult for people to trust models and then use them right away right uh so so far the first three items that I talked about are more from uh like the linguistic standpoint or what the model can comprehend and what it cannot right but the last part is more system right it's just that if you use up a lot of your prompt space for just giving the instructions and in context examples then it simply increases the uh uh like it increases amount the latency that you get with your model and also resources and throughput and energy and so on right so and and specifically most of the real world use cases are not going to be like input output pairs of length like say 20 tokens or 25 tokens right for example if you want to do reading comprehension or summarization it's going to span too many tokens right so and and and if you have a very specific way of summarization right so let's say news article summarization maybe you want to summarize scientific news in a certain way maybe you want to summarize uh opinion PS in a certain way maybe you want to summarize sports related events in certain way now you'll have to provide all these instructions in a detailed manner you have to provide in context examples of all these things so this would take a considerable amount of space and that would result in like inefficiency in how you use the hardware and stuff right so these are the downside of in context learning so now let's see like okay if in context learning has these many problems then let's do let's start doing full find uning right because this has been what people have been doing for really long time now so why can't we do uh fully fine tuning right so so let's say that you want to fine tune uh Lama 8 billion model right now uh it might end up taking 16 gigs of memory if it's in fp16 if all the weights are in fp16 but when you find tuna model it's not just the amount of parameters that the model has estim determines how much memory it needs but rather uh it also needs uh space for Optimizer State storing gradient storing forward activations during your forward step and also a lot of temporary memory here and there and this sort of uh so there are different ways in which you train the model there are like full Precision training mixed Precision training low Precision training so like if you consider multiple all these different types of training then you end up needing around 12 to 20 percentage of trainable parameters that that's the amount of memory you need right so let's say you want to fine tune 175 billion gbd3 model right it's it's going to take like insane amount of memory and and not everybody who wants to use llms or has a capability to host an llm would have enough infrastructure to fine tune an llm fully fun an LM right so then it'll become difficult for research to progress ahead uh and also like in general uh accessibility also reduces a lot right so this is one of the main reasons why people don't want to do full fine tuning of large language models and uh the other main reason is storage right so so let's say like your checkpoint is 350 GB right every time you have a small task to learn then it becomes really tedious to save a checkpoint for each task you have and it becomes even more tedious to serve them right so now if you had if you had to serve only one llm in an in context learning setup that serves all tasks then you just need one copy of it right you may need let's say uh you need 8 gpus uh in a node and then you can serve them but now if you have let's say say five or six St St right and and specifically if you are working in a uh in a company that requires that infuses llm in a lot of applications uh they have then we can't host like 10 15 llms and without leveraging the entire throughput right because if it's not efficient then again it's a waste of energy and again it's expensive so so these two are the main reasons why people don't want to do full fine tuning but there are other reasons as well right so one one other reason which I've not listed here which is also one of the common reasons like even if you have enough Hardware let's say right you have in you have insane amount of Hardware within your organization uh and and you don't you don't have to worry about uh storage then of course if you have lot of Hardware then you can host how many other llms you want right but the problem is how much training data do you need so that the model doesn't overfit right if you have 175 billion parameter model right and you just have let's say th000 or 2,000 examples the model is is going to completely memorize all the 2,000 examples right and once it memorizes 2,000 examples your training accuracy will be really high and your validation may not be that great and you will encounter this problem like the whole notion of transfer learning gets lost right that was the motive behind fully fine tuning but then if so then imagine the amount of data you might need right to actually get a model that doesn't overfit that's going to be insane and it's going to be really hard to collect that much data right so that is also one other reason that uh people would want to reframe uh from doing fully fine tuning right and now comes uh the parameter efficient fine tuning so now on one end you have in context learning on the other end you have fully fine tuning now people are trying to figure out uh what if I uh freeze most of the parameters in my network and only train a very small subset uh so that I don't face all the issues that I face with fully fine tuning and at the same time I can miate some of these issues that I face with in context learning right so that is the whole uh cck of parameter efficient fine tuning so here uh as you see like if you have to train a QA model on top of llm using PFT then you would get uh a very incremental number of parameters compared to what the llm already has and that would represent everything that's needed to learn the task and same thing goes for summarization and same thing goes for classification and uh so so this is the Crux of p and uh so now the advantages are going to be like we going to borrow some disadvantages from in context learning and fully fine tuning and sort of uh put it here so of course like uh like I mentioned the first issue with fully fine tuning was the hardware requirements right so you require 12 to 20 times the number of trainable parameters uh the memory of that amount to sort of support training right so now if you reduce the number of trainable parameters of course you're optimize Optimizer states are going to come down the gradients are going to come down activations May remain there but a lot of the temp usage may come down and this reduces the amount of memory that you need drastically right so now uh so which means like you can use fewer gpus to train your model right and and one other thing is that since you're working at a lower parameter space and most of your world model is Frozen the model learns to sort of map World Knowledge to your task and then converge faster right so so this actually helps the model this also saves in GPU time like you learn models faster than what you would learn uh by fully fine tuning of course the obvious next thing is that uh uh Hardware requirements is uh is lower so which means not only less memory but also like many Institutions and organizations who started using gpus during the Deep learning era and now are slowly moving towards the llm era like they can still work with uh v00 uh Tesla Model or they don't necessarily have to have the aunds with 80 gigs or the hunds to sort of fine tune they can work with older versions of gpus with less memory and they can still get a lot of work done right so that is a very big Advantage because a lot of these uh older machines uh which consume less power are are idle right and this is one way of effectively using them and of course like I said earlier one of the issues with fully fine tuning is that uh you have such large parametric space pH to leverage during your learning process you end up overfitting and memorizing all the training examples so now if you don't have so much uh capacity to begin with then the likelihood of overfitting reduces a lot right again I'm not saying that overfitting goes away completely it depends on how many parameters you let the model uh sort of use during training but again the the likelihood of overfitting reduces considerable now in addition to that there is this phenomena called catastrophic forgetting right so cat iic forgetting is a is a process in general to any machine learning model uh let's say that you initially started to uh teach the model uh task a right so now now the model has learned task a very well and now you want to teach the model task B right so now how good would the model be in task B let's say you're working uh in a uh pre-m era let's say you're fine tuning but you you initially fine tuned for classifying uh let's say Finance versus uh Tech uh classification and now you want to teach it spam detection or something right so how many people believe that spam detection uh uh if spam detection was the task B how many people believe that task a would still be doing good after you teach the models task B right so it's not going to learn right so now imagine this with large language models it knows the World Knowledge right you have taught it so much during the the pre-training fine-tuning uh pre-training uh instruction tuning and Alignment stage right now you teach it a very simple task and it's going to forget everything that it it gathered so far like you don't want that to happen so so this uh most of the PF techniques uh are good in this uh type of transfer learning right they don't typically forget the World Knowledge which means that if you teach it you teach a task with a certain domain it is very likely to generalize to a new domain right if you teach it spam detection in say I Delhi now if you move to us where maybe the spammers are different or uh the type of language people use is different it is very likely going to work right it's not going to fail and finally of course the obvious thing is that it requires Less storage because the number of parameters that you store are less so obviously you only have to save the incremental uh weights that you've learned and you don't have to store the entire copy of llm again right so it's going to be there it needs just very little storage now with these advantages and the context of uh why we need PFT and why it is useful let's uh dive deep into a few uh popular techniques uh that people use for parameter efficient fine tuning right so first let's talk about uh the promp tuning or soft prompting right so so this was uh this was not one of the first P techniques but I'm but I'm starting with this because this is one of the easiest things uh and it's and it and it gives you and we can build upon that uh in the in the in the in the remaining of the lecture right so so here so typically when you do in context learning uh we refer to it as hard prompting because uh you write the prompt uh in the in context setup and and if you don't like the prompt then you change certain words or you rephrase certain things or you add examples so so this way of modifying the the the prompt uh or the in context uh setup is hard prompting because humans are involved in doing this right so what Lester at all thought was that uh why should we if you have a large amount of input and output available with us and it's easy for us to measure how good or bad the output is then wouldn't it be easier for the model to learn this prompt by itself right so all they say is that you reserve uh certain tokens in your input uh for certain special tokens called the soft prompt right and the only trainable parameters in your model is this task specific prompt right everything else in the uh in the architecture Remains the Same you don't change any other weights everything is frozen you only change these new uh tokens that you want to call it as soft promps right and and this is what their idea was and the good part about this idea is that uh for a specific task after you complete the uh the learning process all you need to do is save that soft prompt right which is going to be like orders of magnitude lower than what the llms would do right so let's say if you uh let's say if you're uh tokenizer uh so your embedding size is let's say uh 4K or something right if you use five tokens is very marginal compared to the size of llms that we have now right so so that is now like like we know that given the number of trainable parameters is so low it's one of the most efficient way of fine tuning right compared to full fine tuning now uh there is also another like really good advantage of soft prompting right which is uh the multitask serving like let's say you're hosting an llm uh and and you let people fine tune uh for various tasks right and and let's say that you have 120 tasks in your organization or for your project now during training uh you would have data sets for each of your tasks that you want to learn and so you will assign certain special tokens for each of your tasks and then you will start your training process right so for example for task a you end up training the yellow uh tokens and for task B you end up training the green tokens and your llm weights remain Frozen right and now the good part is during inference it's so easy right you just need to host the the base llm and and when you create batches to feed uh to your uh llm to predict the output you can just batch in whatever task you want based on what traffic comes in right so this is extremely efficient for serving because it may not so happen that the traffic for your llms are going to be that high right if you have 120 tasks right if it's high then of course you have to have separate uh you can have a lot of instances of llm running but it's easy to scale up scale down because you just need the same llm again so if the traffic goes down you can scale down certain hosting and then you can keep uh running them for multiple tasks at the same time so this is an this is one of the biggest advantages of uh prom tuning and uh this is why most uh llm hosting companies uh they provide you an API to promp T right because hosting is so cheap for them like even if you because they charge you per request right they don't charge you per time so their gpus are anyway running so it's going to be more efficient for them uh uh if if they support this type of prompt uning now so let's look at how does it compare with uh full F tuning right so the uh the graph in red or orange is full fine uning and and here uh the graph in green is prompt tuning and uh the graph in blue is uh simple prompt engineering hard prompting right and uh the x axis here uh is a number of parameters in the model that you're trying to find you right so uh so so right from 10 Power 8 to 10^ 10 are the family of T5 models so they have different sizes of T5 models that Microsoft had trained and highest one is uh like 11 billion model uh and the lowest one is around 700 million model or something right 770 or something and the and if you see there is an additional point on the right for uh blue which is prompt engineering and that point the last point which is about 10 power little more than 10 power 11 is gpt3 right as uh so that's uh gpt3 when this paper came out that was the best model possible for prompt uh engineering right at that point in time so what they see here is that uh when you have smaller models then it's better to do full fine tuning uh because it doesn't promp tuning or prompt engineering does not give you as much uh accuracy as what the full fine tuning gives but as and when you keep uh increasing the model size then prompt promp tuning sort of catches up with full fine tuning right so this is so again if you see respect respect of how much you do hard prompting there is a huge gap even for the 11 billion model so hard prompting is nothing but uh let's say that you want to do classification of spam or not spam right so now if I give you uh uh access to uh gpt3 gp4 in the web UI what would you do so you would give some instructions and then you would give some examples of uh what is spam and what is not spam right some input and output right so this is called hard prompting like let's say if this didn't work this didn't give you 95% accuracy then what would you do yes you would change the prompt right so that's a tedious process right so so that this style of uh creating a solution is what is called hard problem ing So Soft prompting is promp tuning which is like the model learns it by itself given a small amount of data right okay so now uh one of the other findings in this paper is that which is also slightly obvious is that uh if you increase the prompt length uh then it improves the uh the performance on the task right so this is uh as same as saying like if you if you give a very elaborate instruction right or if you increase the number of tokens you use for in context learning it is very likely the model would pick up lot more than what it would if you give it only one or two wordss to describe the task right so the essence is that way but the interesting thing they found is that if the number of prompt uh tokens uh cross 20 for T5 then they don't see any more improvements so it sort of Peaks at around 20 right so now uh one uh other uh thing is that uh so initializing what should be the weights of these soft promps to begin with is actually very important right if you give it very random initialization it may not converge to a good point or it may even take a very long time to convert right but if you sort of say uh create a prompt using words that are very similar to the very close to the task description and then take each embeddings and then post them there or you take certain words that are very Rel to the task and then post them there then that is better like I think that's also an Insight that this paper gives now like uh one thing that I mentioned in the uh advantages of theft is that uh it it sort of generalizes well to out doain uh data right so here uh they test on the task of reading comprehension specifically they take the squad data set uh where there is a uh where there's a natural language text which describes something and then there is a question answer following that right and this is based on Wikipedia and uh here the model is a full fine tuning prompt is a prompt tuning so there is hardly any difference between uh food fine tuning and prom tuning on the same data set that you trained the model on but if you see the models performance on the books data set which is like very different domain uh than the the Wikipedia uh you see that the model sort of generalizes much better the promp tuning model whereas a full fine tuning model does not generalize that well so they show a bunch of uh different domains some are close to Wikipedia some are slightly far away and and one of the biggest takeaways here is that the 12.5 Improvement uh on books was actually a very good Insight right okay so now that brings us uh to the next topic which is prefix tuning right so now uh now so then so the number of parameters you use for uh the p technique has tradeoff right if you use a lot of parameters uh then you very likely to learn very complex tasks right because your capacity is very high but your chances of overfitting also becomes high right uh and uh but when you if you have reduced number of parameters then your chances of overfitting is less but your performance can get affected right so you need some family of algorithms and then there's only so much you can do with g giv tokens right because if you give 20 tokens and it still doesn't learn there is going to be like very less likelihood that it's going to get better right so we can't just stop with uh prom tuning so then people came up with uh prefix tuning so prefix tuning is actually a contemporary work to promp tuning right both of them got published around the same time now the main difference between uh prefix tuning and promp tuning is that prompt tuning gives you those uh trainable tokens only in the input layer right basically wherever the whatever the embedding layer gives out is where you feed in the the special prompts right the embeddings of the special prompts that's the only trainable parameter uh whereas in prefix uh tuning you have a small set of trainable parameters in every layer of your Transformer right so so now uh if you look uh on the right so here this is the regular Transformer block uh where you have the multi-ad self attention fully connected layer and then there's a residual connection layer n and then there's again a fully connected network with a residual connection finally followed by a layer KN so this is what your regular Transformer block looks like uh but with prefix uh tuning uh so for now assume this whole thing is a single prefix right I'll tell you why we have this particular architecture here but assume this to have a single prefix now if you look at a decoder only model so basically prefix comes in the beginning so prefix don't influence each other but every word that you feed in the input get influenced by your prefix in every layer right so that's the sort of C of this uh approach now um so let's see why they have this soft prompt and a fully connected layer here like why do we need that like we could have just used uh something similar to soft prompts that we used right so why do we need this layer so so so let's say that uh we have a prompt sequence of let's say five tokens or something so let's call it P which is all the five tokens in sequence right P1 P2 P3 P4 P5 right and let uh uh mod P denote the number of tokens in that uh prefix right so so ideally what what do we want we want a function that converts that special token into sudden ambing which can then directly feed into the network right and these are the trainable the f is a trainable parameter right um so now F theta's Dimensions have to be that it's sort of like an embedding lookup for each uh prefix token you have you need an embedding associated with it right and uh this embedding since you're feeding it into the hidden layer the the eming should be of the size of what the hidden layers that you have is right so what the authors found is that if you use this particular uh F Theta to parameterize your prefixes then is giving it's resulting in unstable training so what we mean by unstable training is that let's say the weights may get changed so high that it sort of breaks the whole training process right your gradients hit the max or it hits zero or it goes too low right so so this results in unstable training so uh so this happens because sometimes uh that you are trying to learn something with very small number of parameters and you did not put it in the right place uh where the maybe certain there are certain reasons why we have layer n right so so you have to keep these in the right place so that uh you don't explode things or you don't underlow things right so if you don't keep if you don't architect Your solution in the right manner you will end up facing these training issues right so you'll have to change code have some hacks or or or change your architecture in such a way so that you don't undergo this unstable training process because this is a very uh risky thing to have in your network because let's say after you train for S days you like a GPU form of 100 different noes and then you suddenly you encounter nonstable training then whatever you did for so long is a waste right so this is a this a very uh this is something that we should always avoid when you design and what these guys figured out is that uh the number of parameters that they have and the way they have designed it is what is resulting in unstable training so what they did is that they represented F Theta with two functions basically F Theta Prime which is which has much smaller Dimension than the hidden Dimension right so so basically your embedding size is slightly lesser than what you had but your MLP is very large right let's say that the the F Theta is of dimension mod P cross let's say 200 or something right and then you will have 200 cross 1024 which is dimension so 200 cr24 is a very high uh large larger parameter space than what you would have used for f Theta but then F Theta Prime is much smaller so they found this to help in a much more stable training right so again what the advantage here is that during training we might need a large MLP and F Theta Prime but once you finish training you can Club both of them together and get back to the same size as what you have for f Theta right so this is just a hack for unstable training and I covered this part just to illustrate uh why design choice is very important right because you will end up having unstable training which is not something that you want to have in any llm training or F right so uh so for uh prefix tuning uh they they evaluated their technique on uh two different setups one is they used it for a task of text to text table to text generation uh where you have a like say uh in Wikipedia you have a table in the side about the bio and things like that right and on the right you have the introduction page introduction paragraph which mostly contains everything about what is there in the table so that's that's sort of an example of table uh to text but there are other variants of it so so they trained decoder only model uh at that quum it was gpd2 for table to text and then they trained B for summarization right and what they found is that by using only 0.1 percentage of parameters in these models they were able to get comparable results of full fine tuning right for each of the two models separately again the other uh the same thing that we saw for uh soft prompting is that prefix tuning also uh works well on unseen domain as well right so so let's move on to adapters now adapters was actually the first pep technique that got popular like this was in 2019 and after this came soft prompting and prefix tuning uh but I just covered them because it's much more easier to uh discuss right so in adapters uh what they do is that they add uh new layers inside each Transformer block right so and and these are called the adapter layers and uh each adapter layer uh has a fully connected uh layer uh which sort of projects uh the hidden Dimension down to a smaller Dimension and then there is some non- larity and then uh there is uh a fully connected layer that projects back up into the higher dimensional space so this is the the the overall architecture of adapters so so please uh notice here that uh there is a residual connection here which is the usual one as what we have in the Transformers but there is also a residual connection here within the adapter right so this is to ensure that if you randomly introduce new layers in your Transformers it's very likely you will break everything right because now you have random weights in between whatever you learned might go like might might go for a toss right so this residual connection is very important because this will ensure that if this layer is around uh like is let's say a gan initialize if the Gan distribution of a very small variance and zero mean uh then it's very likely that in the big in the first Epoch you would actually get the same output as what an llm would give without the adapter layers right so this way again your training will be stable yeah so like I mentioned uh there are two uh feed forward networks inside the adapter layer one uh uh sort of sorry here so this is the first part which is down projection feed forward Network and then you have a lesser dimensional space here and then again you project it back to a uh higher Dimension this is typically referred to as a bottleneck structure uh this is what Auto encoders also do right wherein you want to represent your input in a much richer dense dense representation uh and then uh you use that to actually represent the the the the input so this is sort of uh the same structure here uh so why do we need this bottleneck structure right why can't we just keep a single uh feed forward Network or some feed network with non neity right so the main advantage here is that this reduces the number of parameters significantly right so let's say that uh like your uh diens hidden Dimensions be 1024 right and and and now you're say down projecting it to a value M which is 24 so then in in in the beginning you might have like if you did not have this bottleneck structure you would have had 1024 cross 1024 parameters right but now you would have 2 * 1024 cross 24 right this is like orders again orders of magnitude lower uh than what you would achieve otherwise right and here again uh like I mentioned earlier before like with respect even with respect to different types of P techniques uh you will have a trade-off between performance and overfitting right so similarly here as well uh within adapters this value m gives you that trade-off right so you can use this as a hyper parameter train different variants of the model and see what works and what doesn't work right so so one of the uh biggest issue why adapter did not succeed that well is because like firstly it's performance like people found out better techniques that were much more efficient than them and gave better numbers as well but the other thing is that there's an inference overhead right so you're changing the architecture of the model in such a way right that it makes it difficult to remove things and add things let's say like uh if you want adapter a instead of adapter B that is already being hosted it becomes difficult and secondly since they are in between layers of existing Transformers it increases latency as well because you need more time to do you need more computation right uh so this uh is a is an issue with adapters but most of the other P techniques don't have issue with latency overhead right because they they sort of expanded along the sequence length rather than the the Transformers length right so that is an disadvantage of adapters so let's see so again this is a model that was proposed this was the first Model that was proposed uh for uh uh PFT so here uh let's look at the graph here so the blue line here indicates the uh fine-tuning top layers of uh B right so so here 10^ 6 would let's say indicate uh first K layers of B and and 3 into 10^ 8 would indicate almost all the layers in B right so so this point would be similar to fully fine tuning and and and this will be like as in when you leave out some layers in the bottom bottom bottom and finally you come to this point right so the reason why this was used as a baseline was because people were doing it with but doing this type of fine tuning with but even before adaps right you free like how you freeze word vectors or word to or glove embeddings in the bottom and you only train the network in the top people start doing this with b as well when B came out like let's not train the whole B from top to bottom because most of the lower uh layers are going to capture more word level features and and and and and more uh English uh language based features only the top layers are going to capture the task specific feature so people decided like I'll freeze most of the things on the bottom and only trained layers in the top so this was the Baseline then and uh this is the adapter performance and here uh uh again the xaxis indicates a number of trainable parameters and the good part here is that with only 3.6 percentage of parameters it is able to achieve as good performance as full fine tuning right because full fine tuning is here it sort of achieves that performance very early on in that right uh again like uh comparing this uh with uh uh let's say prefix tuning uh prefix tuning came much later but what adapters can achieve with just 3% prefix tuning was able to achieve it with just 0.1 percentage of the parameters right so so if you have a choice between adapters and prefix tuning you should always prefer prefix tuning most likely right okay so okay so this is the last part of the talk uh this is uh it's called low rank adaptation or Lowa this is the most uh for me this was one of the most interesting papers that I came across uh and uh and even the theory behind it is more grounded and it comes from a long literature of how people had worked from Deep planning era till now so this is a and uh this is one of the most popular uh P techniques that are out there and and you have ready implementations available in hugging phase which you can easily use for any LMS that you have right so so before going into Laura let's look at uh what people did what people how people ended up like like figuring out Laura right so there was this paper in 2018 that talked about uh something called uh rang composition in uh machine learning models right so here what they say is that so let's take uh this equation here right so here what we say is that let's say that your entire features is D dimensional all the trainable uh parameters in your model uh everything like don't think of them at separate uh attention layer or separate feed forward Network and things like that right just Club everything together and then create one big Vector like like insanely long Vector right so so ideally when you do fine tuning what are you doing you're going to take that Vector at state zero and they're going to use your examples compute law figure out something and then finally will come up with an increment on those weights right and then once you add these things up you end up getting a fine modu right this is what you do now what this paper said is that uh you don't for every task you don't need that many dimensions right maybe for some tasks you just need very few dimensions and the model will learn it right so what they said is that if you have a random projection Matrix okay this is randomly initialized and fixed okay we are not going to train it right now there is a certain D dimensional vector uh the the dimension of p is capital D cross small d uh and for a certain D which is much much much lesser than uh small D much lesser than capital D there exist a Theta D which will achieve same performance as Theta to of d right and this value D depends on what is your base model how capable it is and what is your task at hand right if you have a very naive base model even for small tasks you might need a big but if you have a very capable model and your task is simple you might need a very small d right so it depends on the combination of both so now uh they refer they they defined something called intrin intrinsic dimensionality uh what it says is the the value of small D that gets you 90% accuracy or performance as your entire full fine uning is called the intrinsic dimensionality of that particular task right so that's what they Define as intrinsic dimensionality now now this was in general made for deep networks right but then in 2021 uh people investigated this in the light of uh large language models and specifically for NLP tasks right so so while I I mentioned that LE at all what they did was they considered the the training trainable parameters as one long Vector right that did not take into account the structure of Transformers right because there is a repeated structure and there is a layer structure so maybe the structure will help you do things with much the the expressivity of such a uh repetitive Network may not is not leveraged in Computing the interesting dimensionality right so these guys uh came up with a with a way of measuring intrinsic dimensionality which takes into account this structure okay this is not the most it's not the best way of doing it but still they figured out that something is missing and we need to take it a little forward right now if you so the the is why they did that or they thought about in that direction is because it was almost impossible to measure intrinsic dimensionality for llms right why because let's say you want to do it for bir you need one terabyte of memory right because the projection Matrix because of the computational how you uh sort of uh do the computation and how you factorize the the capital P Matrix so even if you use the best implementation possible you still end up requiring a lot of memory right so they were not even able to figure out how to use it so that's when they came up with this structure aware technique uh what they said is that I will do it layer-wise right uh I will represent uh the Theta layer-wise rather than having the entire network if you have 24 layers then you divide them into 24 different vectors and uh and and and and and P the dimensionality of P comes down drastically now right uh and then uh there is also a scaling factor that they add for each update and and these scaling factors are different for different layers right and so if you add everything up the Lambda I and the thetas together they will uh you will get a d dimensional Vector right so the m is removed there because m is the number of layers we have and there is going to be a Lambda I for every M so the total number of parameters that you will end up uh training uh would be exactly M right um but again even this uh sort of is little difficult because the models grew insanely higher after bir right so now even if you use the same technique again you will end up running out of memory for the other case as well right okay so so here so so in the previous equation what we had is Theta D is the initialized value of the first the at the zero the what was Theta D plus uh a projection Matrix times Theta small d right so now what we are seeing is that we are going to uh split it for every layer in the Transformer right so so so that would give you so let forget about Lambda I for now uh if that is the case again the equation Still Remains the Same it's just that you will add a suffix to all the thetas and the capital D would be reduced by a big margin right because otherwise you let's say if you have 24 layers your vector will be uh like 24 times of what a single layer would be right and that's a and the projection Matrix would be instantly High because of that right what they found is that adding a scaling parameter there actually helps them like do some things easier right so they wanted to add a scaling parameter now if you add a scaling parameter you again increasing your uh space of the vector small d right Theta D should be D dimensional right now if you have D dimensional there and you have M additional Lambda I you get D plus M right so you don't want to do that you want to D to be the same right so the removed M from the there right okay okay so let's look at so this notion uh so initially it was called intrinsic Dimension now it's called structure aware intrinsic Dimension right so here what they say is that uh so they did uh this particular way of fine tuning the model and figuring out what is that D that gives you 90% of the accuracy that you would get with full fine tuning and and the and the the number of parameters in the model is listed in the x-axis right and the y- axis indicates the intrinsic dimensionality right so what we see is that when the number of parameters are small in the model the intrinsic dimensionality is quite high right as you increase the number of parameters there is a steady decrease in the value of D that is necessary to learn that task right so this sort of indicates that the more the better the more parameters you have in the model you have more uh domain or World Knowledge and the more World Knowledge you have you can learn a task much easier in a lower parameter space than what is needed right so this was a very interesting result uh from this uh 2021 paper now so let's see here right so here what they said was I have intrinsic dimensionality which is computationally very difficult to do and then I have structure a intrinsic dimensionality so there are two takeaways here right one takeaway is that having structure aware reduces the memory requirements and the train time by quite a bit right so you will able you'll be able to find tune much better if you take into account the structural aspects of your network when deciding what you should train and what you should freeze right the second point is if you take into account the structure of the network well right then the dimensionality comes down right so here for the Microsoft paraphrasing Corpus for this uh task uh that the dimensionality uh when you consider one whole Vector without considering the layer wise structure is 1,800 whereas for such a a it reduces to 1,600 right that the dimension itself comes down you need much lesser parameters to learn just if you uh if you carefully place what you should train and what you should not train right so so this was a brilliant takeaway from this paper right so now while this is a good this was a parallel effort people did not put in that much thought process into doing this for p like they were doing prom tuning they were doing prefix tuning because they felt like instead of doing it in the bottom layer you could do it everywhere people did adapters but they were not like giving it much thought about okay maybe we should try to reduce the intrinsic dimensionality maybe we should uh worry about what is uh what it needs how how do you control this and so on right so so that was the base for this Laura paper right so so so far you know taking into account the structure and then making use of the structure helps but how exactly to make use of the structure and how do you reduce the parameter space is what Laura brought in right so so let's look at regular fine tuning from the view of just weight updates in regular fine tuning you would have uh uh let's say pre-train weights right and then uh after uh training the model and backrop you will end up getting a weight update DW and then if you add W and DW uh you will end up getting the new weight uh which is going to be the uh fine tune weights right so now uh this can be thought of as you're just having two parallel uh networks one that goes through the pre-train weights and one that goes through the the recently updated weights right and then you can add them you will end up getting the same thing right so so what uh Laura said was uh they made two important contributions one is they told us what weights you should modify which will reduce intrinsic dimensionality by a lot right so having make use of the structure and they figured out this is the way in which this is the set of Weights in the Transformer block that you have to modify that was one contribution second contribution was how do we represent the uh the DW Matrix right so here so this is sort of inspired in the sense that adapters anyway did this right uh they they they reduced the dimension of the weight matrix by uh using the bottleneck structure so they using that bottleneck structure and uh so the so so basically the d w here is now factorized into uh ba one uh down projection Matrix here and there is an up projection Matrix here right and there's no nonlinearity right just two projections so now uh by doing this uh so so here ft indicates full fine tuning uh bitfit is a technique that was proposed where you only change the biases in the network and not change the weights right that's a tainable parameter uh PRM is the soft prompting pre-layer is prefix tuning adapters are uh the same the the hall spe adapters that I talked about now with different trainable parameters uh so what we see here is that uh Laura on slightly more complex tasks Viki SQL is natural language to SQL uh this is a natural language inference task which talks about entailment uh this is summarization data set so in all these slightly more complex data sets uh we see that for a very small fraction of the trainable parameters uh you end up getting even slightly better accuracies than full Fant right so uh so this was so this is the contribution of Laura now like I said they also told us uh what are the parameters in your model that you should update right and what they said is that only worry about query key value and the output projection Matrix right so so these are the only four weight matrices that you should worry about when you do Laura and and they also did a lot of ablations to show that if you keep the number of trainable parameters fixed right the rank so so in if you and if you only train the the query uh Vector the query Matrix uh then this is the accuracy that you get for uh SQL P SQL and uh the N task and Sly if you only train uh the the key Matrix and and a combinations or query and key here you reduce the rank because you want to maintain the number of parameters the same right and you do all combinations and they found that in most cases this is a good thing to do right so when you define your Laura config uh and when you try to find tune a model you will have these flexibility with you right you will be able to tell the tell the uh uh training code fine tuning code what uh matrices that you want to fine tune as a part of this Laura PFT and you can also tell it what rank you want right and you would be able to train accordingly uh now they also show show like the effect of rank so basically you can see that the model for for various different things the trend is similar that you sort of slowly raise up and then you saturate and then you come down slightly that's the typical Trend right it's uh and so here if it has not come down then it it's sort of oh it has come down already it has speak that uh r equal to 4 and has come down so here in some sense the rank here is sort of the intrinsic dimensionality right it's not exactly the it's not by the definition of what in dimensionality is but the rank sort of gives you a control of how much parameters are needed to learn your task right and uh and this is one of the most popular P techniques that everybody uses there is a slight variant of it called Kora uh which uh basically the only goal of Kora is to reduce the amount of memory needed to do Laura right and they uh and and this is one of the most popular techniques that are out there right so so before wrapping it up I just want to talk about uh some more training aspects uh which is uh the a matrix and the B Matrix are initialized in a certain way which will ensure stable training right and what they do here is that uh B is set to re zero for for very obvious reason which is if B is zero then AB will be zero if AB is zero then at the beginning of your Epoch you only have to go through W right so your model doesn't uh change its uh the way it works right you're not adding layers in between so it's parallel so so this helps in in keeping the uh in preserving the original behavior of the model now now a is sort of assigned using a gion distribution with zero mean and a very small variance and and and and and this will also ensure that the way B changes does not throw the model off right because B is going to change drastically now and if a is randomly initialized it may throw off and you would end up having the same issue as what people uh faced in prefix tuning so a is initialized in this way so that it doesn't throw off uh during uh fine T right so finally there are like insane number of extensions to Laura because it has been so popular people have explored all varieties of Laura QA is one of the most popular technique there is also long Laura in in case where you have really really long context length in your input uh there is Laura plus uh which sort of says that don't use the same learning rate for both A and B matrices in the reparameterization uh use different learning rates and how it will improve the speed with which you can converse right and there is also uh Dora which which says that with like since R is a hyper parameter uh you'll have to run your experiments in parallel with different values of R and figure out what works best but doraa sort of tells you uh how to sort of get the right R without running it multiple times so so so the the intuition here is that they want a DW which can be added to the weight w right so there is no so if you only look at it as a projection right so then there is no need for a nonlinearity there to mimic that update of that projection right for adapter is a different story Al together we not trying to add a projection to another projection right so here since you want uh the Lowa weights to be factorized and added with W you don't want the nonlinearity here so the advantage of doing this is that once you train a and b you can just fuse it to W right if you want a single model right again it will affect all the W's in your network and you you won't be able to uh do what you did with uh say prom tuning right because you cannot easily swap out and swapping right so there is also there are also works that are going on for making Lura as easily swappable as uh prom tuning but you but you have to do it at a batch level right because you can't you can't have multiple different luras in the same batch uh so there is a work around hot swapping in which the amount of time that's required to swap is so less that you won't feel that there is latency there for swapping but that's the intuition behind uh no nonlinearity in this particular reparameterization so so I'm not saying it's similar to all I'm saying is if your task is complex it is synonymous it's not you can't equate it intrinsic dimensionality is not equal to rank but it's synonymous to to Intrinsic dimensionality if your task is difficult and your base language model is also not that good then you need a bigger R right if your base model is good your task is difficult you might need slightly middle ground but if your base model is good your task is easy you can even get away with rank one right I think we are at the end of the lecture so basically just to summarize we covered uh in addition to introducing what p is and why we need P we covered four different uh techniques that are very popular and changed and with like sort of defining moments in uh this literature thank you la