Multi-Task Learning Basics | Stanford CS330 Lecture 2

[00:00:05]So the plan for today um as I mentioned in the previous lecture we're going to try to start with the basics and so this means that uh today we'll be talking about methods that were starting to be developed in the ancient times of the 1990s and uh then starting I think uh next week we'll get into more uh newer stuff that said even today we'll start to talk about a case study of a real real problem that's quite relevant today and is from a very modern research paper and by the end of the lecture the goals is to really try to convey what multitask learning is and what the key design choices are when trying to build these multitask systems in practice awesome so um let's start off with some notation so uh we'll start off with something like a neural network it could be a convolutional network like the one shown here or some other neural network and we'll denote the input as X and the label or output is y for example our input might be an image like an image of a tiger here and the label might be a classification of that tiger it might be an image classification problem or maybe something more interesting like trying to classify what to do if you see something like this and alternatively instead of being an image it could be something like the title of a paper in this case you probably wouldn't use a convolutional network you would probably use something like a Transformer if you're trendy or if you're a bit older maybe an lstm and the label could be something like the length of the paper and so in this case uh if it was the length of the paper this might be more of a regression problem rather than a discrete classification problem now we'll refer to the parameters of the neural network typically with Theta and so this will correspond to all of the parameters of that neural network you can think of it as a vector that basically flattens each of these weight matrices and appends them into a single very large Vector this may have millions of values in this vector and we can then refer to the function represented by this neural network as F which will give us a distribution over y given the input X parametrized by Theta um so this should follow fairly standard notation that you may have seen before now in single task supervised learning we will be given some form of data set which has input output pairs so a number of examples of images like the tiger and labels like the options on the right and then we will Define a loss function that tells us how good is that model at performing that task and our goal will be to find the parameters that minimize that loss function so try to find the parameter setting of one of these neural networks such that we do well on a classification problem for example um and so a typical form of this loss function might be something like negative log likelihood this would look something like this where we're measuring the likelihood that F assigns to a given label given X and then negating that because typically loss functions are things that we minimize and then trying to minimize the negative probability of the label given the input uh and this kind of Vlog likelihood loss this is equivalent to something like a cross-entropy loss or a mean squared error loss um that you may have seen in uh in other machine learning courses cool um so that should mostly be review for folks um now how do we go from single test learning to multitask learning um for that we need to figure out actually what is a task what does it mean to be a task we defined what a task is in the previous lecture but we're going to cover a task more formally this time and in particular in the context of this course we'll think about a task as a set of three things a distribution over x a distribution over y given X and a loss function and so you can think of these as the distribution that generates the data and the reason why that we're gonna We want to Define it something like this is that we want to be able to say whether or not the network is doing well at a task and doing well at a task is a little bit different than doing well on a data set and uh in particular you could have something that does really really well on a training data set but doesn't actually generalize to other examples for that task and therefore by defining a task as the data generating distributions and the corresponding loss function for that task we can then um capture Notions of how well it's actually doing on that task in general so we can have corresponding data sets that are sampled from these underlying distributions like the training set and the test set and note that here I'm using I to like index into the task so for task I it has these two distributions in this loss function and um and it has data sets that are sampled from these distributions um in practice you won't have any any sort of access to these ground trees data generating distributions you'll generally only have access to the corresponding data sets um and then also in the future slides I'll generally use di as shorthand for Di train um just because typically we will refer to the training data set a lot and it can be convenient to drop off the the TR uth Okay so that is how we'll Define the task and now let's look at some examples of different multitask learning problems so generally a multitask learning problem is one where we're our goal isn't just to solve one task but to solve a set of tasks and the tasks could vary in different ways so in something like multitask classification the loss function will probably be the same across all the tasks it will probably just be the cross-entropy loss and for example this could be something where we want to be able to recognize handwriting in different languages so each language will correspond to a different task uh and you could have a data set that looks like this this is actually the Omni glot data set and you have uh different alphabets or different languages and so in this case you're going to have a different P of X distribution a different P of Y given X distribution because the characters themselves will look different and also the label given those characters will be different for different languages but the underlying loss function will be the same because it's all still a classification problem as another example of this kind of multitask classification problem you could also have a personalized spam filter where different tasks correspond to different people and different people will receive different kinds of spam and they will also have different preferences for what is spam and what's not spam and so it will also in this case have a different distribution over X and a different distribution over y given X but yet again the same loss function so um that's multitask classification we could also consider a scenario where both P of X and the loss function are same are the same across these tasks and the only thing that differs is y given X in a case like this you could think of face attribute recognition as an example of this where one task is maybe to detect if someone has black hair or brown hair or blonde hair or white hair and a different task is to predict their eye color um in this case all the images are the same you just have different distributions of your labels so you can the question was it can mult can image net be considered multi-label learning and I guess the thing that really differentiates multi-label learning here and something like image.classification is at least the way that imagenet classification is framed there's only one label that is correct um and the thing that's different here is that um you actually have a different set of labels and so for example someone can have both brown hair and brown eyes or they could have um blonde hair and brown eyes for example and so you actually have different sets of labels itself and that's um that's like the key differentiating factor and so in general it's something like imagenet classification would be considered a single test problem that said you can also frame things like imagenet as multi-label problems because oftentimes there are actually more than one thing in an image and you may actually want to classify all the things in the in the image rather than just one thing all right another example of multi-label learning is something like scene understanding where you have an image you have images of lots of different 3D scenes and one task is to predict the depth one task is to predict key points in the image and another task is to predict the surface normals in in that scene and so this is again an example where actually all the images in the data set are the same the only thing that's different is the different labels um so these are a couple kind of instantiations of multitask learning problems um there's also scenarios where the loss function might vary as well so in both of these examples these are settings where the loss function isn't changing but you could also have scenarios where for example some of your labels are continuous and some of your labels are discrete and you use a a mean squared error loss function for the The Continuous and a cross-entropy loss function for the discrete labels or you might have multiple metrics that you care about and you want to optimize those objectives simultaneously cool um so the pause here is there any questions on kind of the the setup before we get into actually solving these multitask problems yeah so for the last time he said like the task should share some structure so that should actually mean my expectations here are getting offensive if they are different languages yeah those are great questions so the question was um last lecture we were talking about how the task should share some structure does that mean that they should share the loss function or share let's show you like one of these three things and what I meant by a structure in that first lecture is something a little bit more abstract so these are three very concrete things and you can actually have tasks that differ in all three of these things but still have a lot of common structure um and so I guess structure is something that's a little bit more abstract we can think of we'll come back to a little bit more to what structure is when we get to some of the Bayesian perspective on it um but I guess as one example uh you could imagine something like uh the per language handwriting recognition these tasks have a lot of shared structure intuitively in the sense that they um recognizing characters is uh implicitly about recognizing the shape of the the handwriting but you could also have a version of this that has different loss functions because maybe in one case one of your loss functions is is more continuous because maybe some of the digits are some of the characters are actually like numerical values rather than characters and something like that you may actually have different loss functions while still having a lot of shared structure cool um so now let's get into actually learning networks that solve multitask learning problems so the first thing that we generally need to do in multitask learning problems is to tell the neural network what the task is and we'll do this with what I'll call a task descriptor which will denote as z i and we'll pass this into the network in some way or another and so the function will no longer be modeling y given X but actually modeling y given X comma z i um and so let's kind of actually better understand what this task descriptor might be so say that you're a very diligent grad student and you were assigned a bunch of papers to review and you want to understand how long it's going to take you to review these papers so you may be one of your tasks might be to take the title of the paper and predict the length of the paper the second task is maybe maybe you want to get an initial summary of the paper before reviewing it and so you the second task is to predict a summary of the paper uh and maybe the third task is uh you're getting maybe a little bit too lazy and you just want to write the paper review for you so we have three tasks here and the task descriptor in this case could be a few different things um the first thing is it could just be a one-hot encoding of the task index and in this case what this means is that we just have a one hot Vector one hot Vector is just something that uh looks like this where we're encoding an integer in um in Vector form where for example the integer one will be denoted with a one in the first position and zero on the other two the integer two will be denoted as with the one in the second position and for the third task the task descriptor would be something like this um so this would be an example the simplest possible way that we could tell the network what the task is so this would be telling it that the task is the first one the second one and the third one but we can also do something a little bit more creative so we could for example give it a language description of what we want it to do so we could tell it give me a summary tell me the length of the paper Give me a review um and so Z could actually be a natural language string of what the task is as well so some other examples it really could just be whatever metadata you have about the tasks so it could be um not really in this example but if you have an example where different tasks are different people they can correspond to different attributes of those users um if you have an example where like this one where you have some sort of natural description of what the task is then you could condition on that um you may also have some domains where you have a more formal specification of what the task is and you could also try to pass that into the network to tell it what the task is cool um so now that we've defined now that we've told the network what what the task is we can also formulate the objective so uh the basic way to formulate the objective is actually what we covered in the previous lectures we just sum up the loss functions for each of the tasks so Li here is Computing the loss function of the network on the training data set for that task then we'll sum up over all of the t-tasks that we have and try to minimize the loss function minimize the parameters of our neural network um over the sum of loss functions yeah very much the same type same as like prompting the model uh like the recent work that's been done yeah so you can think of the task descriptor as a form of prompt you typically wouldn't think of something like this as a prompt um but yeah it basically can be a prompt or it could be something a little bit more basic the um a set of XY Pairs and so what exactly were you asking so you'll have different data sets for different tasks in in general so um as we talked about before the each task will be defined by its own data generating distribution and then the training data set is drawn from that that data generating distribution you may have cases where kind of P of X is the same across tasks and so in that case all of the x's in your data set may be identical um but then then even in those cases the Y's will be different for different tasks and in general these training data sets will be different across tasks does that answer your question is foreign you're asking if the data Center previous um so I mentioned here that for example the X the images might all be the same across tasks um so in this example uh in scene understanding the images will all be potentially identical across tasks but the labels will be different so the labels for the depth task correspond to the depth annotations the labels for the normals tasks correspond to something like this and so forth yeah um you mentioned a task to step first um there are an awful data sets where you don't get the task description what happens then yeah so you might have a data set where you don't have the task descriptor in that case you can just um go with the basic kind of one hot encoding um assuming that you at least have some sort of differentiation between the tasks it's also possible that in some cases you might be able to tell what the task is just from the input if if P of X is different across tasks um you might just be able to tell without a form of task descriptor but in general um in general in multitask learning we're going to assume that we at least have some separate data sets for each task [Music] yeah so we'll talk about that next cool so I think that we can just transition um into the next part so this is the um kind of the basic setup but there's still a lot of design choices to make here we needed to sign the model the objective and the optimization process like what was just asked we need to figure out how do we actually pass in zi into the network we also need to figure out um if we should use this vanilla objective or if we can use something a little bit more sophisticated and also how do we go about optimizing this objective so we'll talk about each of these design choices one by one uh and we'll start with the model so we'll start by trying to think about how can the model be conditioned on zi and what parameters of the model should be shared across tasks versus separate cool um so now let's assume that zi is a one hot index like what we talked about before um I have a question for for all of you which is um how should we choose to condition the network on this one hot Vector in a way that we'll try to share as little as possible and what I mean by this is we want to have as few of the parameters for different tasks be shared we want to basically get as close as possible just training completely separate neural networks on each task does anyone have any thoughts yeah all right finding good loss functions can you explain a little bit more yeah the question is um if we assume that zi is just a one hot Vector for each of the tasks um how should we condition the network on zi such that the network is sharing as little as possible such that we are as close as possible to just training completely different neural networks yeah I guess we could just have like a switch statement for each of the different conditions so if it is if 1001 that do a separate Network program yeah exactly so um in particular what we can do and what was suggested is you can basically just have different neural networks uh two different neural networks and basically have a switch statement have Z basically modulate which of those networks you use to make a prediction um and so formally what this would look like if Z is a one hot Vector is you would pass say y1 is the output of the first Network Y2 is the output of the second Network and so forth and all of these have separate parameters then we can compute the output as basically just switching between these different outputs and so if the task is the first task then you would output y1 if the task is the second task then you would output Y2 and so forth and what you get as a result of this is that you basically this is still a single neural network but you get completely independently independently trained neural networks within the single neural network was there a question um and so essentially this corresponds to having no shared parameters across the tasks yeah [Music] and then it's performance across all it does with as many shared parameters as possible so that accommodation yes exactly so in general um this is not a great way to go about doing multitask learning um it's kind of one extreme and on kind of The Other Extreme of the problem we could share a lot more we could share basically everything and so what we could do instead is to have a single neural network and basically just um for example concatenate Z with the with one of the layers and in this case if you just concatenate Z to one of the layers and then have everything else be normal then um then in this case basically all of the parameters are shared across the tasks um with a small exception of the the kind of the parameters following zi technically will not be shared across tests but all the other parameters in the network will be shared um so you have these two different extremes and there's becomes this choice of um what what well what should you actually do in practice um and so an alternative way of viewing this this form of conditioning on zi is basically splitting the parameters into shared parameters into specific parameters so um if you split it the objective into shared parameters into top specific parameters we saw one example where everything was basically top specific we saw another example where everything was shared you can basically formulate rewrite the objective as something like this which is exactly equivalent to the previous objective the only thing that we're doing is we're now just saying that we um we have uh we basically split up our parameter Vector into these two parts and the thing that's useful about writing it this way is that it really shows us that the task specific parameters for task I are only optimized with respect to loss function I and so um those parts of the network will only see data from one of the tasks rather than all of the tasks yeah like for example and like uh the MLP space if you just train a large language models yeah that's an awesome question so the question was um is there a constraint that in multitask learning that we're going to be learning all the tasks all at the same time and in general for the purpose of this lecture we're only going to be considering things that we're learning at the same time in the next lecture on Monday we'll start talking about transfer learning where we learn one task and then learn another task um and then the last lecture in the course we're going to also talk about lifelong learning where we are learning a set of tasks in sequence one task and another test and another task and so forth um in general a lot of the underlying ideas underlying multitask learning are also applicable to the setting where you're not learning everything at once um and one thing that is somewhat nice about looking at things like this is that you could imagine training on a few tasks um and that gives you maybe some good shared parameters and then additionally training some task specific parameters separately after the fact um and so things like that uh are are often very reasonable to do yeah my class learning is understanding are the links are good and they make payments like method is good is like to speak about immediately do something which is sharing is [Music] yeah so the question is um it seems annoying to have to manually break this up and figure out what should be shared and what shouldn't be shared can we just have it like have an algorithm figure it out for us uh and so there are some approaches that do something like that um in general it's it's um is somewhat of a chicken and egg problem because if you have something else choose what to share and what not to share then that thing that's choosing will probably use data from all the tasks and so that's going to be shared across all the tasks so um in general it's there is some there are going to be some manual choices regardless um but there are some some techniques that do something like that yeah yeah so the question is um do we know anything about the kind of the comparative rates of of different task learning I'll briefly talk about that when we start to talk about the optimization process so the next thing that I want to talk about and specifically thinking about breaking things into shared parameters and task specific parameters is that in the previous two slides we saw one extreme of sharing nothing and we saw one extreme of sharing everything just based on how we condition on Zi and one of the things I think is interesting here is it suggests that basically choosing how to condition the network on zi is equivalent to choosing how and where to share parameters um and so in general um choosing how to condition on zi is actually a very delicate choice because you need to be careful about how much you should be sharing versus how much you shouldn't be sharing um so we looked at two extremes of of conditioning um I want to go over two other common choices in conditioning well actually sort of three other common choices so we talked about just concatenating um the zi with the activations at one of the layers and so this looks something like this where you take the the input or the activations at one layer you take your zi you concatenate them together and then pass that into the network another thing that you could do instead of concatenating is adding together a representation of both of them so you could pass a z i through a linear layer and pass the input through a linear layer and then add them rather than conditioning them um and then get the resulting output now one thing that you might notice here is um these are two options that seem somewhat different but it actually turns out that the concatenation-based conditioning here and the additive conditioning here are exactly equivalent to one another um and so I'm curious maybe it's one thing I want to try something new so typically I ask people what why they're the same thing rather than just asking you I want you to take Maybe one-ish minute to think about it and then one-ish minute to talk to your neighbor about it and then after two minutes I'll ask you um I'll ask you to share why you think they're the same thing so you can think about it for a minute I'll tell you in a minute to talk to someone and yeah foreign okay um cool let's come back um does anyone want to share what they came up with [Music] yeah awesome so in particular if we have some some input X and some task descriptor Z we're gonna uh if we concatenate them we'll do something like this and have a weight Matrix right here and you can instead think of this weight Matrix as having two parts W1 and W2 um so this is just the left half of the Matrix this is the right half of that Matrix and this is equivalent to W1 times X Plus W2 times Z and so one thing that I one thing that's kind of important in the second figure is that actually the X and the Z are first going through a linear layer before you add them together um and uh this is exactly kind of the additive conditioning version and you can kind of visually see this like right here where the red Matrix corresponds to W1 the blue Matrix corresponds to W2 and yeah yeah when you're aware of the additive conditioning which you have with the network could have like more expressive power and would possibly additive conditioning be better than that is um seriously if you had a non-linearity uh basically here so if you had something like a non-linearity here basically um yeah so something once you add non-start adding non-linearities it does get more expressive um the and yeah so if you did do have some non-linearities on these separately it would be a little bit more expressive um you could also Imagine yeah so in that case it would be a little bit more specific yeah is there any computational trade-off between the two um in general I think that uh I would guess that on well so I would guess that on Modern computers um this is better because we have very good Matrix Vector multiplication modules um but I haven't tested it myself on Hardware and I guess I would guess it's probably not that significant compared to other layers of neural networks like convolution layers [Music] um I guess yeah I guess in both of these cases in practice you probably won't just be like having this literally be the output of your network you'll probably also continue to pass that through fully connected layers and so forth um and so in general um when we talked about adding non-linearities there I'm guessing that wouldn't have a huge effect in terms on the expressive power because this is a part of a larger Network cool um so that was concatenation and additive conditioning um two other choices that are quite common one is to use a multi-head architecture where you have some shared bottom layers that are every like all the tasks you pass in the input through the same exact layers and then for different tasks you have different heads or different sets of task specific layers and a generalization of this that you can consider is multiplicative conditioning so before we saw additive conditioning you can also multiply the task descriptor or representation of the task descriptor with the activations and so what this will look like is basically just replace the additive um the addition operation in the previous equation with the dot product or with a sorry with an element-wise multiplication and something like this is uh is going to be more expressive than at least per layer than adding and it can actually it could actually represent things like the multi-head architecture um and the reason that we can see that is if you remember the kind of very first example that we looked at where we're essentially multiplying the z i with the with the outputs the sort of multiplication can sort of gate the network and create um I basically modulate uh which layers are used for which tasks cool um and so yeah in general multiplicative conditioning is going to generalize having independent networks and independent heads yeah um tension is sort of like a task um so insofar his attention is basically like dot products um then then yes uh something like attention can be uh viewed as task specification um any other questions yeah yeah where you where you multiply will matter um although that said you could also Imagine like even if you do this sort of gating at the very beginning of the network um you could imagine well I guess yeah even with fully connected layer well it's yeah the architecture will generally Still Still matter um and so it can represent certain kinds of gating and uh but yeah the architecture still matters [Music] um so the question is if you do this sort of multiplicative conditioning on every single layer does that let the network sort of figure out what to share and what not to share um to some extent yes I guess I should also mention that even if the network is sharing all the parameters and it's getting gradients from all the tasks for all the parameters you could also imagine it sort of zeroing out certain certain parts of the weights so that it represents something more like independent networks and so the the network can arrive in a setting and where where some things are being used for only one task um even without um even without like a multiplicative conditioning like thing cool um so we covered really the basics I mean the basics are really either concatenation slash additive conditioning or multiplicative conditioning um and you can have more complex choices there's a lot of papers that consider more complex choices although even even just um the basic approach typically works pretty well now unfortunately figuring out how you structure the architecture and how you condition on the network is um is a lot like just general neural network architecture tuning which is that it often is fairly problem dependent and it's a little bit more of an art uh than a science and it's oftentimes guided more by Intuition or knowledge of the problem as opposed to having like a really rigorous set of guidelines for exactly what you should do um that said we'll talk a little bit um in some of the coming slides about uh some things that can help guide that uh that process yeah that we're using so if you're using like a natural language condition on that differently um and also do you have any insights on like what sort of descriptors might be better like is the one hot better because it's more explicit or could knock your language would be better includes more like nuanced yeah so there are two questions there one is what short descriptor should you use and the second is is the way that you condition the network on that descriptor differed based off of the kind of descriptor that you have so in general uh for the first question the more information you give to the network typically the better if you give it just a one hot Vector these one hot vectors are orthogonal to each other and so you're not um you're not giving it any information about how these tasks might relate to one another and if you instead give it a language description of the task where one of the tasks is write me a story another task is write me a poem another task is translate between these two tasks or between these two languages then that will give it a little bit of information about the similarity between tasks because write me a story and write me a poem those are similar sentences and so naturally the task should be a little bit more similar as well um so generally the more information you give it the better if you have access to that information uh and then in terms of conditioning um I think that my my general advice would be that multiplicative conditioning is generally what like the way to go because it gives you more expressive power uh and in practice we've seen things like attention and multiplicative conditioning through like feature-wise modulation to be the um at least one of the approaches that you see the most in um in what people do has there been any work done on my task embedding to accelerate how we have word embeddings if you could have a task embedding so it's not just like one more factors yeah so um the question is like can we is there any work on getting a task embedding um and I should note that if you um if you have a weight Matrix that's going after a um after a uh a task descriptor then this is going to convert that one hot Vector into a dense vector and so in that sense this weight the first weight Matrix that comes after that one hot task descriptor is going to give you an embedding of the task and it's essentially if you learn multitask learning from scratch then it's going to learn these task embeddings from scratch but it would be interesting if we could develop things like the notion of word vectors but for tasks um and one thing you could do is if you have a natural language description encode that into a sentence encoding and use that representation as your task descriptor yeah the question is how do you implement the multiplicative game so if you have four tasks do you just add a soft Max gate later over dimension for out of the last layer yeah so um the way that you can implement the multiplicative gate is going to look a lot like this so um if the say the um say the dimensionality of one of your activations is um is uh like d dimensional then what you'll want to do is take your one hot Vector also multiply that by a weight Matrix so you get another D dimensional Vector um so once you have two D dimensional vectors then you'll just do element wise multiplication uh and so you'll replace this this plus sign with a element-wise multiplication operation foreign cool um so we've talked a lot about the architecture the model um now let's talk about the actual objective so earlier on WE formulated this multitask this vanilla multitask learning objective but in many cases we may want to weight the tasks differently so we may want to formulate an objective that looks like this where we are going to assign a higher weight to some tasks compared to others does anyone have any thoughts on how we might choose the weights yeah perhaps uh how many times you see data for each tasks so stop that you see more lesser so that it's not imbalance yeah so you could have something where you weigh um you would change the weight based off of the amount of data that you have maybe if you have a lot more data it could actually make sense to down weight if you have less data it may make sense to up weight um one thing I'll mention here is by formulating this objective as a sum over tasks this is already going to somewhat normalize over the amount of data per task um because uh rather than if we instead summed over the data points that we had then that would assign higher weight to um to tasks with more data okay yeah absolutely so if you have some loss functions that are much higher in magnitude then you may want to down weight those and up weight loss functions that have a lower magnitude yeah MTL and you see that the Network's really struggling on a specific task then you take away the test quicker yeah so if you find that the model is uh doing poorly on one task then you could try to upweight that task and we'll actually cover a method that will do that automatically later on this slide any other ideas yeah so there's some tasks that you care about more than others like in some cases maybe there's actually only one task you care about and you just have these other auxiliary tasks that you are hoping might help out then you can up weight the tasks that you care about the most visions yeah so you could in some ways treat the WIS as hyper parameters as well although when you choose those hyper parameters you need some overall objective to tune them with respect to and so uh you also and for example you could that overall objective could be the vanilla objective but you may also have cases where the vanilla objective is not suitable like when the magnitude of the losses are different or when some tasks matter more than others one more function to help a consisting gun when it has when someone is going to stuck in a local member or maximum you have a less weighted objective than them kind of kick it out so it'll take a little bit more weight [Music] yeah so you may if you run into some optimization challenges it could be that actually changing the waiting AIDS and the optimization challenges for example maybe if you have a task that seems to be stuck maybe if you start planning placing all of your weight on that task it will help push it out of that kind of that local Optimum or something or maybe you could if it's stuck on that task maybe you should actually stop optimizing and revisit it later so um there are a number of different approaches that you could take um the first thing that I have listed here is actually just based on some importance or priority which would be some sort of manual selection um but there are also various heuristics that you could use to uh to choose these weights as well um and the other thing that I'll mention is you don't have to have these weights be fixed throughout training you could actually have them vary at different points of training for example if you have optimization challenges or if some tasks are doing worse than others in the optimization um and so in addition to some of the the things that you have also adjusted and other heuristic that some prior work has looked at is encouraging gradients to have similar magnitudes um that said there's a pretty large body of work on different heuristics that I've looked at different different ways of approaching this in general the vanilla objective or manually chosen weights is generally one of the strongest approaches that you can take but there has it's worth acknowledging a lot of work that on certain problems you can't see improvements now the other thing the other approach that I want to mention here which actually came up before is you could optimize for the task that is doing the worst and in particular you can formulate this as a Minimax optimization where at each point in training you pick the task that has the highest loss and you update the parameters on that task and this is exactly going to um basically try to normalize or equalize the task to some extent and this is relevant when you think that all of the tasks matter equally um and so um in particular kind of what this will look like is if you have um if you have task one task two and task three and you plot their loss value and for example for task one your loss values down here for task two your loss values up here for task three your loss values right here what this will do is um you're going to estimate these loss value at your current iteration of training you'll notice that this one is doing the worst and then you'll start only optimizing on task two and then after you start optimizing on task two you'll reevaluate this uh chances are hopefully this has gone down a little bit maybe one of the other ones has gone up a little bit because you weren't optimizing on that one and then you'll start optimizing for this one um at the end of this process in general you should end up with loss values that are more similar across the three tasks compared to if you would only optimize the sum of them because if you optimize the sum it might just prioritize it might minimize the tasks that are easiest rather than trying to maintain equal value uh and this can be especially useful In fairness settings where maybe different tasks correspond to different users or different demographics or different subpopulations or different geographic regions and in those settings you want to have similar loss values for those different subpopulations because you don't want to have some customers that are getting a really great experience and some customers that are having a really terrible experience or likewise some people that have a great experience versus a terrible experience yeah so this in general becomes a harder optimization problem there are also um yeah so there are a range of challenges with this kind of approach um another thing that's somewhat challenging is you need to compute what is the worst loss um and it the way that you would optimize this exactly is every single iteration you would compute what is the worst one and evaluating the loss function on your entire data set may be expensive but there are ways to um to approximate uh which one is the worst one by keeping like a running average or something like that um and in practice it's not something that is too hard to optimize especially if you have a relatively small number of tasks you have a lot of tasks that gets a little bit trickier yeah so in practice um it is good good practice to normalize your labels like your um your uh yeah normalizer labels and and make sure your loss functions are all on the same scale um if you don't do something like that or if it's difficult to do that what this is going to do is prioritize the loss functions that are the most difficult or that are the highest in magnitude yeah so my question is about opposite or outside if you wanna so let's say if her objective is on a primary that's great I mean if you want to optimize our friendly desktop I mean many of times we use multitask learning so another question is how do we find our Optical how do we configure the objective function to decide on what auxiliary the auxiliary tasks would be helpful for advisory yeah so if you really only care about like one of the tasks for example um then in that case actually treating wi as a hyper parameter makes a lot of sense because your outer objective is I want to do as best as possible on task one and I want those other tasks to help me and so um you can manually figure out what wi leads the lowest loss function on the validation set for task one or you could apply automatic hyper parameter optimization techniques as well to do it for you yeah I was wondering so that this type of um Trying to minimize the maximum loss um it kind of reminds me of like L1 versus O2 versus alternating Norms um so it's possible to or have people try to change the exponent for for the loss like instead of just doing uh wait times loss because you lost squared and it's kind of like the L2 Norm of putting everything down and I was going to be kind of similar to the mid-maps yeah actually that's a great question I haven't um I haven't come across other like things that aren't uh L2 or L infinity or or L1 like something that does some other exponent basically uh but that something like that could be interesting to think about and um one of the challenges with this kind of objective is sometimes it can be a little bit too pessimistic and can place a little bit too much focus on the worst case uh and something that is somewhere in between L2 and L Infinity may actually mitigate that challenge so that could be interesting to explore um one last thing that I'll mention um if you're interested in digging more into this this looks a lot like what's called distributionally robust optimization or um Dro and so if you want a keyword to learn more about it you could take a look at that um and the math gets quite deep you can actually like um there's a lot of nice guarantees that you can get about this kind of optimization great um and then lastly we'll briefly talk about uh the optimization process itself before going into a case study uh and I'm just going to go over the standard optimization process that we might do for this objective because um in general uh in general it works pretty well uh and so the basic version of this is we'll sample a mini batch of tasks so I mean if we only have three tasks then we can just sample all three but if you have a very large number of tasks you might just sample a subset of those tasks then we will sample data points for each of the tasks that we sampled so this is going to be another mini batch and then we can compute the loss on that mini batch so we have each of the tasks in our batch of tasks and then we have the data set the mini batch data set for that task that we computed and this will correspond to kind of a mini batch loss function in the multitask setting and so once we've computed our mini batch loss function then we can compute the gradient of that loss function and back propagate that into the neural network and apply the gradient with your favorite Optimizer so you could use vanilla stochastic gradient descent perhaps with momentum or you could use something like atom which is often used in practice so this basically just corresponds to stochastic gradient descent on the multitask objective the thing that's probably the the most different from that is we are going to be sampling a mini batch of tasks and so this ensures that the tasks are sampled uniformly regardless of the quantities of data and so if you have a lot more data for one task than another task this is going to make sure that those two tasks are still sampled at the same rate of course if you care a lot more about the task that has more data then you may actually want to do something a little bit different than this first step um the other thing that came up is that the um you may actually have loss functions that are at different scales and if you have a regression problem even if your loss function is the same if it's mean squared error if your task labels for different tasks are at different scales like one varies from negative five to five another varies from negative 100 to 100 then your loss function is going to be scaled correspondingly and so it's good to try to normalize your labels so that your loss functions are at the same scale cool um so actually before we go into the case study um there's a few different challenges that I want to bring up and this is going to affect some of the design choices as well so one challenge that comes up is negative transfer between tasks and what negative transfer means is that sometimes if you try to train with multitask learning the resulting model actually does worse than if you were to train completely independent neural networks um and as one example of this you can formulate a multitask version of The cfar 100 data set and you can compare the performance of a multi-head architecture of a cross-stitch architecture which was an architecture proposed by some prior work and also just independently training the models and what we see here is actually the independently trained model is getting 67 accuracy which is more than 10 percent higher than the multitask learning approaches um so why might this be the case um this could be the case because of optimization challenges so there might be some interference between the tasks they might be trying to use their representation in different ways or the test might be learning at different rates and if one of them one of the tasks is more or less converged and one of the tasks is still learning that may make it difficult to um to keep on updating the network um it could also be not an optimization challenge but just a challenge of limited representational capacity uh oftentimes multitasking networks are doing more and so they need to be larger than if you were to just train on a single task so um if you have a negative transfer uh the natural thing to do is just try to share less across tasks and so you can see if you're having negative transfer by just trying to train independently if you're seeing that independent training is doing better then you can just try to make it more like independent training um and so we saw a few different ways to try to do that before um by for example having a multi-head architecture that explicitly has different parts of the network that are not shared at all between the tasks [Music] yeah so the question is are there any unsupervised methods for telling if um if there's going to be positive or negative transfer um between the tasks in general um I guess I'll show that in a little bit in a minute um but in general you certainly without the labels of the task you certainly can't um it's certainly very difficult to tell if there's going to be positive or negative transfer ideally it'd be awesome if you could just have a description of the task at each of the tasks and then something that tells you like are these going to work well together or not uh unfortunately I think that something like that is nearly impossible because it's not just going to depend on what the task is but also the nature of the data set the nature of the model that you're training potentially also the nature of the optimizer as well so um in general these sorts of things are very hard to tell a priority yeah if you are getting negative transfer and you see that sometimes yeah it's a great question so if you're seeing that independent training is doing better than your current multitask model why not just go with the independent networks um it's it's very reasonable to kind of stop there and go with the independent networks although there are definitely scenarios where you could still you could get better performance with a different architecture for example if you started by trying to share everything there are certainly scenarios where training a multi-head architecture will do a lot better than both sharing everything and training independently how do I tell them um so the question is is there a way to tell if there's like if the tasks are just incompatible versus did I mess up I guess um the uh in general I think it's a trial and error process it's kind of very similar to like trying to tell a priority if there's going to be a negative transfer or not yeah uh it's not in this bit of sharing this could it be possible to play train from the other stuff yeah so another way that you could possibly share less is to basically pre-trade on one toss and then fine-tune on maybe the tasks that you care more about or something like that um and actually this ties in a little bit to what I had on the rest of the slide which is that you don't actually have to either share parameters or not share parameters it can be a little bit more flexible of a decision it could be something like pre-training and fine-tuning um or you could have something which is referred to as soft parameter sharing where you actually have separate parameters for the tasks like a pre-trained versus fine-tune network but perhaps you just have some soft constraint that encourages those parameters to be similar to one another and the way that you would Implement that is take the same exact objective that you had before where you have some set of task specific parameters perhaps even the entire network has top specific parameters but then try to tie them together with a loss that encourages the tasks the parameters of different tasks to be similar to one another and so there there is actually much more of a Continuum than just sharing versus not sharing um some of the benefits of this is this can allow for kind of more fluid degrees of parameter sharing and and things like fine-tuning are also an example of that um it's also worth acknowledging though that this has some limitations because um this introduces yet another set of design decisions and Hyper parameters you need to figure out how to weight that that loss function that ties them together um and um and this is also going to be more memory intensive because you have to store separate parameters cool um the second challenge that you might encounter is based almost the opposite of um of negative transfer which is that uh you might not actually be sharing enough so um if you see that you are training your multitask learning problem but you're actually overfitting a lot on your problem UM multitask learning is a form of regularization because you're giving it these auxiliary loss functions that should help learn representations and if you're seeing that you're overfitting on your tasks then it could actually be beneficial to try to share more than what you're currently sharing um and now the last challenge that I'll come to which is um kind of related to some questions that came up before which is that um generally if you have a number of tasks and you want to determine if you're going to see positive transfer or negative transfer like and trying to understand should you train all of them together can you figure out which ones are going to be complementary um in general I think this is somewhat of an open problem and the bad news is that as I mentioned there's no thing that will just tell you will it work or not at the outset there's no closed form measure of how similar two tasks are how complementary they are in practice and the reason for this is that uh it depends not just on what the tests are but on the data set on the optimizer on the architecture and so forth and it could even depend on where you are in the optimization process so for example one task might be to pick up a fork and skewer and another task might be to pick up a fork in that case if you're early on in the optimization then picking up the fork is the first step that you need to learn and so they might be very complementary early on in the optimization but then later on they may actually be not very complementary because one of them actually needs to do something once you've picked up the fork whereas the other thing doesn't need to do anything um this somewhat better news is that there are some ways to try to approximate task similarity from a single training run rather than trying to brute force uh and see what happens when you um train different sets of toss together um and so here's one example of something that tries to basically do a single training run of a single multitask Network analyze how those tasks are similar to each other by looking at looking at the gradients looking at the optimization process and then ultimately figuring out which tasks to be grouped together and which tasks should be trained separately okay um so to recap um most of the lecture um we talked about what a task is as these data generating distributions we talked about how each of these tasks have data sets um how for the model architecture we could have multiplicative conditioning versus additive conditioning and that multiplicative is a bit more general or a bit more expressive and then you could also try to share more or less in your network depending on the transfer that you observe um we also talked a little bit about the objective and the optimization about choosing task weights and as well as stratifying your mini bouches so that you have a similar amount of data per task in your mini batch question is there like a data science relation to the success of these approaches so if you have less data would it make more sense to train like separate models for each task as opposed to like a more complex multi -time yeah so um in general if you have less data per task then multitask learning has more potential to be beneficial because it's a form of bringing in data from bringing in additional data basically the data from other tasks is being brought into the optimization process whereas if you have a ton of data for all of the tasks then um you'll probably chances are you'll probably do well if you just train from scratch on um on those tasks yeah it's very very reasonable way to be distracted by how much structure to tax charity makes it similar usual information but how you can like you better be easier to quantify how well you can create one thing for another is there something analogous for taxes yeah so the question is is there a way to quantify um the similarity between tasks and with something like Mutual information um and yeah so maybe I'll try to I usually cover this a little bit later but I'll see if I can try to cover this in the next lecture um basically there's a way to formulate um to basically think of it in the language of graphical models and think about what are the statistical dependencies between the um between the data sets and the strength of those statistical dependencies will translate into the similarity between the tasks awesome so in the remaining 10 minutes I'd like to get into a case study of where um where people actually use multitask learning in a real world problem and in particular this is a paper from some folks that work at Google and their goal was to make recommendations for YouTube and so they are basically trying to figure out what should you put in this right column right here so a very real real problem probably even something that you may have encountered yourself uh and it's pretty cool because the paper actually goes into a lot of detail in um in how they actually try to solve this problem um so before we get into how this is a multitask learning problem let's get into the setup so as input they have information about what the user is currently watching and they also have some features about that user so they have features about the video and features about the user um and once they have that input they're going to generate a few hundred candidate videos and then try to rank those candidate videos and then ultimately once you have a ranking then you'll serve the top ranking videos in that right side panel uh the candid videos are um are going to be pooled from multiple candidate generation algorithms these candidate generation algorithms are going to use things like matching the topic of the query video looking at videos that were most frequently watched with the query video and also other approaches the focus of this paper isn't on candidate generation it's really on the second step of once we have a few hundred candidates how do we rank the ones that we think are going to be the best options cool so um in this ranking problem uh the input is again the information about the query video information about the candidate video and also information about the user uh and and other context and so in particular um those input features are showing the bottom yellow boxes so features of the Korean candidate video and features of the user in the context that's going to be passed into the neural network as input and then there's the output so the output of this model is going to try to Output measures of Engagement and satisfaction with the candid video and so intuitively if we want to be able to um in this case their goal is to try to figure out things that have higher rank and so things that have if you're able to predict the engagement and satisfaction you'll be able to rank the videos and so more concretely what engagement means is they're going to build they're going to be predicting binary classification tasks like whether or not they click on the video um and also regression tasks that relate to the time spent watching the candidate video and satisfaction is going to correspond to things like clicking the like button on the candidate video and also the rating that they give to that video which I believe is um from surveys um one thing that's interesting is once they have the model output they're trading a model to Output these things from the input features that's what the machine learning problem is um to get the ranking score they actually manually they manually weight a combination of these different predictions and tune the weights of on these different things in a manual process based on what they think seems to do the best um cool so I guess question for you before we move on to the approach um so the objective is to predict kind of Engagement and satisfaction I'm curious do these objectives seem reasonable and what are some issues that might come up with um with the subjective yeah so you might have um kind of missing data that comes up because people might not respond to a survey or they might like maybe people like a video but they don't click like for example yeah like a video may be like sorted maybe you didn't like buttons yeah so things like time spent if you have a longer video maybe it'll people will spend more time watching that than short videos um and so you may want to kind of control for that yeah buttons I mean it's possible that there is a lot of Engagement but uh satisfaction data is to do this yeah so you're going to have data imbalance you like the surveys for example you're gonna have a lot less survey data maybe a lot less um data about whether or not users click likes click like or not but more dense data about engagement yeah a video that a user engages with a lot they might not necessarily like we're showing action that would like a negative review but they ask you just like status issues for them yeah so people um people might watch something but not actually like what they're watching yeah something like self-reinforcing and group cycling but if a user's creating another video and then you show them more of the same type let me just keep clicking on the same thing but they you know actually lose your way to content you're just restricting what they're seeing yeah so feedback loops so once you start deploying the system and actually have it collect data then that might affect what people are clicking on and um and that might uh you might have like that might lead to a data distribution that is a little bit skewed and difficult to make Generations on yeah well this is an editorialization but from like a social perspective you're optimizing for wasting people's time yeah awesome so um the the point was that maybe optimizing for a time spent on YouTube is not uh good for society and I I really like this response because last time I asked people last year I asked people this question and everyone was pointing out technical challenges and not pointing out actual like ethical or societal challenges with this so actually thinking about the subjective not just from the standpoint of a technical standpoint but also like what should YouTube be optimizing for I think is really important yeah they want to make sure that so along with the exploration is also some exploration topics like some readers might not be that popular but later for the platform this is more diversity and like solve these kind of problems they might not read that and do yeah so I mean in general these metrics are very short-term metrics and um actually factoring in what's going to happen in the long term um and whether or not videos that like and yeah exactly it's like sort of like what that feedback process will look like um is really important for um for their business model and also in general for um for actually serving stuff that people want to see cool um for the sake of time um let's kind of go into actually how they do this um so the basic option this is really the Baseline is a multi-head architecture where they pass as input the input features into a shared bottom layer and then and then have these top specific heads that are predicting different engagement and satisfaction measures um they found that this can harm learning when the correlation between tasks is low and they're actually going to try to improve upon this architecture right here by actually allowing the model to share a little bit less between the tasks um and so what they choose to do is they use an architecture which they refer to as the multi-gate mixture of experts modeled and what essentially this is going to look like is there's still going to be one shared bottom layer but there's going to be a number of different experts and it's going to try to actually have the model choose how it uses those experts each of those kind of parts of the model based on what it finds works well um and essentially what this will try to do is to allow different parts of the network to specialize for different tasks while also having it figure out whether it's useful to reuse some components so specifically what this looks like is um after the shared bottom layer you have a um a set of expert neural networks so these are different modules denoted by I and for a given set of features for input X and task K we're going to um try to basically predict which expert we want to use for that input and for that task um and so the way that we're going to do this is we're going to pass this through a softmax function you can think of a softmax function as something that's trying to predict a one hot Vector but in a soft way so that we can differentiate through it so in particular this is going to give us a probability distribution over the experts that we want to use and then once we have this probability distribution we're going to weight the predictions of the experts by these probabilities um and then once we have the output of this this weighted expert outputs will then compute the output from there um and in their experiments they implemented this in tensorflow with tpus they trained on videos in temporal order because they have a ton of video data um and so they're going to be running training continuously to consume the newly arriving data and then they did online a b testing in comparison to a production system and then here also model computational efficiency matters as well and then the results they found that uh this mixture of extra models with eight experts is able to do three percent better on satisfaction metrics and 0.45 percent better on engagement metrics which is actually pretty substantial given the scale of this kind of system um and this is in comparison to a shared bottom Network and you can also um look at how it's utilizing the different experts for different tasks and we see that there is some specialization for example expert 7 is used a lot for satisfaction task four but there's also a considerable amount of sharing for example expert five is used for a lot of the tasks um cool so we're basically out of time so to recap the lecture we talked about multitask learning and how it learns a neural network conditioned on zi we talked about how the choice of the task waiting is going to affect the prioritization of the tasks and we also talked about how conditioning on zi will affect how the parameters are shared and if You observe negative transfer it's helpful to share less and if You observe positive transfer it's helpful to share but potentially try sharing more or if You observe overfitting um so really these are the key design choices when it comes to multitask Learning Systems um and next time we'll start to cover transfer learning and get to some cool meta learning topics as well