Regression Metrics Explained: MAE, MSE, RMSE, MAPE, MPE | Machine Learning

[00:00:00]hello everyone and welcome to this section in this section we're going to give an overview of regression metrics which is if we have or we build a regression model okay either if it's a simple linear regression model multiple linear regression model or polynomial regression model what we wanted to do is we wanted to assess the performance of the model we want to see if that model is good or not does it actually can we deploy it in practice and when can we use it to predict data that the model has never seen before okay so to do that we're going to cover kind of the most important metrics so let's go ahead and get started first is how can we assess them other performance okay so after we perform model fitting which is simply we took all the data we divided them into training and testing and then we take the training data which is let's say maybe 80% of all the data I have and we train the model we fit the model okay so now we have a regression model that looks good so after we put for the model fitting we would like to assess the performance of the model by comparing the model predictions to actual true data so what we do is after we take our train model and then we're going to deploy it in practice so we're going to use for example let's say the 20% of the data which is a testing data to assess the performance of the model okay and a very important kind of key point to illustrate that testing data one of the requirements for it is it has it should not be seen by the model during training so the model should not have seen it before ever during training and kind of you know a metaphor for this is let's assume that you are for example let's say a student and you know like as a professors for example what you do is that you give the student a couple of exercises right so the student keeps training and training learning and so on and then during the test okay when you actually test that student you do not give them the exact same questions that you trained them upon it doesn't make any sense because then it becomes like a memory dump right you just feeding an information to the student the student just memorize it and go to the exam and actually retrieve that information and that's it we call it today this is obviously not the way to learn we wanted the model to generalize we wanted the model to learn and to like apply that learning on a data set that the model has never seen before during training okay all right so let's get started so let's assume that we have again our ice cream cart our ice cream example and we wanted to develop a regression model that models the relationship between the temperature on the x-axis and the revenue on the y-axis so I went out there and I collect the data you know as an ice cream let's say cart owner or a business owner we said okay at maybe temperature let's say ten degrees C that revenue was very small as the temperature goes up you know like people tend to like ice cream more that's why the revenue becomes higher so we collected this data set and then we fit our linear regression model so if it's straight line that can best fit our training data so the question is how can we evaluate the model so what we could do right now is to do this is that we're going to calculate the error okay which is the difference between what the model is saying so these are my estimated values and that's what called the predicted values so any point here that lies on that line these are estimated or predicted okay because because they're coming from my equation they're coming from a line or my Fit however these are the true data points these are my true label and that's why we call this Y actual or Y I I stands for an index which can be zero one two three depending on which data points were referring to and why I had okay this refers to estimated or predicted that's what the model is saying so what we do to do this we just calculate what quality residual which is the error or the difference between the predictions - the true or the ground truth well if the model was perfect then that means the the estimated will exactly match our our true label so what is the error will be basically equals to zero okay all right so now I know how can we how can I calculate the error for each of these points so now I have I'm going to have error one error for the second data points error for the third data points error for the fourth data point the problem is how can I combine all that information how can I combine all this error into just one metric and that's the beauty of it that's why we're going to be covering in this lecture all the different metrics that can rely on that difference on the error again please bear in mind that here all this calculation we're going to be doing it or repeating it for every point because here I have subscript I which means I have an error for this point error for this point at all for this point and so on and now I need to calculate the error just as given one number so I want to combine them somehow okay another note another important point is that here okay here I'm trying to calculate the error for the exact same data so here on this this is my training data and that's how I fit my model and I'm collecting I'm trying to calculate the error for the training data in practice we're going to have another data points which is our testing data points that we're going to calculate the error based upon okay all right so the first metric that we're going to cover is what we call it the mean absolute error or M AE which is pretty simple mean absolute error is obtained by calculating the absolute difference between the model predictions and the true or actual values and first why it's called mean and why it's called absolute first it's called absolute because we're going to calculate the absolute value simply put as you guys can take a look at the equation you will find that I'm going to take every point with subscript I calculate the difference between the Y I which is my model - values - my predictions which is why I hat that's what the model is predicting I'm going to calculate the absolute value okay and then what I'm gonna do and again I obtain the average just average all these points which is pretty straightforward I was just gonna basically sum them up some all this these errors so basically are just gonna calculate the absolute error for here absolute error for here absolute error for here and whatnot and then I'm going to simply sum them up and divide by the overall number of samples or number of data points that's all what it is and that's why it's called mean absolute error or ma e so ma e is a measure of the average magnitude of error generated by the regression model and again these are the steps to calculate it first we calculate the residual of every data point second we're going to calculate the absolute value which is the objective mainly is to get rid of the sign so you know here the sign would be positive all the time and then we're going to calculate the average of all residuals just sum them up and divided by the overall number well if ma E is zero this indicates that my model predictions are perfect that's great that means it's exactly match then we would call it today okay all right the next metric we're gonna cover is well call it the mean squared error or MSE it's actually pretty much the same similar to the previous one but then instead of obtaining the absolute value I'm going to take this and just square it instead so let's take a look at it so MSE is simply we're gonna take okay the summation of all the samples why I - why I had were estimated I was just gonna square them up and divide by the overall number of samples and that's what we call it mean squared error MSE is actually very similar to mean absolute error but instead of using absolute values we're going to square the difference between the model predictions and the training data set MSE values are generally larger compared to the mean absolute error why because we simply squared it up okay so because we are taking the residuals and squaring them up so you'll find that in general MSE will be larger than mean absolute error and in case of data outliers if we have an outlier within the data MSE will become much larger compared to MA think of it as kind of we are penalizing the mistake if we made a mistake okay this this error will be squared so we are penalizing it okay and the MSE in MSE in general error increases again in a quadratic fashion because we are squaring it up while in case of ma e it increases in a proportional fashion because here were not squaring it or just its direct it's proportional basically here it will be quadratic because we're having a square term and in MSE since the error is being squared any prediction error is being heavily penalized we actually cannot punish it much harder if in case of an outlier on K so major error MSE is calculated by the following steps first we calculate the residual for every data point second we calculate the square value of the residual so we take the residuals we square them up and then we calculate the average so simply sum them up the squared values I mean and then we divide by the overall number of samples and alright okay looks great the next metric is well call it root mean square error or our MSE so root mean square error is actually pretty simple when you take the exact same values here and we're just gonna square it I'm sorry obtain the square root for it the question is why why do we need it so the problem will ms e or mean square error if that here the units are very different to the actual data points because we already square them up right so it's hard to compare and that's why if we obtain the root mean if we obtain the square root of this data then we're gonna go back to match the units of the original data so good then now I can compare it okay so root mean square error represent the standard deviation of the residual if you guys recall and statistics that standard deviation is a measure of the dispersion from the mean which is how much am i spread away from the mean okay so our MSE can be easily interpreted compared to mean squared error because our MSE units match the unit of the output or match the unit of the original data here simply I'm squaring it up and then obtaining the square root so it's kind of canceling the effect pretty much okay so on MSE provides an estimate of how large my residuals are being dispersed again it's a standard deviation okay and the steps to calculate it first we calculate the residual for every data point second we calculate the square value of the residual the second step and then we calculate the average of the squared residuals which is exactly step one two three are exactly the same as before and then we're going to obtain the square root of the results and then we're going to come up with the RMS e a root mean square error all right okay it looks great the next metric is what we call it the mean absolute percentage error or map so one problem with mean absolute error values or metric if that mean absolute error can range from any number between zero and infinity you can actually give it whatever number you want which makes it very difficult to interpret the results compared to the training data okay and that's why giving the answer in a form of a percentage is actually very beneficial because everyone can understand it I can say for example my error is let's say five percent well maybe my error is let's say seven percent it's understandable however if I'm for example in my example here the revenue I can tell you my error is 10 or my error is twenty or thirty okay like how can I assess that compared to my training data it's a little bit harder to interpret that's why percentage error actually prefer to give me an idea of how far off I am compared to my testing data normally actual to labels so mean absolute percentage error or math if the equivalent of mean absolute error but provide the error in a percentage form and therefore overcomes the MA II limitations mean absolute percentage error or map might exhibit some limitations why if the data point value is zero okay why because there's an a division involved let's take a look at it so map is simply I'm gonna take okay every data point I'm gonna subtract the estimated so this is a true value this is my estimate so that's my residual I'm gonna divide by the original value of the data point which is why I and then sum them up and then get the average divide by N and then I multiply by a hundred percent so that will give me a percentage value as compared to the actual value of the data point so I'm getting the error divided by the original value of the data point sum them up divided by n so I walked in the average of all of them and then multiplied by a hundred percent so the problem here if my point y is zero then I'm gonna exhibit some problems okay because I'm going to divide by dividing by zero all right okay the next metric is mean percentage error which is pretty simple so instead of having the absolute or just gonna remove the absolute and that would give me kind of you know like an insight of how many positive errors are compared to the negative ones I'm not gonna be here I can cancel them for example however here before in the map because I'm obtaining the absolute value I'm not gonna if the sign doesn't matter okay so if they are positive or negative doesn't really matter here it matters so M PE is similar to the map but without the absolute operation it's exact same equation but without the absolute MPE is useful to provide an insight of how many positive errors as compared to the negative ones and npe can be calculated as follows alright okay sounds great and that's all what I have for this lectures I hope you guys enjoyed it in the next lecture I'm gonna walk you through an important metric which we'll call it R squared or coefficient of determination I hope you guys enjoyed this lecture let's see cap so in this lecture we covered kind of an overview of regression matrix so I've been able to see how can we calculate the error first which of the residuals difference between the true or the ground truth - what the model is saying and then we knew the mean absolute error by simply containing the absolute and average them up and then we cover the mean square error which is by just obtaining the difference between the between the predictions and my true ground truth and squaring it up okay that's all what it is and then we learned the root mean squared error which is simply the mean squared error that we had before but we obtained the square root just to come back and make the unit match the units of the output and then we have been able to come up with a percentage value using the mean absolute percentage error or map by simply obtaining the absolute value of the residual divided by the original value of the point and multiplying it by a hundred to give me a percentage value and then we have been able to obtain the MP e which is mean percentage error just instead of having the absolute I just got rid of it and then I came up with an estimate just to get an idea of how many positive compared to how many negative ones and that's all what I have best of luck and see you guys in the next lecture