Machine Learning Math: Vector & Matrix Operations Review

[00:00:06]hello um welcome back on uh today's lecture which is going to be on the review of basic Vector notations and operations so this uh is part of the course for you know learning machine learning so let's get started I'm I'm Professor woo okay let's uh start by thinking about vectors a vector I hope most of you by now have had some experience with vectors but let's start from the basics you can think of a vector as basically a column of numbers just Columns of numbers enclosed in a bracket for example here is a column of numbers enclosed in a bracket so here's the bracket right here's another bracket right it does it it doesn't necessarily have to have two or three it could have any number of elements in it and each element could be numbers or something else so for this class we're going to mathematically and symbolically represent vectors using the lowercase cursive sign sometimes they use the lowercase bold as well so you might see both in literature but for the most part we're going to focus on just using the cursor um um font to represent vectors now with a vector we can denote the length of the vector using this notation this essentially says that this Vector has three elements in there you see one two three three elements and of course this one says is a vector with only two elements the r here stands for real so he has two real numbers as opposed to imaginary numbers and it's also saying that this Vector represents by two number is basically like two Dimension and this one is three dimension okay besides this all right of course vectors doesn't have to be just X it could be basically any letter here we use y we can even use Greek letters so so we can essentially use XYZ any letter as long as this lower case in cursive to represent a vector okay so that that is the very basic concept of what Vector is and in machine learning vectors we normally use vectors to represent a single sample of data so let's say we wanted to collect data about Northeastern students so each student will be considered a single sample right so but with a single sample you might collect multiple informations about the student right we call them these multiple uh characteristics as features features so a single sample a single sample may have multiple features so for student one for example we may be capturing the weight hair color eye information and what we do is we set this as a single Vector so for student one you will be 160 black and brown okay and for student two you will be 174 blonde and blue so you can see that these vectors represents a single sample in this particular case a single student and the elements inside the vectors essentially represent each feature of the sample a single sample can have multiple features right so so so you can kind of picture we have a vector and vectors are normally used to represent a single sample and then we can distinguish different samples by putting numbers on them so this is saying the first student this is saying the second student or the nth student whichever student you are talking about you can just put that number there okay so this will be the standard we're going to use for the class this is how we represent a vector and different samples could be represent using the number uh subscript okay once we have a collection of samples so over here like I showed you it's not just one single sample we have multiple samples right where each first sample is X1 and second sample would be X2 when we have multiple of them what we like to do is to stack them together so over here 15 four notice I stacked them together and 223 I stacked them together now this right here when we have more than a single column where we have multiple columns this right here is the Matrix so X is a matrix and notice that we tend to represent Matrix with a capital letter Capital as opposed to a single sample we will tend to represent those as lowercase lower case also when we're talking about a single sample the standard we're going to use is is going to be a single Vector column Vector specifically column however when we're talking about the whole database like all the samples together then we're going to stack them horizontally horizontally notice how as a as a vector we're going to always assume it's a vect the column vector but when we stacked together we're going to assume they're organized in a way so it's horizontal okay so this is a key uh standard that is used a lot and you need to make sure you remember that okay more notations on the data so like I just told you if you have the entire data set we tend to represent them as a matrix the way we can denote the dimension of the Matrix is X has n rows and D columns so they n rows and D columns that's that's what that's how you know the dimensions of the data once you have the data you can also denote a very specific element so for example you can do x i j so I is saying is the first it's the first row J is the second column so he talking about this one so X12 is equal to zero alternatively you can talk refer to an entire column so entire column here is X uh entire column and it's saying the third column one two 3 So it's talking about this column notice how this dot this colon basically tells you you're taking the whole whole column or you can you know denote the entire row so over here is you can use that notation except the colon is at the back as opposed to the front and this is saying that you want the I the E row now there are two ways to represent this General we just write XI lowercase or you can write it this way they they mean the same thing so X1 is essentially saying it's the first sample and here is horizontal once again when you're just talking about a single sample it is vertical okay now in machine learning it's really uh when it comes down to it it's all about the manipulation of data either as vectors or matrices right in the that's basically all we're doing we somehow represent the data do some kind of calculus on them calculation and then we're done so so the first step for us to learn machine learning is to really review and understand the basic Matrix and Vector operations now um I assume a good deal of you guys have already done this and for those of you that haven't don't worry well we'll spend a class to kind of review these Concepts so so let's start and and luckily none of them are really hard for example addition and subtraction addition and subtraction is really easy let's say you have a vector X and a vector y so what happens when you try to add them together well when you want to add them together you just match up the element and add them together so 1 + 2 which is three 3 plus 2 which is five so adding them together is not a big surprise right and Visually what's happening is if you have 13 which is this vector and then 22 which is this Vector your final Vector is 35 which is talking about this one right here so that's what's happening visually but but mechanically what are you doing you're really just adding the elements together and subtraction is the same idea so if you have x minus one instead of plus one then it'll be 1 minus 2 which is -1 and 3 minus 2 which is 1 so subtraction and addition of vectors are really really simple now another operation is called the transpose so if you have a x Vector X transpose it's telling you that whatever is in the column form turn it into a horizontal form right another example is y the Y transpose T transpose right so if it's transposed it converted now if for some reason a vector is in the horizontal form which it shouldn't be but let's let's say for some reason it is and you write WR the transpose of this then the transpose becomes column so column becomes row row becomes column that right there is the transpose operation next is multiplication there are multiple ways to do multiplication and they are different products right so one of the product is called hatma Product the symbol for hatma product is you take x times y this is how you do multiplication a circle and a DOT and the hatma product is basically the same as addition and subtraction you take this you match up the elements you multiply them together here's two you take them you multiply them together that's six right so the hatam up product is really just an extension of the addition subtraction extended to multiplication now another one is called a DOT product the dot product is written this way X doy y however an alternative way to write it is X transpose time y okay when you have this x transpose times Y what you do is you multiply this with the first element you multiply this with the second element here and here and then finally you add everything together and this is called a DOT product after you add everything together you you should get eight so the do product is very very similar to the Hader product right we also take one element multiply and multiply the only difference with the dot product is once you have this final product you add all the numbers together resulting in only one number 6 + 2 is 8 right so so another thing that's different is the notation we write them as X transpose y okay if you have X and Y that doesn't make sense it has to be a transpose okay later on once we get to Matrix you'll see why that that that notation makes more sense okay now another thing is called inner product so we learned a DOT product well inner product is notationally written this way with two brackets like this now with this inner product for our purpose you can treat it to be identically the same as the dot product so so you can see now it's exactly the same so when you see this notation this notation or this notation they're all the same and they are all three of them are used like quite regularly so once again hadam a product you take element and multiply them together the dot product which is written this way most commonly written this way it's basically the hatma product except you add everything up in the end okay and the inner product you can treat that as the same as a DOT product okay so now that we did three products this is the fourth one we just did the inner product now we there's another one called the outer product the outer product remember the inner product is X transpose y outer product which uses this symbol here right this is this is the outer product symbol notice how the transpose is on the outside right the transpose is not in the inside anymore it's on it's on the outside therefore when we write them together when we we essentially is have a column and a a row now the outer product the way we would do this is we will take this element multiply by the column which will give us this column and then we'll take this next element and then once again multiply by the column so we got this so the inner product and outer product have different operations where the where the outer product essentially you take this multiply this multiply the inner product is like a handma product you multiply one element at a time and then sum them together multiplication by a constant value so if you like right now we talked about Vector multiply by Vector what if it's just a vector multiply by a number well if it's a vector multiply by some number Alpha for example you simply take this number and multiply one at a time so it's Alpha time X1 Alpha * X2 for example if you have three well it'll be 3 * 2 3 * 1 giving us 6 and three okay next just like vectors when we talk about matrices we can also transpose them so over here we we saw how you can transpose the vectors which converts a column into a row a row convert into a column now what about a matrix how will we transpose for example this one well it's actually really easy you simply convert all the columns into row okay so over here well you take this column convert into a row this column convert into a row this column convert into a row and that's it that's it this is that easy now if you put a transpose here you will convert back okay so you will convert all the rows to columns and columns to rows and over here notice how if you do a transpose and a transpose again it's identical to the original okay so so that that's essentially how you can do it you take each column convert into a row take the next one converted into a row so that's that's what the transpose is for now let's uh take a little bit of time why don't you try to solve these following equations using everything we've learned so far it takes a while it takes a while to basically absorb I just jumped somewhere takes a while to to absorb all this information and it takes takes repetition like you just got to do a couple times and then then you'll get it so take 10 minutes pause the video make sure you can do this and we'll move on okay so all right pause the video because I'm moving on I I wrote a solution out for you for each one of these so you can essentially go and double check if what you've done uh makes sense so these are the solutions okay so now let's continue with products multiplication we talked about multiplication between vectors but now what about multiplication between matrices so for example here x and z we want to do a hatam product remember this is a symbol for hatma product for Hada product was the one where you match up the element and multiply them together so over here you match them up you multiply them together 1 * 1 is 1 0 * 3 is 0er Right 2 * 2 is four so you just match them up and multiply them together the head of my product is really really simple for either multiplication or vector multiplication however um um it gets a little bit harder when we're doing other products so over here is an example of a Hada product right and then multiply by constant so here if you have a constant multiplied by a matrix you may have guessed at this point you simply take this and multiply by every single element so once you do that you have this and then you do the hatma product which is when you align every single element together okay so like I just showed you if you have a multiplication of matrix by a constant then you simply like 2x you simply take the two and multiply by the entire Matrix now when you have an addition for example you have a matrix like this and you're trying to add to just a number so if have Matrix adding to a number um the standard we're going to use here is you simply take the number and add it to every single element so just add two to this so 2 plus this will be three 2 plus this will be two 2 + 2 is 4 and 2 + 1 will give us three okay all right the next one is called The Matrix dot product when we have the Matrix dotproduct then let's see um when we it's written as this The Matrix dot product is probably the most commonly used multiplication and therefore the standard is you don't even need to write anything you just write XZ and that assumes that we're doing a matrix do product and how does it work well what we do is we take the First Column and multiply by the first row so this right here is your dot product okay so you do the dot product between the First Column and the first row and then you do the dot product again between your First Column and the second row and then you can do the dot product again between your second column first row and then finally you do the dot product here remember when we have the dot product we always treat this as the horizontal well this notation carries over here so you take the column and you multiply by the row and once you calculate the dot product for each one of them right this will year number you number number number which give us your final result of a 2 by two Matrix okay let's do um let's do another example so let's say we have this Matrix and the multiply by z x z so what we what would you do well you will take this and the multiply by the first row so this will give you one and then you take this again and multiply by this using the dotproduct so 1 * 2 and 2 * 1 add them together is four okay so this is how you essentially calculate the multip multiplication of uh matrices now quick aside this is this is a very use useful property of the if you have a DOT product between X and Y and if you're trying to find the transpose of the result this is equivalent to multiplying the transpose of Y times transpose of X so notice how you rotated the two of them and then add a transpose on this okay so this is a property which will go over uh return to over and over again all right so another property we need to learn is that the order of the Matrix of dot product cannot be flipped what that means is if you have x x y if this is X and this is y this is not the same as y * X you can test this out you just multiply x * Y and then you multiply y * X and you will notice that the results are not the same okay so you cannot x * y when you do a matrix dot product is not the same as y * X cannot flip this okay the diagonal and trace of a matrix so Matrix you can extract various information out of it one of it is the diagonal what is the diagonal well if you have a square Matrix like this the diagonal is simply the diagonal element okay so the diagonal of this Matrix is a vector 14 and N notice how this is 14 and N here's another example the diagonal of Z is simply four and one okay diagonal is always this way it is never this way not that diagonal the diagonal is always this way okay now when we talk about the diagonal we got the diagonal when we talk about the trace of the Matrix so if we have a matrix and somebody is asking you what is the trace well the trace is simply the diagonal right and you sum up all the elements so right here will be 149 149 so you take the diagonal and you sum them up which giv you the trace okay next is the Matrix inner product so we we did The Matrix in a uh Vector in a product The Matrix in a product is when you have x times the transpose of Y okay so we this right here is the dot product x dot product the transpose Y and then you should have a square Matrix and you take the trace of that we just talked about what Trace was right you had diagonal which gives you the diagonal and then we had the trace which sums up so the inner product is kind of accumulation of everything we learned to do an inner product you need to know how to do transpose you need to know how to do matrix multiplication and then you need to know the diagonal of this Matrix because Trace is the summation of the diagonal so if you can do the Matrix in a product that in itself is essentially like everything we've learned okay and the way we would write this you can see you multiply the two Matrix together you take the trace the transpose once you have the transpose you can multiply them together after you multiply them together you simply sum up the diagonal to get the trace and that's how you got the Matrix in a product okay all right so let's spend a little bit more time now to to kind of exercise your muscles so that you have the chance to kind of absorb cuz because we're doing this really quickly and and I can talk here all day but not until you get to do it you won't it's not nothing's going to really stay so give it a try and see if you can solve these problems right as and and if you have time solve for these okay pause the video and give it a try hopefully you're able to uh hopefully you're able to get it over here I have uh the solutions for you so you can you can go and kind of study to see if you did it right or you did it wrong okay so these are all the solutions make sure you are able to solve them um okay so I'm moving on like hopefully you pause the video and you have already gone through this already okay all right so now now that we've gone through the mathematics of it you we also in real life obviously you don't want to be calculating all these stuff by hand what you want to do is have an the ability to calculate them in the computer so uh whoops what just happened so what we uh traditionally have been using is this tool called Python and numpy num numpy okay and once you have nump installed you're able to represent vectors matrices um together and essentially calculate various information about them so for example over here you have a vector right so you can you can write this Vector this way notice how each column is represented by a bracket so that's a column so so uh I mean each row is represent by brackets so that's one and two over here you can represent a horizontal vector by writing it this way so you can represent either a column vector or horizontal Vector you are not going to need this very much because we normally represent things as a column Vector it's it's very rare you're go in this class you're going to be represent things anything in terms of um or um horizontal Vector right over here is another way to do to denote a vector and it's essentially just saying p is an array right that that's what we saying what is magical is once you have defined these variables p v u and P when you want to do addition subtraction you simply write V plus P right which would which you create operation and you will see that V plus P will give you the result here or you can do the dot product so you take V remember you have to do the transpose and then you dot p and this gives you the result v. P you can also denote matrices so over here is the Matrix right well you have this Matrix here so so so it's now defined a is equal to this once you have the Matrix you can get the entire row entire column out of it so over here you can get the entire row now notice this says zero so instead of starting from one the first row it starts from the zero throw in in programming since you're most of you are CS students uh you should be used to that that in mathematics we start from one in um in code they tend to start from zero or you can get the first uh the second column so this is three and four so it's getting the column remember zero is the first one and one is the second one um so so um once you have the vector in The Matrix you can run various other operations such as Max so this gives you the maximum value in V which should be two see and the max value in The Matrix a which should be four okay or the minimum value minimum value in a which is one or you can even find the average right what's the average of V which is 1.5 okay now here is here is some cold examples so so here's the vector v right I can represent V that way and then I say that I can print out the V so you can actually visually see it I can print out the U exactly the same here's the a matrix we talked about and we can do this one is the outer product here's example you U Dov transpose remember the trans transpose is at the outside which is the outer product outside outer product um we can also do the inner product which the transpose is inside and then you can also do like more complicated V transpose a v right so this multipli by a matrix and then multiply by Vector so yes you can multiply a matrix by a vector and vector by a matrix or you can do a hatma product the hatma product remember it was this symbol in Python it's simply a star you write a star and and that does the job here's an example of a trace right so this right here give you the trace okay now try to consolidate your knowledge by solving these by hand hopefully at this point you've had a little bit of practice and solving this will be a lot easier Okay and like solving this by hand is not fun so you can always confirm like there no solution to these ones so you want to confirm this by typing these into python okay so so that's that's the final exercise of the day and I will see you in the next lecture