What Linear Algebra Is | Machine Learning Foundations

[00:00:00]hi though this is John crow welcome to my machine learning foundation series this is the first subject in the series it is intro to linear algebra this is an interactive primer on the theory and practice of tensor manipulation in Python and here is a wonderful illustration of my puppy oboe by illustrator Hagley Basson's so in this intro to linear algebra we are going to have three segments we're going to talk about data structures for algebra then we're going to cover common tensor operations with tons of hands-on examples and finally the final subject in this subject is matrix properties we're going to kick off segment 1 in his video data structures for algebra all of the topics in here are what linear algebra is a brief history of algebra tensors particular types of tensors including scalars vectors simple manipulation of vectors such as transposition ways of characterizing vectors like norms and special types of vectors like unit vectors more special vectors basic orthogonal north normal we're going to tackle building vectors in numpy and other types of tensors as well as matrices tensors more generally in tensor flow and pie Torche the two most popular automatic differentiation libraries in this first video we're going to talk about what linear algebra is and just do a single flight on a brief history of algebra you'll have to wait for future videos for the rest of the content so what is linear algebra well first let's talk about what algebra more generally is so algebra is arithmetic that includes any non numerical entities like X so here's a simple example if we have this equation 2x plus 5 equals 25 well we can subtract 5 from both sides of the equation and then that will leave us with 2x is equal to 20 we can then divide by 2 on both sides leaving us with 10 is our answer and you can confirm for yourself that X must be equal to 10 because you can plug it into the original equation and when X is equal to 10 2x plus 5 comes out to 25 it's the only solution to this equation so that's what algebra is now to take it into what linear algebra is a bit more it isn't linear algebra if it has an exponential term so for example an equation that has 2x squared plus 5 well that isn't linear it's a nonlinear transformation and then a square root that's also a nonlinear transformation so this equation is also not linear outgrow to give you a really nice and tidy definition of what linear algebra is we could say that it's solving four unknowns within a system of linear of equations so let's talk about that idea of a system of linear equations multiple linear equations were resolving four unknowns like X across several equations simultaneously so here's an example let's say a sheriff has a car that travels at 180 kilometers and out a bank robber has a slightly slower car it goes 150 kilometers an hour but that bank robber gets a 5-minute head start on the sheriff so how long does it take the sheriff to catch the robber and what distance will they have traveled at that point for simplicity let's ignore in celebration traffic etc and we'll just assume that they're traveling in a straight line in one direction so we can solve this problem graphically with a plot we can note here that 150 kilometers per hour corresponds to two point five kilometers per minute and a hundred eighty kilometers per hour corresponds to three kilometers per minute you can plot things out here and so we have a plot of time along the x-axis the horizontal axis in minutes and then we have a plot of distance in kilometers along the vertical axis you can see here there's a green line corresponding to our bank robber going two kilometers per minute so that's a particular slope and our sheriff is traveling at a faster speed so it has more slope a slope of three instead of two point five corresponding to three kilometers per minute it's bitter to the half kilometers per minute and then we have our 5-minute head start to the bank robber accounted for by the gap here along the x-axis so what we're trying to solve for is this crossover point where the sheriff catches the bank robber so what time is it how many minutes have elapsed at that point and what distance have they both traveled at that point so you can see that graphically you could come up with a solution alternatively we can solve the problem algebraically to represent the bank robber you can have this first equation which is distance D is equal to 2.5 times time and then a second equation representing the speed of the sheriff however we also take into account that five minute head start so we have a penalty of five for the sheriff now both of these equations are equal to D so we can actually set this term here equal to this expression here so 2.5 T is equal to 3 t minus 5 then multiply the three into the brackets to expand out this side of the equation and then we start trying to isolate T by moving 3t over to the other side of the equation here so we subtract 3t from both sides of the equation when we do that now 2.5 t minus 3t gives us negative 0.5 t is equal to negative 15 and so now we can divide both sides of the equation by negative 0.5 and that isolates T completely giving us a time of 30 minutes so now we know how long it takes for the sheriff to catch the bank robbery to figure out distance this is really easy now that we've solved for T we can simply plug that 30 into the T in either equation so for the first equation 2.5 times T 2.5 times 30 is equal to 75 kilometers and then we can plug it into the second equation 2 so 30 minus 5 is 25 and 3 times 25 is 75 kilometers of course we get the same answer because it is a system of linear equations now that was a nice neat and tidy situation where we had one solution however there would have been no solution if the sheriff's car were the same speed as the bank robbers so if they both were traveling at exactly the same speed forever then there would be no solution to this problem on the other hand we could also have an infinite number of solutions if both the bank robber and the sheriff were traveling at the same speed and have the exact same starting time so the slope is the same the start time is the same now they overlap at every time point these are the only three options in linear algebra so you either have one solution no solutions or infinite solutions it is impossible for the lines to cross multiple times so this is a key part of us being linear algebra in these systems of equations in a given system of equations there could be many equations there could be many unknowns in each equation in the example that I just showed you there were two equations and there are two unknowns however let's consider another example here where we are building a model specifically this is something called a regression model which I'll get into a little bit of the detail here in case you're not already aware of it in this model here we are trying to predict Hesperus so we have for a given house we have its price that we're predicting Y and then we also have a number of features or variables that we are collecting to try to predict that house price and in a lot of cases the more features that you have the more accurately you're going to be able to predict what you're trying to predict the more relevant features so some of the relevant features here might be distance to school or number of bedrooms and so on I have M features here and so II C and M these examples of features this can be an integer the number 3 distance to school could be a value like 2.5 or episode me 225 kilometers and so on for however many features we have in this case M features we also have to have a y-intercept here so this is one last variable it allows us to across all of the house prices that we have have an average house price and without this y-intercept within that kind of an average baseline before we start factoring the other features becomes much more difficult to fit a good model so here we have many unknowns and we could have effectively unlimited rows where every time we get a house price and collect the pertinent information associated with that house bracelet just a school number of bedrooms and so on that's another row for our system of equations it's typical in machine learning problems even a regression model like we're looking at here to have thousands of rows representing different houses so n rows we could have thousands of rows here and maybe a dozen features so M could go up to twelve columns in a typical deep learning model in contrast it would not be uncommon to have millions of images say for a machine vision model so we'd have millions of rows in our set of equations and millions of columns representing millions of parameters potentially high-resolution images would have a lot of pixels so here's that equation from the preceding slide I've now generalized it to a lot of different houses and houses and I've made it into a set here using set notation from linear algebra these square brackets and so for any house I in the data set so we can call that particular price Y I which could be say $300,000 it would have you then x i1 2x I M are all of the features for that particular house I now across this whole system of equations we were trying to solve for the parameters a B C M Y is our outcome that we're predicting say a house price X is the variables the actual values that we have associated with that house price so X 1 1 X 1 2 X 1 M and so on throughout this set of equations and these parameters a B C and M these are the unknowns that we're trying to solve for you'll notice these unknowns are consistent across all of the equations so whether we're talking the first row the second row the end throw a ABC and M are going to be the same value so we're trying to find a ideal value to represent all of the rows in the data set for each of the features in each of the columns I have some quick examples to show you here of how this looks in practice so if you go to john curran calm slash deep tf1 this brings you to a deep neural network in tensorflow and by the end of this whole machine learning foundations journey a lot of what's happening in this notebook will start to make sense early on in this journey I'm not expecting this stuff to make sense the reason why I'm in here though is just to show you that this same kind of thinking that we had in the slides from linear algebra book applies as well to various algebraic objects what I'm going to introduce momentarily is tensors so we have vector tensors here one-dimensional tensors and matrix tensor is two dimensional tensors here and so we have inputs these X's are our inputs just as they were in the preceding preceding example so these X's are inputs into the model that we're using to predict the outcome Y in our deep learning model in this particular model it's a machine dish model that's being passed in 784 pixels per image so we have this vector of 784 pixels being input and we're using that to predict outcomes is this by variable T of this vector of length 10 and in the middle we have these things called weights and biases weights matrices and bias vectors the specifics of that don't really matter except that these are I'm just trying to show you some linear algebra data structures and these are encoded in whatever software languages or that we're using in this case this is tensor flow here and so we create bias vectors B 1 2 3 and B out corresponding to B 1 2 3 and B out these vectors here and then we also have weights so of a particular shape W 1 W 2 W 3 and W out so these are these matrices with these particular dimensions so this is just an example of how we're going to have various parameters in this case weights and biases that are going to be tuned by our machine learning algorithms in order to be able to map some particular pixel inputs into a prediction of what those pixels being input represents some particular outcome so it's this is just a slightly more complicated situation than we had with a linear regression example and we can take it just one step further by showing that these linear algebra objects don't have to be two-dimensional we can have higher dimensional linear algebra objects higher dimensional tensors I'll explain all this terminology in more detail in the next video but for now it suffice as to say that we have these many dimensional convolutional objects that are specified by many dimensions and these allow for some especially powerful machine vision models called convolutional neural networks anyway just want to get you excited about what we can do with linear algebra these notebooks these tensorflow notebooks show you an endpoint that we're working towards in the series for now we'll focus on the fundamentals speaking of fundamentals I find it absolutely fascinating to consider the origins of algebra so let's take a quick moment here to do that this guy here is Abu Jafar Muhammad bin Musa more commonly known as al look what Eve me because he came from a place called so according to me means the man from hadith um and so this is a former center of Persian culture in modern Uzbekistan today called Hiva so in English we pronounced this algorithm II so this medieval Persian mathematician who was around from 780 to 850 in the Common Era was not only critical to the history of algebra as I'll get to in a second but is also the namesake of the word algorithm so very prominent old mathematician so al good evening wrote the compendium calculation by completion and bouncing and in Arabic this word completion is algebra so algebra is actually right here in the name of the book and completion and balancing refers to various ways that we can rearrange algebraic equations some of which we've already looked at at the beginning of this video the medieval Islamic Arab Empire made most significant early contributions to modern symbolic algebra so the kind of algebra that we work with today although it was refined significantly by the French such as Rene Descartes in 19th century though that said many other cultures developed their own approaches to algebra so rhetorical algebra this spoken algebra was developed as early as 1900 BC by the Babylonians there's an Egyptian papyrus dated to 1650 BC with linear algebra on it so we know they were up to some algebra Indian mathematicians we're doing linear equations around the 6th century BC as we know from some mathematical documents from them and the Greece as long ago is 400 BC were themselves doing geometric algebra an example that we have here so this was in the time of the philosopher Plato and is a particular example of this geometric algebra here by Euclid in his book elements the Chinese have linear equations solved in a book from 250 BC this book was called 9 chapters on the mathematical art and although the Europeans started much later in the late Middle Ages initially by translating Arabic texts to Latin in the 12th century by the 13th century European mathematics rivaled other cultures and when the Islamic empire declined after the 15th century European culture was left carrying the baton of algebra to today this technique devised initially thousands of years ago in some respects and in its symbolic form many centuries ago by Persian mathematicians we today have tons of contemporary applications to take advantage of lots of compute power and perform these algebraic applications in tons and tons of devices all across the world billions of them I'm sure so one kind of straightforward application is solving for unknowns in machine learning algorithms including deep learning algorithms so this is our primary focus across a smidge learning foundations program it's also used for reducing dimensionality so if we have some high dimensional data we can reduce it down using techniques like principal components analysis we can rank web pages in order of importance for example with eigenvectors we're going to talk about both icon vectors and principal component analysis in subject 2 of the machine learning foundation series called linear algebra 2 and same thing with recommender systems so if you want to build a recommender for movies for example based on other movies that somebody has watched you can use singular value decomposition to do that and again that is something we'll be covering in linear algebra 2 once we get past the foundational content in this linear algebra 1 course it's also used to process natural language you know with some single value of the decomposition or another technique called matrix factorization and so whether we're talking about processing language coming out of somebody's mouth the language sounds or written language we can be using linear algebra to be modeling topics identifying what topics are being discussed in that natural language as well as for semantic analysis so in this machine learning Foundation series this intro to linear algebra subject that we're covering right now is foundational for of course the linear algebra 2 matrix operations subject that is coming up that will cover eigenvectors and SVD and PCA in it also I'm putting it in non bold here because it also means foundational for though not as directly as it is for a linear odd for to for our calculus one woman's a derivative subject as well as for calculus two on partial derivatives and integrals it's also foundational for probability and information theory and intro to statistics the only one that isn't really providing much for is the algorithms and data structures class subject which is subject 7 and it is very important as basically all of these prior seven are for the final optimization class that will tie everything together and show us in incredible detail exactly how modern machine learning algorithms optimize all right up next we are going to be introducing what tensors are I've mentioned them a few times we're going to dig into exactly what they are in the next video see you there to be sure not to miss the next tutorial in this series subscribe to my channel thanks for taking part in this tutorial I hope you enjoyed it if you did please like and comment to be sure not to miss any of my content head to John chrome comm and sign up for my email newsletter you're also welcome to add me on LinkedIn simply mention that your viewer of the machine learning foundation series and finally you can follow me on Twitter too if that's your social medium of choice see you next time