Building Recommender Systems in Python | Machine Learning Tutorial

[00:00:00]so thank you for joining me everyone over here so today I'm going to talk about recommender systems so first a little bit about me um so my name is uh Divia sardana I am doing my PhD in computer science uh in the area of machine learning from uh University of Cincinnati back at Ohio and uh my dissertation area specifically is in uh graph mining so what I did in my dissertation is I built a new graph cluster ing algorithm and then I applied it to different real world data sets mostly protein interaction networks and then some social network and recently I applied it to a GEOS temporal network of all the crimes taking place in San Francisco I and did some clustering and more analysis over that um so today I'm going to talk about um recommend a systems using python okay let so let's get started so here is a brief outline of my uh talk today so first we I'll talk about why do we need recommended systems and then um we'll discuss some examples of uh recommender systems then we'll move on to building some real world recommender systems so we'll cover of three types of uh categories of recommender systems today popularity based recommended system then classification based and collaborating filtering based and then finally we'll discuss a measure an evaluation measure to evaluate different types of recommended systems okay so let's get started so uh why do we need recommended systems so uh let's begin by thinking about the traditional uh view of how the on the shopping used to take place earlier so this is an online so this is a traditional bookstore wherein uh we have a limited set of books which are limited by the brick what can fit in the brick and mortar store and uh sometimes the popular books would be listed over here in the center so U it was easy for the Shoppers who used to so it was easy for the Shoppers who used to just go to the store and pick a pick from whatever is lying there uh inside the store and buy whatever they liked but now coming back to what's going on today is that we have these huge set of products online and U there could be some for let's take the example of music for example so somebody could be selling some music online which is very old music from 50s which some traditional tribal African music so people there is like now a set of very Niche products which fill the market today so this leads to a phenomenon something called as the longtail phenomenon where these orange set of products represents the popular products which we can find find both in retail stores as well as online and then there is a huge set of products which we find only in online stores so there we need some system which tells us that these are these will be relevant items for you to look at as compared to uh just bombarding the users with so many choices so what a recommended recommended system does is the goal of a recommended system is to identify products which are most relevant to the user for example top 10 promotions top 10 news articles okay so some examples so what are some examples of real world uh recommended systems anybody in the real world today Amazon Amazon yes how about movies Netflix yes let's see yeah movie or TV show recommendations like from Netflix then uh product recommendations uh by Amazon then friend recommendations in Facebook right and then job recommendations LinkedIn uh sends you many emails almost every now and then that uh these jobs match you perfectly and then um okay so before we move on to building recommended systems let's first gain a naive understanding of what a recommended system means so let's think about in the real world when I'm uh going to shop something how do I go about uh this shopping process so let's see so let's say I today I want to go eat something something new so my friend here tells me that hey I ate this new dish at some new open newly open Tha restaurant so you must go and try that dish out and then I think okay my friend is recommending this to me and I know that my taste is similar to my friend and I I should go uh to that tire resterant and try that dish out and then sometimes my sister would tell me that uh I bought this dress from this store and you must go buy this dress it's such a nice dress and I will think okay I will take her recommendation and then uh sometimes uh my professor will tell me that uh I you must read this book it's a very nice book about machine learning and uh I might take his recommendation and read that book so you see what we have in our system is a set of users on the left hand side and then a set of items on the right hand side so the the users they might be related to each other and then the items might be related to each other for example different books on machine learning are related to each other dishes related to a Cuisine might be related to each other and then there is a mapping between users and matching items and then a recommended system it exploits this mapping between users and the matching items in order to make recommendations so so it is it is just the similar as we we will do recommendations in the real world it's just that now we are doing it uh in a clever way using a machine learning algorithm called recommender system okay so a quiz now so uh let's go go back to the uh real world examples which we talked about before so uh what are the users and magic items in the following cases LinkedIn so what are my uh users in LinkedIn the members members of LinkedIn and what are my uh items in LinkedIn jobs yes so what about Facebook what are my users here members and what are my items here members again members yes and what about Amazon the users are again members and items are it items yeah products the books and whatever Amazon is selling right and then how about Netflix users are members and then items are movies yes so uh yeah so this is was simple okay so before we move on now I will I want to talk about this uh success story of recommended systems that how it can have a huge impact in in the market so uh for example so these are two books about uh mountaineering Adventures so this book on the left hand side called touching the void came out in 1988 by the author Joey Simpson and it was uh it became a modest success at that time and then people they forgot about this book later on and then uh another book called into thin Lair it came out by John Crocker in the year 1996 and it became a huge sensation and then what happened was that Amazon it picked up that people who were buying this book about mountaineering Adventure called Into Thin Air they were also buying another book called touching the void and then suddenly what happened almost a decade decade later is the other book touching the void instantly became famous a decade later so you see the U recommended systems if used properly they can have a huge impact in the Market okay so now we can move on to um building a recommended system so uh we'll start with the very simplest solution uh to build a recommender system so uh it's called a popularity based uh recommender systems so uh let's first consider this example about uh popularity based recommender system that uh we all see when we read news daily so you see on Google uh news when we read news we see it lists these are my most popular news and then when when you go to some uh clothing stores say coals it would it might list these are my popular products so how do they calculate what are the popular products a very simplest approach would be to uh calculate popular product simply by account of purchases so all those products for example in this uh dressing store store the uh popular products could be calculated based upon products which are being bought the most does that make sense right so and similarly in news the most popular news would be the news articles which a lot of people are reading at in a in that during that day so essentially my recommendations here are a ranked list of items by their purchase count right so okay so now a quiz so which one of the following is true for a popularity based recommender can it generate personalized recommendations yes no correct so for example if I want recommend what are the popular news for me versus you news popular news for the day would be the same right um okay so the correct answer is no okay second can it use context for example time of the day I mean um so we could say that these are my most popular items during the morning versus the evening and for example some Shoppers they purchase uh these popular items during the evening uh Friday evenings so uh we could say that it could use uh uh context in some way can it use uh user features so user features would be what is the age of the user uh gender of the user can it use user features so I would say the answer is it it can use uh user features but it won't be very meaningful the only for example what are the most popular items amongst teenagers versus um 30y olds so but it won't be it is just that the meaning of that it it it is possible to do but it may not be very uh intuitive and then uh can use item features so item features would be like uh what is the size of my item what is the color of my item cost of my item so again it uh theoretically it is possible to do that that what are the most popular Nike uh uh clothes clothings but it may not be very meaningful to suggest those items on a website and can it use purchase history so yeah so we are using purchase history uh like the most purchased items in the past it uses Pur purchase history of the users and how about uh scalability is it scalable yes so simply we are just sorting all the items by their purchase count so uh yeah so we just need to do sorting so uh it is scalable yes okay so uh let's move on to our little more comp complicated solution now U so a second way to build a recommender system could be to use a classification model so you know about uh linear simple linear classifications like logistic regression or uh nasan classifier so how about we use such type of classifiers to build a recommender system which um so for example we have uh on the left hand side the input to the classifier could be my user features such as age of the user gender then product features like cost or quality of the product and then purchase history and we could have these as my uh input features in my classifier and then the classifier could give an output uh just a binary output like like uh a new user which comes in will will he like the item or not like like the item so for example it we the classifier could help us uh separate All U teenagers who have some fashion uh very different fashion style as compared to all 30y old people who go to office so such type of classification method can help us uh categorize build these categories which we could exploit to build mappings between users and items and hence uh build recommend systems but what is uh a limitation of this approach uh is it uh how easy or difficult is it to collect such data oops [Music] so uh in a real world scenario it is difficult to collect such uh such information okay so a quiz now we we'll keep going on so uh which of the following so which of the following is true for a classification based recommender system uh can it generate personalized recommendations yes so it is using user features item features so yes it can generate uh personalized recommendations uh can it use context for example time of the day yeah so time of the day could be one feature in my classification mod input classification model right okay all right so can it use user features yes so it is it can and then how about item features yes in the uh classification model can it use purchase history yes it can and then how about scalability it might not be very scalable because uh when my features and uh when lot of uh data comes in then it might be difficult to build a classification model and that to keep on updating it when people when in an uh very online manner okay so uh the next set of recommender systems would be my uh nearest neighbor collaborative filtering so uh these collaborative filtering types of recommended systems the recommendations are based based upon uh the nearest neighbors so uh so nearest neighbor approach is used to find out similar uh either similar users or similar products so this type of uh U let's try to First gain an intuition of these type of methods uh so by let's look at this figure so this is about people some people and uh what fruits they uh buy from some store so this problem of collaborative filtering can be looked at in two ways one way would be to look at the similarity between users and the other way would be to sim look at the similarity between the items so the Let's uh look at the first way F uh first so we can say in this left side figure that the girl the first girl and the last boy they are similar in some terms because they have a similar taste in what fruits they like so for example both of them have purchased strawberries and watermelon so we can say that they have a similar taste so now uh this uh this so we can use this uh similarity information in order to recommend to this boy whatever the girl was eating and so the grapes the the boy didn't try grapes yet so we can recommend grapes to the boy right so exploiting similarity between the users and then let's move on to the figure on the right hand side we will say that uh grapes are similar in some sense to watermelon because they many people have been buying them together so of course there may not be like a real similarity between the fruits but uh we we they are similar in terms of recommend a systems because many people are have been buying them together so the two girls above have been buying these two fruits together so therefore when uh the boy here he purchases watermelon we will recommend him the next nearest neighbor to watermelon which in this case is grapes so uh what so what exactly um in this approach we would say so in the left hand side approach or the user based filtering we would say something like user X is similar to user y because both purchased items a b and c but uh about item based collaborative filtering what we would say is that items A and B now are P were purchased bed both by users X and Y so they are similar right so you see the difference in the two ways of looking at the same problem right so uh in this talk I'm going to focus more on uh item based collaborative filtering but uh in similar ways we can map the same problem to uh user based collaborative filtering as well okay so let's uh go into more detail of item based collaborative filtering and uh let's talk about here an examp sample of movies so uh on my left hand side we have uh some use sample users Ted Carol and Bob and uh the movies they have been watching movies a b and c and let's call this as our history purchase his history uh Matrix in this case so uh so in this method what we do given a history Matrix is that first of all we build a cooccurrence matrix what do we mean by a cooccurrence matrix is is that we take all the items here or in case this case all the unique items in this case they are the movies and we uh create a matrix between all possible combinations of uh items and what are the values of of this Matrix so um the value is that how many users have seen both movie A and movie B so so in this case for movie A and B we can see here that uh Ted has watched only Ted has watched both movie A and B therefore we will have a one here and then how many users have watched both movies A and C you can say Ted has washed both A and C then Carol also has wased both A and C so therefore we'll have in The currence Matrix 2 here so now we the question here is to generate recommendations for Bob so what we do here when we need to generate recommendations is that we see what all movies Bob has already watched in this case Bob has watched movie A then what we will do is we will we will see in we will go to the cerence Matrix now and we will see the row for uh this movie A and then we can sort this Row in descending order and uh then we can give out the recommendations for uh Bob here so what is the intution behind sorting them in descending order is that uh it it will give us an tion that uh what most people have been viewing we want to recommend them first right okay okay so in this case Bob's recommendations would be the top recommendation for Bob would be C and then uh the second best recommendation for Bob would be B right is that clear okay so uh so now let's uh look at uh some effect of uh very popular movies so let's take an example of a very popular movie let's say Titanic which uh let's say all our users almost everyone has seen this movie so uh so in let's so in The currence Matrix when we get this movie in The co-occurrence Matrix we can see that uh it this value may be very high for almost all the users right because uh every person has watched that movie so the cooccurrence of this movie with each other movie will be very high so uh what now if we want to recommend movies for Bob we will look at the row for uh the movie which Bob has already seen in this case movie A so what would be the movie recommended to Bob in this case Titanic right but uh so so this means that this recommended system would would almost behave like a popularity based recommended system so whatever my popular items are they won't be they would only they would be recommended so we don't want this it would not be very interesting so how do we deal with the effect of such popular items how do we do that so um most frequently uh what what we do to deal with this problem is normalization so we normalize our cooccurrence matrix by popularity so generally uh there are many methods of uh normalizing the very uh basic and knive method of normalizing ing is this Jacker similarity measure wherein uh we will divide all the U all the cells of this cooccurrence matrix by the number of users which have bought which have uh watched either of movie A or movie C so remember our numerator here was Mo uh the number of users who have watched both it is the intersection on the uh numerator both A and B it was one in this case in the the denominator now we have a union of users who have watched movie A or movie B so that in this case is three right so it will be divided by three this case would be divided by two and uh the Tit for the Titanic case would be divided by 100,2 it would be a very small number now so now what would be the topmost recommended uh item for Bob here yes so it would be C so that is what we wanted so you see normalization has helped us um deal deal with the effect of very popular items okay so uh let's consider another case now in which uh Bob has watched more than one movie now so earlier Bob had only watched movie A now Bob has watched more than one movie so in this case how do we recommend more items to Bob so remember earlier we were looking at the only one row in The currence Matrix for movie A now we have Bob has watched another movie called movie D so what we will do is that we will look at not only row a in this movie and another row D Suppose there was another row here for movie D we will take out both the rows A and D from this Matrix cooccurrence Matrix and let's say we list them out here and remember it's a normalized matrix this time normalized cooccurrence Matrix so uh then what we do is that we simply take a weighted sum of these uh columns and list them out here so in case Bob has watched so many movies we will take all those rows out and then we will simply take a normalized uh sum of all those rows in order to get our final recommendations so in this case the top most recommended item for Bob would be movie C right okay so now a quiz so uh given a user item ratings Matrix of size 480,000 cross 177,000 approximately 177,000 which model will you apply given the Matrix is very sparse uh popularity based recommended system how about that um so I would say maybe we we could apply that but uh it would it wouldn't make any sense here because uh we could simply take it it could simply tell what are the most uh highly rated movies um but it won't help us much here and one thing we need to uh keep in our mind that is that this Matrix is uh very sparse as well okay because to the top right yes the yeah the top top high if it weren't SP then that would possibly be a very good choice right uh it would be a good choice but it won't be personalized I would say so um depending upon our problem um these days we we we want personalization is a big thing so it may not be very useful being the key word on popular that has right say again sparse is the key word that you're looking for when you're trying to decide M yes yes yes okay so how about a classification model no so it uh it again uh may not be the best uh best one to do okay how about an item similarity based recommender system remember we just talked about it in which the item similarity Bas recommend a system wherein we need to build a cooccurrence matrix so what would be the size of my currence Matrix in this case 177,000 cross 17,000 and moreover uh in the previous slides when we were talking the currence Matrix simp L were was uh the history Matrix simply listed uh whether the user has watched a movie or not it was a binary Matrix over here this history Matrix on the left is a binary Matrix now we are given ratings so if we are using item based similarity model we are not making any use of these um ratings right and similarly user based uh similarity uh user similarity based recommended system again may not be uh the best choice in a similar way uh so all of the above is definitely out of the question and uh seems like the answer is none of none of the above if we don't count popularity based recommender systems so you must be thinking whatever we have uh done so far it it may not be very applicable in the real world Choice cases wherein we might have be having many sparse U matrices out there so uh why did I choose this m Matrix uh 480,000 cross 17,000 anybody knows about this Matrix yeah so this is a million dollar matrix it's worth million dollars it uh it was released as a part of Netflix competition uh somewhere back in I think 2006 it was released that and the goal was to uh come up with a a new algorithm for recommended systems which would um improve the accuracy of over the exist exting Netflix algorithm by more than 10% and um and this Matrix over here has about 100 million ratings you think that's a lot of uh ratings no seems like it is not a lot of ratings because um only it means that only so the total possible uh ratings here is 480,000 multiplied by 177,000 which is 8.5 billion ratings and out of that we have have only 100 million nonzero values which is just 1% of the whole Matrix so it is indeed a very sparse Matrix so uh what do we do in such cases when we have uh sparse matrices so what helps in this case is our uh third solution fourth solution third solution Yes actually here which is a model based collaborative filtering uh method or a matrix factorization um based methods so what U so another problem which which would occur which I which we didn't which we missed in this slide is that uh it would be uh The Matrix is spse that is one thing another thing would be in order to store uh an item cross in order to store in case of item similarity based method storing uh 177,000 cross 17,000 Matrix uh uh each time when we are using the model it may be very inefficient so uh what model based collabortive filtering methods they do is that they take the input user ratings Matrix and then they um build a model in a low ranked dimensional space and U and they generally used some existing Matrix factorization based methods so what do we mean here by a low rank dimensional space is that they uh these Matrix factorization methods they identify a hidden dimens hidden Dimension or in a way we can say that they identify latent featur from the input uh user items ratings Matrix to represent user and items as vectors in N dimensional space Let's uh try to understand this using this example over here so what do we mean by user vectors and item vectors so um intuitively what our Matrix factorization here is doing is that so we have this input user ratings Matrix it has many it is very sparse right and it has uh but it it still has uh some nonzero uh entries so it will uh take that user ratings Matrix and convert into a user Matrix of vectors and a um items Matrix of vectors what are my vectors here so my vectors here user vector and item Vector could look something like this so in case of movies it is telling me that how much that user likes these serious movies so what is the yre of my movie it would uh it would be so for example my different yandra could be how comedy horror or in this case serious Escape as geared towards males or females so um if you know about uh um information retrieval based classes wherein documents are represented by vectors you could think of it something in in a similar way wherein uh we are representing users as vectors uh where in each uh value in in our vector is telling us that how much the user likes uh each how much the user can be represented for each yra and then uh when we represent these vectors in these in this low dimensional latent space over the left which is a in this case is a twood dimensional space we can we can see that we could represent all the users in the space and then similarly all the items also could be in this case the movies could be represented here um as well so what happens now when a new user comes in so so new user comes in and we remember we have this input Matrix of users and items and then we will um and now our Matrix factorization has given us user vectors and item vectors right so we will use that Matrix factorization to get a representation of this user as a user vector and then this user can be represented in the same latent dimensional space and and when we represent this user like that now we know in this space that what all are my uh movies which are located near me in this latent D dimensional space and then we can uh exploit this information to uh recommend movies which are close to the user in this uh hidden dimensional space right okay and uh just going into a little U detail about Matrix factorization so most common method of doing Matrix factorization here uh that is uh very uh used in used as uh singular value decomposition so uh let's think of Matrix factorization in a way that uh we all know about uh integer factorization right may we we read about it in our uh elementary mathematics that for example how about integer 12 what are the prime factors of 12 3 yeah 3 two and two so uh you see if we Factor ize an integer the 12 we we get prime factors two uh 3 2 and two what do these prime factors tell us they tell us that any multiple of 12 say 144 will not be divisible by say five because five is not a prime factor of 12 and no matter whether we represent 12 in binary form or octal form hexadecimal form 12 will always have these prime factors right so U so Matrix factorization it tells us about some Universal properties of that mathematical object irrespective of how we represent that mathematical object and then that uh factorization helps us to gain some intuitive ideas about that mathematical object now we transfer that our knowledge about integers to matrices over here so in the similar way when we uh factorize a matrix using some Matrix factorization method then it tells us some interesting information about in the user space so in in the from the previous example it was telling us some interesting information about the users for example this user likes comedy movies very much and he doesn't like horror movies a lot and we are quantifying it by assigning some values to how much he likes comedy movies how much he or she likes horror movies so this Matrix factorization is in a sense what it is doing is that it is extracting this most interesting information about users and items and then this interesting information it is quantifying it and this quantification is in some way helping us to predict the ratings uh for new users right so uh remember this Matrix was a very sparse Matrix the sparse Matrix will be used to by The Matrix factorization method to get this user movies uh m m matrices of user vectors and movie vectors and then uh these matrices will then be used to fill all the zero uh Val cells in The Matrix in order to make recommendations right so um yeah this is what I just talked about that mathematically we can represent the SVD by this uh in by this uh mathematical formula and then um so the winning entry uh of the Netflix price what it used is uh a number of SVD models including some SVD Plus+ or restricted boltman machines and uh they achieved I think slightly more than 10% accuracy over the Netflix existing information and then I have listed a paper here where in it where it talks about if you want to read more about the Netflix price and how the Matrix factorization method was used here I would recommend you to read this paper Okay so okay so uh now some performance metrics uh for recommender systems so um the most common U used performance metric here are precision and recall so let's understand this uh by an example so let's say I only like Will Smith movies and and I don't like any other movies and uh what but but still what I was recommended was I was made some recommendations for some Will Smith movies and uh some Johnny de movies as well so what are my relevant items relevant items meaning all those I like so the relevant items will be all these items all the Will Smith movies and the recommendations will be a combination of some of the Will Smith movies and and some journy de movies right so uh so these here are in this cell represent relevant items that are also recommended and this cell represents uh irrelevant items or the journey dep movies that are recommended which I don't like and then this cell recommends the Will Smith movies which my recommendation algorithm didn't um didn't recommend me so how is precision calculated is it is calculated as the uh number of products which are relevant and recommended in this cell divided by all the number of items that are recommend that are recommended so in this example all the Will Smith movies which are recommended divided by all the Will Smith and Johnny de movies that were recommended to me and then recall how is it calculated so Precision is in a way a measure of exactness how exact are my predicted recommendations and then recall uh here would be calculated as number of products which are relevant and recommended in this cell divided by the total number of relevant items Al so it in a here in this example all the Will Smith movies which were recommended and then all the Will Smith movies which are actually like so intuitively what recall is doing is it is a measure of completeness that whatever movies I liked has have every every one each each one of that movie has it been recommended or not so uh these are two important uh measures of evaluation and uh then what we mostly generally do in order to compare recommended systems is that we uh build uh Precision recall curves so what we uh normally what is is done is that we uh recommendation algorithms they will recommend top n items so this algorithm this curve what is it it is representing is that so we will calculate Precision recall for say top one recommendations for all the users we'll plot a point here then top two recommendations we will take for all our users we'll be then we will calculate average Precision average recall for all the users for top two recommendations we will plot a point Point here so what this is representing is that as we go from Top K recommendations to top K plus one recommendations how is the quality of recommendation engine uh changing so why is this this sort of pattern so think about it that if we go from uh top K recommendations to top k+ 1 recommendations two things can happen the k+ 1et recommended item would be either uh whether I will like or not so for example in our previous example if the K plus one item is a Bill Smith movie which I like then both precision and recall will go up and this represents this uh increase this in this sort of pattern and uh what if the recommendation in this is in this case was a Johnny de movie so uh Precision would fall by one like this but recall would remain the same because all the relevant items are that have been recommended the will smth move these are still the same so the so it so it will have no matter what our recommendation system is that we will get these either we will go high like a spike where both recall and precision increase or we fall down wherein recall Remains the Same or Precision uh drops by one so these patterns will uh will be the same in this curve okay so performance so there are some other performance metrics also um which uh such as uh mean absolute error accuracy and Roc curve and uh I would recommend you to read this um paper I have listed it has a very good survey of uh evaluation metrics for recommendation tasks and uh I also have a following notebook wherein we will be uh using precision and recall to evaluate uh different types of uh recommended systems okay so this is my last quiz here so comparison so which recommend a movie uh can uh can handle uh brand new items for example a little uh for example the newly released movie so this movie I think uh this little men movie was uh released I think just last Friday so uh what do you think that which recommendation system should be would be the one which would pick that movie up and uh display recommendations so this last column so how how about popularity based recommendation system would it be will it be able to recommend My Little Man movie no right because uh this movie just doesn't have any viewing history so far so how can how can it become popular so soon I mean it may be very popular movie on the first day many people went but as compared to many other movies which have which have which have been released since a very long time they may be having such a huge count of view count but this movie was just released it is not in our our system yet so maybe popularity based recommended system will not handle be able to handle this newly released movie how about a classification based system well it it will not be able to use purchase history but it could use some features of the movie for example what is the cast in this movie what what are my actors in this movie what type of movie is this is this a comedy movie horror movie and then uh based upon that we could find out we could feed that in our classification model as features and then we could get uh similar movies uh based upon just these uh these uh features and not considering uh the purchase history at all and still yes so this classification model based method could help us right it makes sense right so how about a nearest neighbor based classification system let's say item similarity based classification system wherein we were building a CO Matrix of all items uh based upon co- purchase history so do you think this movie will have any co- purchase history not really right because it is so much of a new released movie and um there will be many few users who will have be have watched so it will be it will not be very fruitful to use an nearest neighbor based collaborating fil filtering algorithm for this movie how about a matrix factorization based movie based recommend a system so again since it's a new movie not many people would have rated it right so it may not be again um a good idea to just simply use a matrix factorization method so it seems like this problem has a name it is called a cold start problem so uh so now in the real world how it is handled is is that say our model is using a matrix factorization based method to build a a system which is being used on a day-to-day basis and then when new products come in then we use uh we handle them using a say a classification based method so these models are called as hybrid based recommended systems wherein we are taking more than one approach and we are uh combining them we might use one in some circumstances another in some other circumstances and uh and then uh use both of them to give our final results okay yeah so uh we are ready to go to our uh notebook wherein we will see recommend some songs so this uh uh data set that we will be using is a million songs data set so this is my uh link to the notebook here so you can either download or clone this directory so for cloning there will be a link to the right side of my GitHub you can copy this link and uh just use if you have git you use uh git clone and then uh this name of the directory or you can simply download it and unzip it uh so in Python uh I built um some some of my own uh classes to um for uh building a popularity based recommender system and an item uh based similarity based recommender system and uh and we will be using that to recommend songs from the million songs uh data set and um then we will be uh also doing a small seeing a small demo of Matrix factorization based method on on a very small dummy Matrix so as to gain more understanding of how the uh latent semantic space is working and then there is uh also in Python an existing uh package called crab don't know if you how many of you know about it it's uh it it also does uh the job of doing recommendations for um is a package for doing recommendations in Python if you want to check that out yeah okay everybody has this link shall I move away from it okay so I already have it uh on my system okay yeah so this is the uh million songs data set Source what I was talking about so uh we will be only working on a subset of this data set so because uh it will be it takes a long time for the code to run on uh the whole data set and U um okay let's move on so so what this first cell is doing is it is we let's import all the methods all the libraries okay so now we will uh load some music data so this step might take a while uh on your system so uh I didn't uh make a local copy of this data set because I got this the original million songs data set is uh an open- Source data set by Columbia University but uh this data set was a subset available uh at this site by Turi or datto and uh uh I wasn't sure I could uh keep that as a local copy so it might take time for you to get this step done but uh meanwhile I will just go ahead so let's uh explore this song data how does it look like so this data here has uh about songs which so we have user ID song ID listen count then title release artist name and year so it is in a way what we need from this data set are these triplets user song and then listen count so what what is my user what what is the song the user has listened to and uh how many number of times the user has listened that song okay so let's see what is the shall I move forward is the data set loaded for you yeah it might it might take some time uh so uh especially we are on Wi-Fi here and and everybody is trying to download from the same uh link but uh it should not be more than uh 3 4 minutes I think so uh initial steps are anyways just data exploration so it's okay if it it's taking while you can uh follow me over here okay let's see uh what is the length of this data frame it's uh 2 million and uh we'll create a subset of this data set so that we can um run it all here at one go so I just changed this uh 100,000 to 10,000 here because I think 100,000 will also take a long time here given the uh um so let's uh so at this step create the subset of the data set just change this to 10,000 and then what I'm also doing here is that I'm creating a new column in our data frame uh which which combines title and the artist name so as to make our recommendations uh so that we can view them a little more meaningful way okay so what are my most uh popular songs here based upon their listen count okay so uh here I have simply what I've done is just sorted the songs based upon how many the listen count how many uh users have listened to songs and then calculated a percentage uh rank list of all the songs Okay so let's count uh the number of users unique users in the data set so we have 365 unique users in the data set right okay so a quiz for you so how about we now count the unique number of songs in the data set how will we do that so so we similar so we can a hint is that follow how we calculated the unique number of users here we can take help from this code over here and uh instead of user ID we just can simply type Here song and uh this will give us the unique songs in the data set now we can check uh let's call it songs okay yeah so um we see that there are 365 users and about 5,000 songs so what does this tell us this tells us that U each user may be listening to more than one song right because we have the number of songs is more than the number of users right so what does this suggest that there is it is possible to find out similarity between users based upon how many similar songs they are listening to right because uh it could be possible right at least we know that it is possible now right okay now how to do that let's figure that out now okay so what we will do now is we will then our split our uh training Set uh into training and our data into training set and test set keeping an 8020 ratio so uh how how are we doing on the data download is everybody got the data not yet okay when you get that data I'll I'll uh repeat that uh make the subset of the data set over here in this cell to 10,000 instead of 100,000 so that it will run um smoothly for everybody right okay okay so now we will uh create a popularity based recommender system for uh users so we will uh create an instance of the popularity recommender class and then we will use the popularity model to make some predictions okay okay so based upon my most uh remember we had uh also listed the most popular songs in the beginning and uh and here again we have calculated a popularity uh for this user which is users of five this method is calculating recommendations for this user okay just to get an idea of how it is working so this is a quiz for you to calculate recommendations for another user now and to compare how do how do these recommendations compare with my earlier recommendations you want to try it out so what we can do is we can simply use our previous code to recommend so earlier we were recommended for users of five and now I will do the recommendations for user ID a different user ID uh user of seven so how do my recommendations compare with the earlier recommendations so it is sah cos Mish by harmonica and Uno by Bor these are my top two recommendations for users of five what are my top two recommendations for users of seven is the same so what does it tell us about a popularity based recommend system it is not personalized right so um okay but it is very simple to calculate right that is the advantage just um rank list by based upon the listen account and that's it it's easy okay so now let's build a song recommend recommender with some personalization so we what we will do here is we will um create an instance of this uh class over here so far are we going good here a question link for GitHub okay yeah this is the GitHub link yeah I think it's on the website think I'll open it here this is yeah so this is the link can you see this now okay so everybody has a link now okay I'll go a little slow now that's fine shall I move over to the notebook now okay I'll quickly repeat uh what I did so far so as to for those who are still catching up so the first cell we will uh import uh all the libraries this includes uh the libraries for recommenders as well as evaluation so the last two libraries these are python files which I wrote and you can read more about them in the from the GitHub page they include classes for different recommender systems as well as uh the evaluation class contains code for uh Precision recall calculation and then uh this is a subset of the million uh songs data set which we uh can um down download it this what this cell is doing is is it is taking this data from uh the story website and uh so the first file here the triplets file what it contains is the triplets of user song and listen count and the second uh file over here songs metadata what it is doing is it contains some information about the song like artist name title and release uh these different kinds of metadata information about the songs and so what I did is I read use pandas to read table and then uh read this triplets file as a data frame and uh then I read the song metadata as a data frame again and finally what I did is I merged these two data frames in pandas to create one single data frame which contains not only our triplets but also the metadata about the songs which we have in the triplets right and then we look at the head of the data just uh to see what is a snapshot how the data looks like we see that we have uh so so far we are good till here at the back okay great so we have user ID song ID listen count title release artist name and then year information okay and then the L length of this data frame is about uh 2 million this means we have 2 million triplets here of user ID song ID and a listen count so what I what we are doing here so in your data in your notebooks in this step uh it would be a number here as uh 100,000 where wherein we are taking a subset of this data so that it runs quickly over here so I I would suggest you to change it to 10,000 over here so that uh our uh code will run smoothly here right are we um fine till this step right so when we run this step um then we our subset would be created and then in this last line here what I'm doing is I'm creating a more intuitive so I'm combining the uh columns in the data frame for title as well as artist name into one column called song So as it will be more intuitive when we check our recommendations that uh the song title and the artist name would be in one column so we see the most important columns in here with us our user ID represented by uh this user ID column and another column called song column which I just created here so what are Rec commender system we'll be looking at here is mappings between the user ID column and the song column which I just created right so essentially the recommendation system what it does is it just looks at the mappings right okay so what this cell is doing is over here are we good till this cell okay so uh what we'll do here is that we will see the popular songs based upon the listen count count what I did here is I uh simply sorted all the songs based upon their listen count so we we get that the S Cosmos by harmonica so you see in the song uh column which we just created we get um the song name as well as the uh artist name together so um these are my sorted list of uh songs popular songs so any questions so far okay okay so now next we count the number of uh users here unique users so we have 365 unique users in our data so far and then uh we count the unique number of songs in the data set uh we have about 5,000 songs in the data set so as we were just discussing that this suggests to us is that the number of users is greater than the number of songs so this suggests that at there are some songs which are being repeated which are common in between users listen history right so this means that it is possible to calculate similarity between users based upon the common songs that they are listening to and this was our criteria remember when we talked about building an item similarity based cooccurrence Matrix then it will be easy to build uh our Sim so we will be able to build similarity Matrix um cooccurrence similarity Matrix where wherein we will have all the songs on the rows and then all similarly all the songs on The Columns as well and our values for the cooccurrence Matrix would be number of users who have listened to both song I as well as song J remember so uh this suggests us that it is possible to build a cooccurrence matrix um okay so um so if there would be no overlap if there were 365 users and 365 songs maybe not um so me means that we can map one user exactly to one song right if there were 365 users on my left and 365 songs on my left maybe we can do a one-on-one mapping right but now we have only 365 users on my left and about 5,000 songs on my on my right so so if we are mapping one song to one user to one song we draw a line like this so when when we are finished with all the 365 users we we would have used just 365 songs from our right hand side list right and we will still have 5,000 minus 365 songs still left which have been listened by some users right so so this so this can only happen if one user is listening to more than one songs I mean it is possible that all those remaining songs were listened by only one user and uh there is no overlap so uh actually it is possible that there there is no overlap uh uh between the uh songs user are listening but uh uh it is very highly likely by I I would say I won't say that it is for sure because it is possible that all the remaining 5,000 minus 365 songs were listened by just one user it is possible yes that there is no overlap yeah you're right uh but it is but this suggests that it is very likely that there is an overlap and since I have already seen the data set I know that there is an overlap so uh we'll we'll see now how how it works okay so we'll create a song recommender now for that we need to we will uh split our our data into training set and test set and usually we will follow the usual rules of creating training and test data wherein we will have 80% of the data as training data and uh 20% as test data so what we will do is uh that we will build a recommended all our recommended systems on our training data and then uh we will evaluate them using our test data using precision and recall curves so U the same model built on training data will be used in the test data users to get our evaluation so our training data just to give an idea looks like this just ahead of the training data and so let's build a very uh the the simple popularity based recommender and um uh this so we'll create an instance of the popularity recommender class so this uh popularity uncore recommender _ Pi it is a class in the recommenders dop file that is in GitHub so this step is simply creating an instance of that class right uh we are good till here okay yeah so uh next what we do is we will use the popularity model to make some predictions and we see that uh these are my top songs over here for users of five the SAR Cosmos by harmonica then Uno Bor by Bor Dog Days Are Over by Florence then you are the one by Dwight Yokum and similarly uh all these other songs then let's take another user now which is users of seven in this case and make recommendations using the same popularity based recommender over here and uh what do we see here what are my recommendations here they are the same as the previous case right so this suggests that there is no personalization in the recommend in the in a popularity based recommender system so no matter what the user is you could change it to any other user and uh we will still see that the same recommendations are being recommended right so are we good till here right okay great so now next we will build a song recommender using personalization what we talked about item based item similarity based personalized recommender system so as I was just talking our cooccurrence Matrix here would be a song by song Matrix where in each cell would be a normalized value of listen count right so let's create an instance of this class and and then again uh we will recommend for the same user users of five which was used in popularity based recommender case let's recommend make some recommendations okay so it's doing the calculation here so okay the results are here so um so in these results we see that uh it says training data songs for this user ID so users of five which is the user in question this is the ID for that user and we can see in the training data that these are the songs which this user has been listening to so far and we have to make recommendations more recommendations for this user so we see that there are some songs by &m some songs by Lady Gaga salt and Peppa and Destiny's Child and so on right and uh we see what are my recommendations here okay so uh we so we can see some songs more songs by Eminem some by Justin bber and uh yeah and then also we can see that these are my score also here um and one thing I would like to point out remember when we were considering the case of uh in the slides uh Bob when we had to make recommendations for Bob so we had when Bob was watching just single movie then we took the row a for Bob and made the recommendations now think that this user of five is Bob here and he has watched the all these movies over here how many movies are these 13 uh unique songs for this user already so how many rows of cooccurrence Matrix will we take in this case 13 rows right so we will take the 13 rows from The cooccurrence Matrix corresponding for these 13 songs and we will calculate uh we will get the cooccurrence Matrix value of all these songs with all other possible unique songs which in our case is 5,000 right so we will construct so our Matrix here would be um that we are considering here is 13 cross 5,000 right and we are calculated um so we will uh normalize we we will just calculate a weighted sum of all the columns in this 13 cross 5,000 Matrix uh to get our weighted sum and then whatever weighted sum we will get we will calculate we will rank them to get these final recommendations that makes sense how we are mapping the toy problem we discussed earlier to a real world uh data set it makes sense okay great um and the those weighted scores that I just talked about these are those weighted values which are which have been sorted now to give the rank of recommendations okay so how about now you try to calculate personalized recommendation for this user users of seven so we can make use of the previous code how about I copy the same code from here and I fill in here okay just run it okay so it's still calculating but uh before that we can see that this is my user ID so in this case this is my Bob over here the ID for Bob in from based upon what we learned in the slides and uh these eight songs are the songs which Bob has already listen to so uh what would be my cooccurrence Matrix in this case 8 cross 5,000 right so these eight songs we will take the rows for these eight songs in The cooccurrence Matrix and we will uh calculate a weighted sum of these uh uh normalized values right and then this uh weighted sum we will rank it we will sort it to give the final scores over here so looks like this uh in this case Bob only likes gold play songs no other song and uh let's see what our recommendations are in this case so some songs from Coldplay and maybe some other songs by some other singers as well I don't know much about other singers but um I I can't say if they are relevant or not but our data was very small so um when we do it I was also running it over a larger subset of data at home and I saw that there were further more Coldplay songs being recommended on the larger subset of the data but here since our data is small we see some other songs as well so uh does this make sense right okay so uh next we move on to another uh usage of the same model item similarity based model in this case we are going to use our model to get similar songs to Any Given song so um let's say we get similar songs to this song You Smile by Justin bber okay okay seems like there are um quite a few songs by Justin bber which are coming in as similar songs to this song and then some other songs as well okay so how about a quiz for you to calculate similar songs for this song Yellow by cplay so what we can do is we can again copy the same code from here from our previous case and let's see you can paste it here instead of this you smile song now we have this Coldplay song and uh let's run it okay yeah so these are the similar songs to this song so in a way these are the nearest neighbors of of that of this cplay song based upon similarity in purchase history so what we are doing here in uh when we are in the model what what it is the code what it is doing is that it is picking the row for this song Yellow by Coldplay in The cooccurrence Matrix and so in this case we are taking one cross 5,000 so for this song We Are calculating we are getting the cooccurrence uh Matrix values for just one row corresponding to Yellow c playay song and then U these normal ized values we are simply sorting them and then the sorted list by score is being listed over here as our final rank list right okay right so next move on to comparison quantitative comparison between the models so we will compare the two models which we just built uh popularity based and the item similarity based model so we can run this step it will I think it will take a almost a minute to do this calculation right this step so what this step is doing is over here it is uh making record so what uh in order to calculate precision and recall what exactly we are we have to do here is that so our train uh models both popularity models and the uh item similarity models were created using the training the training set now we also left out some data in the test set earlier remember 20% of the data is test set so what what I did was that from the test set I took out users which uh using a user sample in this case which is 0.05 005 okay actually we can increase the user sample to 0.05 here but that's okay okay so uh the idea is to get some test users which are also there in my training set right so so why do I why do I want this because uh my training my recommended systems model was built on training set and in order to evaluate it I need some users which are also there in the training set as well as in the test set so these users will be so we'll be able to compare that uh this user listened to some songs in the training set so those will be so the recommendations made if we keep in mind our image that from the slide where I had some Will Smith movies and the Johnny Depp movies so let's say all the relevant songs for the users are the songs which are there in the test set the known songs which I know that the user likes and all the recommendations the which which I had in that uh slide on the um horizontal axis let's those are the recommendations from my training set right is this clear this mapping that what is the basis of calculating precision and recall here so I'll repeat that all my relevant items are the ones from the test set I'm assuming that and all the recommendations are the ones which are in the training set which are generated by the model which were which was built on my training set okay so I will change this value to uh 0.05 just to have a bigger sample and run this again because uh 0.05 sample was just having one user in in it so we need some a little more uh bigger user sample and uh so what it is doing is it is it is calculating uh recommendations both using popularity model as well as the item similarity model and for um each step it is calculating a precision and recall uh values and then uh in the next step when uh the Precision and recall are all calculated for all the users in my user sample we will calculate the Precision recall curve in this step so we are just waiting for this step to finish over here are we going doing good so far good great okay so it's done now all right so this code now next cell we will run which simply plots the Precision recall curve and then this will blot okay so this this is what we got what we get here so uh just so this is our this is a very small subset of the data that's why the Precision recall curves will not be like what what were there in the slides um but just to get an intuition uh the mo so this blue curve here is the popularity model and the green curve here is the item similarity based model and uh we can see that the Precision recall values are higher even in this subset for uh item similarity based model and uh I want to make a note that whatever curves you will be getting on this data set uh would be different because the user sample what the code is generating Is Random so the shape of the curves you will be getting here would be different but uh what I also did is that I uh I also ran the whole model at home on a larger data set in order to get Precision recall curves on a bigger data set so as to give you an idea of how it will look like on a larger subset of data so what I did is I ran the code on that subset at home and then I pickled the results so um uh meaning that I persisted the results in Python we do that using uh a library called uh pickle and uh here uh I'm simply reading those pickle results and I will simp just just shortly show you how the uh results will look like on a larger set but before that I want to point out that uh for persisting the results I saw that uh for Python 3 and Python 2.7 it's it was supporting different formats therefore I persisted the results once using Python 3 and then once using Python 2.7 so those of you who have Python 3 I would suggest you run this first cell over here and those of you who have Python 2.7 I would suggest you run the second cell so I have Python 3 on my laptop so I will uh run here so this is what I get so on a larger data set you see the the green curve shows the curve for item similarity based model and the blue curve shows U the curve for popularity based uh model so you see the area under the curve for the Precision recall curve for the item similarity model is larger and uh so we can say that uh it's this uh curve suggests that the item similarity model which was a p more personalized model is giving better recommendations than a simple uh popularity based model so were you able to run this step okay great okay so if we are clear till this step I want to move on to Matrix Factor ization based system um recommended system okay right so um for this Matrix factorization based recommended system I wanted to give a just a brief demo of it that how the Matrix factorization is working behind the scenes I don't want to use it uh on a real data set where we don't understand at all that how it is working that for that reason I simply chose a very small Matrix so that we understand the behind the scenes work of this uh method so this is the Matrix what I chose I I simply created U this um Matrix which which has users on the as my rows and items as my columns and uh these values here are my ratings so it is a simple Matrix and uh then the code that follows will be implementing uh SVD so this code I want to mention is that uh this is I this is an adaptation uh of a code from this GitHub source so what we will just uh run it now so I want to point out that you can see here that in Python there is a support for Matrix factorization using this Library sparse SD which we are importing and then uh python also supports sparse matrices using this Library CSC Matrix which we will be using and then there is this scipi um Library which not only supports sparse matrices but it also supports linear algebra based calculations in this Library L alge and we will be using this for Matrix factorization so we'll just run it okay and then we run this step as well which is which has the methods to compute SVD as well as the final recommendations okay so uh now we will what we will do is we will calculate our recommendations we will use the methods which have been just built in order to to make recommendations for a test user so this is my Matrix remember the Matrix uh which I had just talked about over here this Matrix how will we represent this Matrix uh in Python anybody knows using a numpy array right the python has a support for uh matrices so we'll create an umpai array uh for this Matrix and uh the last we can see that uh I I didn't I created this user ratings Matrix as that uh the first 0 1 2 3 users they have all all the ratings with them but the last user which is user number four has rated only one movie and our task is to generate recommendations for this last user right so what I did was I created a test set in which I simply put just user number four and then I call this um the methods which we wrote above to calculate compute estimated ratings for this user let's see what we get so we get here um for user 4 two these are my predicted uh predicted uh movies actually so first U Movie number two is being predicted and actually uh one thing I want to mention here that is that I purposefully left the movies here in the recommendations which the user has already seen so you see the test user over here has already seen user number two uh has only seen uh movie number two but uh so it is being over here this is the top most movie being recommended and I purposefully left it to get an intuition so uh so my top recommendation here is two movie two then movie one so we can uh go back in this Matrix and check out this is my user 4 so uh this is this item he has already uh washed and then item number one is the second most uh second item that is being recommended here and and I wanted to just show you a demo that uh if we change this um the movies which are the which the last user has already watched then the recommendations will change so previously our recommendations were Movie 2 1 0 and 3 and now they have changed now right and so based upon what uh so the weighted sum that is being calculated will be different here so the recommendations change so what uh I wanted to show here next is that um yeah so this quiz we already did just now that we changed okay let's change it to this value so so the uh now we'll simply change it to some other value and see how the recommendations change okay yeah so the in for this quiz we see that uh when we when our test user has watched has rated only movie Zero then we get these recommendations over here okay so now now the most important part now comes to understand the intuition behind SVD here that uh so given our uh user items Matrix how is the SVD acting on on that Matrix so remember we talked about that SVD result gives three matrices as outputs u s and V transpose um where T here means transpose so uh we talked about how Matrix uh SVD will convert will transform that initial given user ratings Matrix to vectors in the latent dimensional space so I created a sample code so as to of that how those vectors will look like in a two-dimensional space so in SVD youu s so the for the s represents how many latent Dimensions we will have here so in this case in the code here I kept the value of K here as 2 which means the number of latent factors so as to give a demo of user and item vectors in a two dimensional space so this K means two a number of two latent factors so remember in the figure that I had in slides the we had two two dimensional figure and the uh they represented how much of a serious movie the user likes and how much of the um we were quantifying that how much the user the movie is uh serious versus non-serious and how much the movie is geared towards males versus females and similar for users we were quantifying a value to each user that how much this user likes serious movies versus Escapist movies so uh just now we map the same slide over to this scenario wherein we have the small Matrix and we are representing users in my and items both in my latent dimensional space wherein it is a two-dimensional latent space so this code here will do that in the end I'm using p laab in mat lab to um make this visualization so yeah so this is what we get so um this is my latent dimensional space for this problem which is a two- dimensional space and these stars represent all my users and U all these items are represented by these Styles so you see uh that in that slide I had these nice captions that my lat dimensional space is represented by these yr serious Escapist movie gear towards males or females but when we are actually doing this Matrix calculation SVD in the code it doesn't tell me automatically that what are my uh Dimensions called in reality when we do the Matrix factorization the dimensions would be very abstract and this is just two Dimension and uh when we taken more yr let's say more Dimensions into account like for example in case of movie yr serious horror comedy and so on we will have a multi-dimensional space in which users and items will be representing represented and uh we have to um uh understand these U dimensional this dimensional space so as to come up with to so as to make a mapping between these abstract Dimensions uh into some specific yra so this is this is done this this mapping has to be done by understanding this data somehow and uh so the point I want you to make clear is that these uh latent spaces are very abstract when you do the actual Matrix factorization in the real world but uh just to give an idea that how these users and items will look like in the latent dimensional space so when a new user will come we will again given its ratings we will plot it out in the latent dimensional space that where this user lies suppose here and then we will find out what are the users and items in the vicinity of that um user in that multi-dimensional latent space and we will use them to make recommendations right this makes sense great okay so this finishes the notebook and I will go back to my slides here okay so uh these are uh some of the resources uh I would like to recommend you if you if you want to study further about recommendation systems yeah so I'm also making a recommendation to you today that uh I followed this book um recommendation systems and introduction by derer janak and U um then this book massive mining data sets by your leco it it builds background first about Matrix factorization methods and other um mathematics based Concepts and then it goes to it has a chapter on recommended systems so if you want to read all the background also then this second book is quite useful and then uh there is a corsera course on recommended systems by University of Washington in fact I I uh followed this course quite a lot uh I had um there's a follow previous course which Builds on onto this recommended systems and I just finished that course and uh I the material um gained some inspiration from that course as well what I just used and uh then there is another course on corera by University of Minnesota on recommended systems and if you want to go further in into details and uh lastly I thank you all for uh joining me here today for this tutorial I uh hope that you enjoyed this tutorial and if you have any questions uh please ask me now thank you