TF-IDF Explained: R Text Analytics for Document Weighting

[00:00:07]hi welcome back so in the last video we Illustrated how we could take our document term vector or document term matrices representation of our text data and use it to build a powerful machine learning model specifically we illustrated in the last video that you could take our document term frequency matrix and train a single decision tree using 10-fold cross-validation repeated three times and produce an estimate of accuracy of more than 94% so generally speaking what that means is that if we built a model a single decision tree model and put it in production we would expect it to be able to predict correctly legitimate text messages that is ham from illegitimate text messages that is spam more than 94% of the time it should do that correctly more than 94% of the time and that that's pretty good that shows us the document term frequency matrices work and more specifically it tells us that this bag of words model that is taking our freeform textual data and running it through a data pre-processing pipeline consisting of things like stop or removal tokenization stemming so on and so forth actually works right it's powerful it works it allows us to ignore things like context for example do we mean running in terms of a human being moving as fast as I can under their own power versus the IT context of a computer program we can actually say look these things don't matter we can we can rely on the bag of words model to to differentiate between those two contexts we also have shown now that we're ordering you know we can get we can get a lot of power out of the this representation without worrying about word ordering okay cool but here's the thing as powerful as this representation is there's still some room for improvement we have some issues that we can address so first and foremost there's this idea that longer documents will tend to have higher term counts and this is actually relatively intuitive idea let's take the example of two documents and they're in this sports subject area more specifically they're about baseball and let's say one is a baseball article from Sports Illustrated magazine and then say it's around a thousand terms you know it's a thousand words let's say total and then we'll have the book Moneyball which is also about baseball and let's say and I don't know for sure but I'm making this up so let's say the book Moneyball is 20,000 words now they're both about the same subject area baseball so we would expect very specific terms related to baseball to occur in both documents Tom's in terms like ballpark in umpire and strike in bunt and you know RBI you name it right there's a bunch of terms that are specific to baseball that we would expect to co-occur in both the Sports Illustrated baseball article as well as the book Moneyball now intuitively we will also realize that where those terms occur in both documents it's far more likely that Moneyball will have higher counts of those terms just because Moneyball is just a longer document the word umpire is just more likely to show up more frequently in that document the Moneyball book then is in the in the Sports Illustrated article that's just intuitive so that's one thing that we need to address that would improve things the other thing that we need to address is this idea that terms that appear frequently across the corpus aren't as important so for example let's say that I have a collection of documents and once again we'll just use the example of American baseball and let's say I have every article about baseball that ever appeared in Sports Illustrated magazine and that is my corpus that is the collection of my documents if my subject area is limited only to baseball and more specifically articles from Sports Illustrated magazine about baseball it's highly likely that certain terms will show up in many of the documents for example the word umpire it may be entirely possible that the word umpire may show up in 70% of documents 80% of documents or here's one here's here's an even better example it's extremely probable it's extremely likely that the word baseball itself will show up in 100% of those documents it's not unreasonable to assume that Sports Illustrated articles about baseball every single one of them will contain the word baseball the term baseball as such you could make an argument that in that particular case the term baseball isn't that important if it shows up in every single document how important can it be it's not providing you a lot of information it's not providing a lot of information so that's another thing that we can address so here's the thing we can improve upon this bag of words representation we can improve upon our document term frequency matrices if we can achieve the following one if we can normalize documents based on their length whether I'm talking about a thousand word Sports Illustrated article about baseball where I'm talking about a twenty thousand word book like Moneyball if I can normalize their term counts across both of those documents irrespective of length it puts those two documents on equal footing and that's a good thing that's a good thing because usually we're not worried about so much about document lengths we're talking--we're tend to worry more about the words that show up and how frequently and how important they are within the document themselves so normalizing the documents based on their length will help us with that second if we can penalize terms that occur frequently across the corpus that will also help us in our representation that will make it more powerful again for example if my corpus is every baseball article that ever showed up in Sports Illustrated magazine and the word baseball shows up in 100 percent of those documents if I can penalize that term so that all of a sudden it looks less important that would also help out because the term baseball is not likely to be predictive if it's in every document how can it be predictive by definition it's not gonna be very predictive so if we can adjust our matrices to accommodate these two things it'll increase its power okay enter in term frequency inverse document frequency this is exactly what tf-idf addresses so first up let's take a look at term frequency and what does it mean so we're going to get a little bit mathematical in these slides nothing too serious so hang with me on this so if we let a function that we'll call SR EQ or frequency the function frequency as defined by T and D and we let that to be that function be the count of the instances of term T in document D so if we just define a function that says look for any term in the document just count up how many times it shows up that's basically all this means okay so for example if it walks like a duck and it quacks like a duck it must be a duck the the frequency of the term duck in that document is three the word duck shows up three times so further let's define another function called TF for term frequency and the TF for T and D would be essentially the proportion of the count of term D of term T and documenting so for example if I had a particular term that I showed up in a document four times and the document was ten words long or had you know had ten terms in it after my processing pipeline after stemming and stop where the move let's say then that particular term represents 40 percent of the document zero point four that would be our expected value for that particular term for the term frequency in that document pretty simple just what's the proportion of the term of the document okay so we can represent that mathematically using this calculation so the term frequency of T given D is essentially the frequency of T in D divided by the sum of all of the frequencies of all the T's in the document this essentially just gives us the percentage the proportion so that's pretty simple right and notice how to suggest this achieves that first goal this normalizes across documents it essentially says look you know what I don't care how long your document is just tell me what the relative proportion of this particular term is feasibly the entire document so again if you take a 1,000 word document that is a B article about baseball from Sports Illustrated and you compare to the book Moneyball you can now say the term umpire is on more even footing because irrespective of which document is longer you're going to talk about how how prevalent is umpire in terms of the proportion of the document in total so it normalizes so this is a way to normalize the term frequency across documents at different lengths sweet so the math works out it helps out our goal okay next up we have inverse document frequency or IDF so if we let capital n be the number of documents in our corpus that is the number of documents in our collection so for example if I had a thousand articles about baseball from Sports Illustrated in my corpus that was my collection of documents I was working with capital n would be 1,000 if I define a function called count of T where count of T just represents the number of documents in which term T appears this is also relatively intuitive let's say for example once again I have a thousand baseball articles from Sports Illustrated magazine and let's say that the term I'm interested in is the word baseball and as we surmised earlier let's say the term baseball shows up in every single one of those documents counted T would be one thousand baseball shows up in every single document in the corpus there are 1000 documents in the corpus ergo count T is 1,000 pretty simple so given that the IDF is essentially the log of the number of documents in my corpus divided by the specific count of a term how and how many of the documents across the corpus does the term actually show up in that's all it is so once again notice that if I had a capital n be 1,000 right because I had 1,000 sports illustrated baseball articles and then I said look I'm looking for the count of the term baseball in my corpus and assuming that baseball shows up in every single article I would have County is also 1,000 so n would be a thousand County would be one thousand one thousand over 1000 is one and the log of one is zero notice how this math also works out term frequency normalizes that was our first goal this one discounts pan Eliza's terms that show up frequently across the corpus and in particular if a term shows up in every single document in the corpus this mathematical equation essentially gives it a weight of zero you can't get much more penalizing the mat essentially it says look baseball is useless if baseball shows that in every single document in your corpus there's no information in there that you need to worry about so cool we now have a mathematical representation of how we can address some issues in our document term frequency matrix matrix representation okay so this leads us to the mighty tf-idf so if we combine TF and IDF we can enhance our document term frequency matrices to make them more powerful and tf-idf is very simple it's simply the TEF of a term for document times the IDF of that term and we'll see what that means here in a second in actual our code okay so here we are in our our environment and I've ran the code through essentially the end of video number four to set up the environment so the first thing that we need to do is if we're going to use the mighty tf-idf in our solution and I would have I would have I would suggest that we should we should try it out because most of the time almost all the time tf-idf actually helps you out so it's actually generally speaking into facto part of most text analytics data pre-processing pipelines you lower case you remove stop words you remove numbers and symbols and hyphens you stem and then you run tf-idf that's got the prototypical data pre-processing pipeline and text analytics so we're going to go ahead and hand roll our own tf-idf functions and we're doing this for two reasons one it's educational illustrate of what's going on behind the scenes and two it allows us to calculate our IDF values for all of the terms in our training data so that we can keep we can cache that we can keep that vector of IDF values and use it later to transform our test data into comparable representations and that's super important and we'll talk more about that later but just to just to give you a heads up okay so first up we needed to find a function in R for calculating our term frequency so here's the code that we can use to do that and notice that it's pretty simple right if a row in my document term frequency matrix it represents a document which it does it's pretty easy take every member of the row and just divide it by the total of the row right R does vectorize processing so this will essentially say go through the first column divide whatever is in that first that first column isn't in that first cell first vector see me the first value in the row and divide it by the total of the row and this just gives me the percentage across all the individual terms in the document nothing too difficult exclaims run a code and create our function sweet now ID F inverse document frequency is a little more complicated but not cut astronomically so first thing as we do is we say okay look you know we done we do we do TF on rows right we normalize across each individual document which is our rows in IDF we calculate by column right because we want to actually look at the terms now term frequency is document-centric IDF or inverse document frequency is corpus centric or column centric notes and we're working on columns versus rows so the first thing we need to do is we need to calculate for each one of the columns how many documents are there let's link the column now generally speaking this is going to be the same for every column obviously because we have a matrix and then our document count is exactly what you would expect which is okay just give me the number of rows where the column is not zero right the count of this particular term for a particular document is more than zero right that's what we care about so we care about for this particular term and how many documents does it appear that's what this this code does cool and then lastly we take the log now notice in the slides I used log generically and I didn't specify a base technically it doesn't really matter we'll use base 10 here because that's actually commonly used in implementations for example if you want to compare this code that we're going to be using to the quantity' tf-idf function this code here will generate essentially the same output as quantity does if you make sure that quantity also normalizes which it doesn't do by default so if you want to double check this with a quantity F ID F function you can go and do so set the normalization to true on that and you will get exactly the same numbers as we'll see this code generates okay so now we've got TF an IDF so if I run this function this will generate my I D F function and then lastly we combine the two into our own custom tf-idf and as we saw from the slide decks that's pretty easy I just multiply X by IDF and good to go actually not let's change this let's make this even more specific let's make this TF instead of x times IDF that's better yet cool look at that nice okay so now we have our tf-idf functions so we have all of our functions that we need to make this work okay so first up first up we have to do some matrix transformations we have this matrix this train tokens matrix right from last time so if I do a quick view on that and then just highlight this real quick pop that bad boy in here we'll take a second to pop up in the spreadsheet view but you can see here right this is just all of our terms across the top and all of our documents down down the side here in just the individual counts right 1 1 1 2 1 2 so on and so forth great we need to transform that first step obviously is that we have to transform all the documents using the TF function we need to normalize all of our documents we want to get rid of that we want to address that first issue which is we want all of our documents irrespective of lengths to be on equal footing so we need to normalize so how we do that is we use the apply function if you're not familiar you can always put this up pulls up in the help file and apply functions over array margins right this essentially says look you can you can have our automatically apply a function over and over and over again either against the rows or against the column of the matrix this is a way that you can loop an R if you're familiar with programming this is essentially an R style for loop so here's what we can do we can apply our term frequency function against this matrix and the one do it against the brows so if I run these two lines of code here what's gonna happen is is that it's going to apply the function against the rows now you'll notice something really super interesting here I ran the dim here so that you're aware of this now originally if I do them on this matrix notice that I had three thousand nine hundred and one rose in 5743 columns notice that this is the output of this function call has transposed our matrix it's essentially swapped the columns for rows and in linear algebra that's known as transposition I've transposed matrix notice I haven't lost any data it's just basically swap the rows for the columns just be aware of that so we can quickly take a quick view with this and just see how things have been swapped from last time now the the bit of the call-out of the rows are now columns and what we used to be columns are now the rows but notice that the numbers are now different notice that these used to be ones right this is this is our first document here excuse me this would be our first document right here and notice that these used to be ones but now they're 0.0625 instead these have been normalized they've been normalized which is cool okay so next up we need to calculate our IDF vector when you calculate our IDF vector and remember I said earlier in the PowerPoint that we're going to want to cache that we're going to want to store it because we want to transpose transpose we want to transform we want to be able to map any brand-new data once we go into production and new documents come in new tweet consuming new text messages come in we need to be able to transform those new text messages into the same space that our machine learning model was built so to do that we need to make sure that we store we calculate the IDF's and we store them because that is the the fixed set of IDF's that we use to train Machinery models therefore in our test data any brand-new production data we need to use those same IDF values to transform new data that the models never seen before into a space that it understands okay so what we're going to do here is we're going to go ahead and use the apply function again against the matrix but notice this time we're going to apply the IDF function the inverse document frequency function against the the data set and this time instead of the rows we're going to go against the columns right because that makes sense because IDF is a is not a row based it's not a row based calculation it's two column based calculation so if I run these two lines of code real quick it'll take a second and you'll notice that I get back a single vector a single numeric vector that is 5743 in length and you'll notice here here are the individual IDF calculations that correspond to each one of our terms Sweden so now I've got the TF I've got my documents normalized and I also have the IDF values that I use okay so not surprisingly here we go so we can apply the function again now this is going to get a little more interesting we're going to apply on our data frame right another place you got our data frame Arnaud transformed our normalized matrix here right this is our TF normalized matrix and notice here because because the the matrix was transposed we're going to run against the columns and we're going to apply the tf-idf function to it and we need to pass in our IDF vector values so now if I run this take a second I notice that I've still maintained my transposed matrix where we originally had 3900 and one call a3 not 3900 and one rose in 5743 columns we still maintain this transposition which is why we oriented to the columns but if we take a look at the data now we should see that it's been transformed notice that the values now for text one they used to all be the same because they were normalized to that individual document vector now they're rationalized they have been combined with the TF in the IDF values to achieve our two goals the goal of one normalizing documents based on their length so that we can compare documents on equal footing and then to rationalize individual terms individual terms in our corpus to say look those terms that appear more frequently are going to be less useful than those terms that appear rarely so for example by definition higher numbers here will represent things that appear less often so for example the word go the term go it's intuitive to us that it appears quite a bit because you'll see the lower score here whereas Jurong has quite a bit higher score and intuitively would say yeah that makes sense I would expect that in English SMS text message is the word go the term go the token go should happen to occur a lot more often generally speaking than the term wrong okay cool so now we've got a mighty tf-idf matrix excellent so here's the thing we need to transpose it back right we need to get it back to be a document frequency matrix right now it's actually in a term frequency document matrix format where the term frequencies are the rows and the documents are the columns that's okay you'll see most of the literature or a lot of the literature in text analytics actually use that representation however when you get ready to train a machine learning model you'll need to flip it back so that's what we'll do here we'll use the T function and if you run the help that the T function is a matrix transpose so that's exactly what we'll do here we'll go ahead and transpose our matrix and notice we get back to our original representation 3900 one documents with each one having 5743 columns if we just view that real quick in the spreadsheet view all is right with the world we've now could we've now invoked the mighty tf-idf on our document term frequency matrix and we've got all kinds of predictive goodness in it now okay but let's test it out let's test it out let's make sure that in fact I get the goodness that I expect but we've got a little bit of a problem that we need to deal with here got a little bit of a problem to deal with here when we invoke tf-idf on something we need to actually check for a particular degenerative case and that is the case of after we've done all of our pre-processing we remove it all the stop orders removed all the symbols and numbers and etc and we've stemmed it's entirely possible that there's nothing left there's nothing left we basically remove everything from the string so imagine if I just put a bunch of emoticons in a particular text right bunch of smiley faces made of colons and dashes and parentheses and things like that those are all symbols so our pre-processing pipeline would strip those all out so we can actually have empty strings now when we do a tf-idf calculation on an empty string will actually get essentially errors from our so to check that we can actually say look you know what go through my new mighty tf-idf matrix and check out to see if there are any cases that are not complete so for example anytime there's an error in the calculations of tf-idf I'll get not a number as a result and that will trip this complete cases function and this is a way for me to test to see look did I in fact have any text messages where essentially I stripped everything out and there's nothing left so I have essentially an empty string so we'll go ahead and do that and you'll see yep in fact I do find kind cases that are not complete in fact incomplete cases and if I go look at the original text message some of those here's what I get what are you doing how are you well we know these are punctuation so we'll strip those out and we also know that every one of these is a stop word so therefore this particular text gets essentially it turned into an empty string from our from our previous Immel a this is just numbers so our pre-processing pipeline will basically strip that all out making an empty string that's just a bunch of symbols so our pre-processing pipeline will strip those all out and make it nothing and so on and so forth right same thing here same thing here so on and so forth okay so here's what we do we can fix those incomplete cases up because if we try to train a machine learning model with incomplete cases most of the machine learning models that you use an R will throw an error so we need to fix these up and that's pretty easy to do all we do is essentially say look for any row or any row in my tf-idf matrix that is incomplete just replace it with all zeros does replace with all zeros it should be fine we don't want to get rid of those rows because it's entirely possible that text messages that are essentially get turned in nothing by our pre-processing pipeline may actually be predominantly ham right legitimate these are probably legitimate text messages these are probably legitimate text messages very likely so we don't want to lose those records so we just replace them with all zeros and if there is any signal in the data what will happen is that our and our machine learning model will pick up on that and actually show us what's going on okay so next up now that we've identified that we have some incomplete cases that we need to address let's go ahead and fix them so simply what we'll do is we'll just say look any place in my IDF that I've got missing data I'm just going to go ahead and replace that with all zeros all the reasons that we just described so if I run on these cookies lines of code you can see here I've still maintained my dimensions everything's great I've got 3901 rose and 57:43 columns and notice now that I have no incomplete cases taking those those instances of those documents those text messages where a pre-processing pipeline has essentially stripped all of the data out and made them empty strings have now replaced those with 0 so again if my machine learning model can do it it will learn some signal okay last step what we'll do is we'll clean up our data frame names just like we did before and get ourselves a nice clean data frame now this time instead of it just being the raw term frequency counts for each of the documents it'll be the mighty tf-idf adjusted scores for each one of those terms in each one of the documents okay so now we're all set up and we're good to go since we're close on time now I'm going to go ahead and stop here and what I'll ask you to do is please please please if you have any questions or comments or concerns please use the comments section of the video on the data science dojo youtube channel we monitor the channel frequently and we try to answer any and all questions and comments and concerns promptly next up if you like what your what we're doing on the youtube channel please subscribe to the data science dojo youtube channel we will be producing new content weekly so if you subscribe you'll keep abreast of all the latest and greatest things that we're doing also more generally if you like what we're doing a data science dojo hook up with us on social media we're on twitter facebook linkedin youtube you name it hook up with us and you can get a nice steady stream of data science goodness sent your way and I hope to see you lastly in one of our upcoming data science boot camps so until next time this is Dave ray and I'm wishing you very happy data sleuthing you