Text Preprocessing for NLP: Tokenization, Stemming & Lemmatization

[00:00:00]hey Everyone Active Learning here welcome back to another video and today's video we're going to talk about text pre-processing for natural language processing this will be part of my introduction to NLP Series so if you want to learn more about nlps I will be releasing more videos on it in my future videos so let's jump right into what we're going to talk about so text pre-processing is basically the first step you'll perform for any NLP project and in this video we're going to discuss the techniques and steps you will do for tax-free processing so here are some of the terms that we'll be discussing today and why they matter and after this video you will understand every single one of these terms in depth right so let's Jump Right In the first term is called tokenization so tokenization um is simply the process of breaking a text into different words so what did I mean by that well let's start with the example say I have this sentence how are you doing well if I apply tokenization it would simply transforms it into different words right I have the word how I have the word are you know I have decembo question mark right these are also broken up right so these are individually called tokens but um we don't have to get into the nitty-gritty details but yeah tokenization very simple we're simply breaking a piece of text into words okay um in our example here without tokenization the sentence will be treated as a single string of characters for our NLP model however with tokenization this sentence will be broken into individual Words which have different meanings right how you know are you right you you know each of these have individual meanings right and then these words can then be processed and analyzed by other NLP algorithms to determine the part of speech the meaning and other linguistic properties of each word right another example another thing is that tokenization also helps with standardization so what do I mean by that well okay yeah this was meant for the buff all right so say with these two colors um with these two words sorry uh they both sound like color right well they also mean the same thing depends on which country you're from let's say I have color and color uh and these two words will be standardized to a single one through the process of tokenization and so you might be wondering why does this matter while standardization can improve the accuracy of our NLP algorithms and make the results more consistent right we'll be treating these two as the same word right it doesn't matter if this word is color or color uh they're they're the same right if we standardize it through tokenization okay now our next step is called stop words okay so in order to understand stop words let's first look at an example right so let's say I have this sentence hey man I can't come to your house tonight in this sentence there are only a few words that we actually really care about in order to understand what the sentence is trying to convey right in other words this sentence will still capture the main idea if we remove some of the words all right so for example we can remove hay and we can remove my end because they don't really convey anything that changes how we interpret this sentence and you know we don't need the word to come to your house we don't need the two there and the sentence remaining is I can't come your house tonight and that basically allows us to interpret what the sentence is trying to convey without um some of the words right now you might be wondering why does this matter well it makes our algorithm faster since we don't have to process some of these nitty-gritty details right and now the term stop words should be very intuitive because we're simply stopping certain words from entering to our pre-processed data right okay now we got that over with so you might be wondering how do I actually stop these words from answering well most libraries can actually do most of the job for you but if you want to stop certain words on your own for example you have certain words you don't want to include then you can actually just make a list and then look for them and then just remove them right it's simple as that uh okay but okay so back to our sentence um the word can't in this example is actually very important because it indicates that the opposite action will occur right without this can it will be I come your house tonight right but with this word can't it means that I can't come to your house tonight so it indicates a completely different meaning that's why this word is so essential in this sentence right so there are certain words that we will value over others in this case huh right can't it's very important word because in the case that the opposite action will happen okay so our next action or step is called stemming um actually stemi and lematization are both steps you can take but let's first talk about stemming right so in order to understand stemming let's first start off with an example right uh let's say we have these two words history and historical and in our case let's just say they mean the same thing even though in certain contexts they might mean different things and here we can apply stemming and we get the word history right since history is shared by both of these words right the we can consider this the stem of these two words and you might be wondering well what does this word history mean well it does actually mean anything and that's actually one of the disadvantages of stemming because we might get words that mean completely nothing right okay so then you might be wondering what is the point of using stemi IF it is flawed well the answer is that stemming is blazing fast right it will help us with text pre-processing for huge data sets right so we don't have to worry about these different words we can assume that they mean approximately the same thing right see the advantages is that it's really fast which helps with pre-processing huge data sets and you also you often have to work with huge data sets when you're working with NLP so um stemi is very much used and some of the disadvantages is that certain words can be meaningless right for example history is meaningless in this case but yeah it completely depends on um where you're applying an LP to we will discuss um this later all right now our next step or term is called lematization and what it basically does is that it overcomes the disadvantages of stemming okay so limitization basically does the same thing as stemming right except now we are converting the similar words into meaningful words right in our case it will be converted to history which is a meaningful word right you know originally from stem and we had the word history but now we get the word history which actually means something right well you might be wondering how is our model able to do that well it basically has access to the entire dictionary that's why it Compares a lot of different terms or definitions to get the one that's the most similar with our case here and that's how it's able to do that right then you might be wondering well why do we use why don't we use this instead of stemming well there's a trade-off right so the advantages is that it's accurate and provides meaningful words while the disadvantages is that it's really slow because it has to compare a lot of similar words right that can be very slow for a huge data set so you might be realizing that oh wow so stemi and lematization have different advantages and disadvantages that each of them sort of overcome for the other so then you might be wondering well which one should I choose right well that actually really depends on the use case right so let's look at the stem inverse lematization use cases okay so for stemming we usually use it for spam classification right because um the accuracy of each word don't matter as much in this case and for common toxicity sorry common toxicity classification um I guess the meaning for each word might not matter as much right but and for limitization we might use it to summarize text right because we care a lot about the accuracy and we also use it for chatbot because we want our chat bot to actually give meaningful responses right we don't want it to misinterpret something and we also use it for translator which you know values values accuracy a lot right so these are the certain use cases and you kind of have to determine which one you value more right do value speed or do value accuracy because if you have a huge data set you might want to Value speed over accuracy but if you have a small data set then you might want to Value lemon value accuracy over speed right so you know throughout machine learning and deep learning there are tons of trade-off and this is one of them right you kind of have to consider which one is more important to you and yeah that's it for this video we talked about uh tokenization we talked about stop words we talked about stemming so now let's quickly just recap right so tokenization is basically breaking each sentenced into individual words and symbols in order to make them mean different things in different contexts right and stuff this sentence being analyzed together will analyze any each individual word right and each individual wouldn't have different meanings right and stop word is that we're basically canceling the words that we don't really need to interpret our sentence and we're keeping the essential ones right that allow us to actually interpret what the sentence is trying to convey and then we talked about stemming which is basically trying to find the similar stem for different words right and then we're converting those words to the stem right the advantages is that it's super fast and it helps with pre-process it helps with processing huge data sets the disadvantages is that it can be inaccurate right providing meaningless terms and for limitization we talked about this it provides accurate and meaningful words but it can be really slow because it has to compare a lot of similar words right so yeah that's it for this video I'll be doing more NLP videos in the future um NLP will take a long time to actually discuss because it's a pretty big field right so yeah see that's it for this video I hope you guys enjoyed this video and how and found it helpful and yeah if you want to support me you can like And subscribe to this video and also you can check out my other videos where I talked about deep learning and sort of computer vision transfer learning that sort of stuff so yeah that's it for this video see you guys next time bye