LSTM Networks Explained: Intuitive Deep Learning Guide

[00:00:01]Welcome to another episode of our deep learning web series, comprehensive and intuitive introduction to deep learning.

[00:00:08]In today's episode, I'll be talking about a form of recurrent neural network called LSTM, which stands for long short-term memory. In the prior episodes, we have seen the limitations of a feed forward network. For example, in the prior episode, I told you that in a feed forward network, the network really does not understand the concept of a sequence and it essentially takes each training sample and it does not consider any ordering or any relationship between the parts of that particular training sample. And I showed you in the prior episode that using a recurrent neural network, you could solve that because a recurrent neural network understands that a sentence actually is a sequence of words and it needs to understand and recognize the context that is flowing across those words to determine the meaning of that sentence. The same thing for with time series data or with audio. So there is certain type of data where essentially looking at the input as a sequence matters and understanding the context flow between the individual parts of that sequence is very important and we studied this in the prior episode season 2 episode 3. Now once a recurrent neural network is trained it can distinguish between these two sentences where all the words in both the sentences are the same just the ordering of one of the word changes from being the second word to the last word. For example, she alone finished the task is very different than she finished the task alone. And a rec and and an RNN recognizes this difference because it basically takes the sentence, breaks it up into individual words or tokens. And then it it essentially starts to study the context flow across these words to determine that this sentence could mean one thing and this sentence could mean another. So we studied this in episode 3 of season 2.

[00:02:00]So one of the limitations of an RNN is that it can handle short sequences essentially sentences that have let's say less than 15 words. Once the sentence starts to get longer or the time series window starts to get longer the accuracy of an RNN starts to deteriorate. Now in the real world sequences are definitely much longer than 15 words. Let's take an example.

[00:02:26]Let's say that I'm trying to build a neural network that can understand the sentiment of a user review. And the product I'm reviewing here is a coffee maker. Now, the review left by the reviewer is something like this. This coffee maker dripped water all over my counter and its drip tray warped after only a week of use. The filter basket rusted and the fragments of metal ended up in my coffee. It also stopped midway through the brewing cycle twice in the first month. However, the brewed coffee tastes surprisingly rich and the stainless steel finish looks sleek. Now, this review is certainly more than 15 words and I think it is about 60 words or something like that. Now, there is a pattern here. If you look at this pattern here, what we see is that the negative sentiment that is being conveyed in this review is towards the beginning of the review or beginning of the paragraph. But towards the end of the paragraph, the sentiment starts to turn positive. But the net net sentiment of this entire review is that the user really did not like this product because it malfunctioned and it had issues. Even though the taste of the final coffee that came out was good, the predominant sentiment here is that it is negative. Correct? And you we have to build a neural network that can actually understand the context that's not just towards the end of the sentence, but also at the beginning and in the middle parts of the sentence. Okay. Now, why is it that recurrent neural networks actually struggle when the sequence lengths are longer? Let's understand that first. So, let's go back to building a neural network that can understand the sentiment conveyed by a review. So, let's say the review is something like this. I returned the coffee maker. It malfunctioned often though it though it made good coffee and lost sleep. And our task is to build a neural network that can takes the take this text and classify it into one of these five possible sentiments. The user keeps the product, the user keeps the product and buys more, keeps the product and writes a review or returns a product or cancels before the product is shipped and so on. Okay, just purely based on the review left, we would like to, you know, classify the reviews into one of these five categories. Now, one might question and say, hey, why do we really need to build a neural network for this?

[00:04:55]Can't we just use keyword matching techniques to determine one of these things? It clearly says here that the user returned the the word returned appears in the sentence. So, most likely the classification here is number four, returns a product. actually keyword based matching really doesn't work because if let's say the review was I was worried because the seller shipped a return copy maker but it looks and works just like new. If you were to use a keyword based matcher you would essentially look at the word returned and say oh you know maybe the user actually returned this product. So just based on keywords you cannot determine the sentiment and you cannot classify.

[00:05:36]So we need a neural network that can actually understand what is being conveyed in the sentence to make the right classification. Okay. Now with this example, let's try to understand why is it that a classical RNN struggles when the context length is much longer.

[00:05:51]Okay. So how is this actually processed?

[00:05:55]How is this sentence processed by a recurrent neural network? We studied this in the prior episode season 2 episode 3. But just a quick recap here.

[00:06:03]So you take this sentence and you're training it. So you essentially need to know the ground truth value. So the ground truth value here is okay this sentence you see something like this. It means that the user returned the product. Okay, you tokenize a sentence and then you initialize the recurrent neural networks uh context state and then for each token like I return the coffee maker and so on you get the embedding. You multiply the embedding with a weight matrix add to it the bias matrix and then you take the prior state which is initialized to zero multiply it with the context weight matrix then you use tanage and then you essentially calculate the next state that you're going to pass to the next token. Again I'll not go into all the details here because I've shown this using different examples in the prior episode. If you don't know what this is I would highly recommend that you go back and look at the prior episode.

[00:06:58]Okay. Anyway, so I think the key takeaway here is the way an RNN processes this is that it tokenizes a a sentence into individual sequence of tokens and then it processes those individual tokens one after another while passing the context from one token to the other. And the way it does that is something like this. It starts with you have an an RNN. You pass the first token which is I and the initialized you know context that you're passing and then the context is updated and then you pass the next token which is returned then the next token which is the and so on and essentially in a loop it'll process all these tokens and then it makes a prediction. Okay. And you do the whole thing about you know you compare the prediction with the ground truth value and if it is not correct then you back propagate the error you adjust the weights and so on. Now let's try to understand why the length of a sequence actually affects how well the RNN can train. So let's go back to our example here. There are 16 tokens and the prediction is made after all the 16 tokens are processed. said essentially the prediction is made after the 16th token. However, we know just looking at this example that this prediction is heavily influenced by something that happens much much before which is essentially at let's say token number two or time step number two.

[00:08:27]Okay. Now what happens if this length essentially the the select set of tokens that influence the final outcomes appear much before you know the end of the sequence which is where the prediction happens.

[00:08:42]What happens then? So let's try to see the math behind that. So let's focus on just this part here. The context that is flowing through the recurrent neural network across all the tokens in the sequence goes like this. The current context let's say H2 is a function of tanage of the previous context H1 times the weight matrix plus the additional you know the input value and the input weight matrix and so on. So this is essentially the formula. So you can see that the current value is dependent on the previous value this way and it goes through a tanh function. Now in order to make this prediction correct what you have to do is the only two levers we have is to adjust these weights wh which is essentially the context weight matrix and the wx weight. Correct? These are the only two levers we have.

[00:09:42]How does back propagation work in a recurrent neural network? Because a recurrent neural network has a loop. The back propagation not just adjusts the weight once over here, but it adjusts the weight once here then it adjusts the same weights again in the previous time stamp, again in the previous time stamp, again in the previous time stamp and so on. I actually explained this in the previous episode. It's called back propagation in time. Okay. Now mathematically speaking if you look at it this way let's say the loss which is the cross entropy loss between the predicted value and the ground truth value is let's say loss now what you're saying is how much weight adjustments can I make to w and wx in this time stamp so that the loss is minimized then I go back right which is this one here then I go back and say how much weight adjustments do I again make to the same weights w and wx X in this time stamp so that the final weight is less and then you go back and then you say how much weight adjustment do I make you know so that the loss is minimized and so on. So essentially you go back like this and you multiply these loss losses like this is basically you know high school level calculus essentially differential uh equation here. Okay. Now what happens when the sequence length increases? As you can see this is a function of how many elements are there in my sequence.

[00:11:12]Correct? Here there are 16 tokens. So it's 16 times you're multiplying. Now let's assume that the gradient of U of this here is basically 0.3. And let's assume it's the same gradient across all these time stamps.

[00:11:27]Okay, just for simplicity reasons. And by the way, derivative of tanh is always going to be a value between 0 and one.

[00:11:35]You can go back and reference your know high school level calculus. You'll know that the derivative of tanh is always going to be between 0 and 1. Okay, which means that it is a fractional number.

[00:11:46]Now in the first time stamp, it is 0.3.

[00:11:49]Let us say which means that I'm going to adjust the weight by decrementing it let's say by 0.3. Okay, that's the simplest way of understanding this. Now in the second time stamp when I go back that 0.3 gets multiplied by 0.3 again assuming that each of them have the same gradient. Let's say it goes to 0.3 again. So 0.3 which is a fractional number multiplied by itself becomes a much smaller than number than 0.3. Then you go back one more step. Now that 0.3 gets multiplied by 0.3 again assuming that the same gradient even for this time step and that becomes even smaller number. So as you're going behind in your time steps and as you're going for to the beginning of your sequence the number starts to decay very very rapidly because you're multiplying small numbers. And here if you notice you know this is the place where we said you know this input influences the prediction which is at time step two. The gradient value is actually very small almost close to zero. Which means that you know even if you were to adjust the weight by decrementing it with this value the weights are not really moving much and as a result as the sequent length increases. What you will see is that the gradient vanishes and as a result you know the for the the tokens in the beginning they would essentially have lesser influence on the prediction that is coming out at the end. Now again this is also a very small sequence length.

[00:13:18]You're talking about 16. Typically in real life cases you would have sequence length of 50 or 100 tokens or words which means that when you get to that size your gradients essentially vanish and as a result the neural network is not really learning. So that is the problem of you know RNN a classical RNN for very long sequences and one of the reason why that happens is because the relationship between the current value and the prior value is multiplicative in nature. Okay, you're multiplying the previous value with the weight matrix to get the new value.

[00:13:56]Okay. And because of this multiplication and because the derivative of this when you're doing the gradient descent is going to be a fractional number always because tanh derivative is always a number between 0 and 1. You will see that as you're multiplying backwards this is going to get very very small.

[00:14:13]So how do we address this limitation of an RNN where an RNN learns very little from details that appear much earlier in a long sequence due to the vanishing gradient problem and there is no way for us to store memory of important information that we can reference later as the sequence length increases. We do that by essentially number one adding to an RNN a concept of long-term memory where important information can be stored and referenced back later and also by addressing the vanishing gradient problem by you know looking by addressing the multiplicative relationship that exists between the current context and the prior context.

[00:14:57]This is the multiplicative relationship that exists in an RNN. So by addressing these two we'll be able to solve the limitation of an RNN in processing a very long sequence. That is where a new form of an RNN an optimization on top of a classical RNN called long short-term memory also called as LSTM was introduced way back in 1997.

[00:15:20]But its popularity was really took off when Google used that in 2015 almost 18 years later in Google voice transcription and they found out that by using an LSTM they actually could improve the the accuracy by 50%.

[00:15:39]And then again in 2018, Google introduced this concept called smart compose in Gmail where as you're typing in an email, it'll essentially autocomplete not just the next word but the it can finish off the sentence. And the finishing off the sentence was done in a very contextual manner. And ever since then any email popular email service that we use whether it is Gmail or Outlook and so on essentially has this feature and we have been used to it. And all of that is thanks to LSTM. One big challenge with LSTM is that it's a very complicated architecture. If you look at its paper, the 1997 paper where the concept was introduced, it is pretty complicated.

[00:16:24]Lot of math and it is very intimidating and inaccessible.

[00:16:28]And if you look at any of the online you know articles on LSTM they they show all these pictures with a lot of boxes and a lot of you know functions like tanh and sigmoid and all that but they really never explain why any of these things is needed and as a result I mean if you ask let's say perplexity you know is it hard to understand LSTM and it actually says this LSTM networks are widely regarded as more challenging to understand than many other neural network architectures.

[00:16:58]is and it it articulates the reasons why it is it is so. So what I thought maybe the right way to understand an LSTM is not first to understand its architecture or the math behind it or even the code but actually first to get an intuitive understanding of what's really happening and for that we can actually use our human brain and how we process long sequences and once we get an intuitive understanding of that we can now then map that to actually understanding the architecture of an LSTM as well as the math behind it. So to do that I will take a very different approach than the ones used elsewhere to explain how LSTM works or how to how we can intuitively understand LSTM. I'll use Breaking Bad which in my humble opinion is the greatest TV series ever made to help us understand the core concepts behind how our brain processes long sequences and then we will map that to how an LSTM is designed and how it really works. Now, Breaking Bad is a TV web drama that ran from 2008 till 2013. It's made out of 62 episodes spread across five seasons.

[00:18:10]There are about 16 main characters and dozens of subplots. There are several other smaller characters. It's almost 50 hours of content. Now, imagine we are binge watching this TV series episode after episodes. And as we are watching across all these 50 hours of content, we start to forget some of the main characters and some of the main subplots. So by the end of the TV series, we will really not be able to make sense of the climax or why it turned out to be that way or why certain characters are behaving a certain way towards the end of the series. Correct?

[00:18:41]So how do we fix that? How does our brain fix that? The way it fixes that is first to we identify what is important information. For example, who are the main characters? What is their motivation? And that builds up over time, episode after episode, context after context and so on. The next thing we do is we continue to remember the important details like you know the subplots among various characters. Why is it that Jesse Pinkman and Walter White behave in a certain way? Why is it that Saul Goodman does certain things?

[00:19:16]So as these main characters are doing this again and again we start to you know remember those details and the other thing we also do is you know we don't have an infinite capacity to remember everything. So we also start to forget unimportant details like some of the subplots which you know are there to make the story move forward but they are not necessarily useful in you know arriving at a conclusion towards the climax of a single episode of the entire series. we tend to forget those details.

[00:19:47]Correct? So how does this help us in understanding LSTM? So let's go through this step by step. So the first thing we need is a memory bank. In the memory bank, we'll have to store important information. Now how do we add information to this memory bank? So let's say we have something called as a you know a gate that will allow us to add information to this memory bank. And then we also have to forget unimportant information because we have limited capacity in our memory bank. So we'll also need in addition to the add we'll also need a forgate gate. And what the fate gate does is it'll start removing unimportant information from the memory bank. Now as you're watching episode after episode, we will filter out the unimportant information from the memory bank. Then we will process the episode and add important information into our memory bank. Now let's go step by step.

[00:20:43]Let's say this is the first episode and in the first episode we have Skyler, Walter Jr., Walter, Hank, and Murray who show up in the episode. Uh, our memory bank is empty. So there's really nothing to forget. So we add all of them into our memory bank. And let's say just as an illustration, we will use the screen length or the time they spend on screen as a measure of how important their character is. Obviously that is not the right way of measuring it but for for as as an example I think that will do the that will serve the purpose for us.

[00:21:16]Okay. So let's do that. So what we do is we essentially say initially our memory bank had none of these characters. So their score was zero. And let's assume that each one each of these characters in the first episode have equal amount of time they appear on the screen. So we score them on a scale from 0 to 100 as 50 because each one has you know equal presence in the TV series. Okay. So they are all of equal importance at this stage. Now we move on to the second episode. In the second episode we essentially have Walter White coming back. We also have Hank coming back. But there are two new characters Steve Gomez and Jesse Pinkman who show up in our series. And we also see that, you know, the prior characters like Skyler White or Marie or Frank Jr. do not appear in the second episode. And again, we're going back to our measure of how long a person appears in the in that episode as a as a score or a measure of how important they are to the story line.

[00:22:17]Again I want to re repeat this is not the right way of measuring it but it serves the purpose of demonstrating how our brain processes information and how later on an LSTM will build on something like this.

[00:22:31]Okay with this second episode what we will do is okay we see that these characters and Skyler Murray and Walter Jr. do not appear so we have to forget some portions of you know their score in our memory bank. Okay. And we also see that newer characters are added. So first thing we do is that we decrease the score of these characters that do not appear like Skyler White goes from 50 to 25, Mary goes from 50 to 25. And Walter Junior goes from 50 to 25. We add the new characters that are showing up in the episode for the first time. So Steve Gomez was at zero. He went to 25.

[00:23:10]And we also see that Jesse Pinkman was not there in the prior episode. His score was zero in the memory bank. Now it has gone to 50. Why? Because Jesse Pinkpin appears a lot more times than Steve Gomez in this episode as an example. And we see that Walter White comes back into this episode and he also spends some time. He gets some screen time. So does Hank. And we essentially add more score to both Walter White as well as Hank. So our memory bank has now this information for each of these characters. Okay. Now let's get to the keep on watching more episode. Let's say we get to the 10th episode. In the 10th episode, let's say these characters appear. You have Jesse Pinkman, you have Skinny Pete, you have Badger, and you have Walter White. Now, we'll apply the same thing. Okay, you see that Walter White is appearing in all the episodes, and he's he's basically spending some screen time. So, and so does Jesse Pinkman. And then you have, you know, Badger and Skinny Pete appearing for the first time. So, we'll adjust their scores accordingly. And we say that okay you know let's say you know here Bajger Bajger is appearing for the first time initially in our memory bank his score was zero. We bump it up to 10 because he appears for just a few minutes in the entire episode. We essentially up the score of Walter White from 75 to 100 because the accumulated score of the accumulated time that Walter White's character spends on screen has been adding up over the last 10 episodes.

[00:24:41]Remember we are at the 10th episode right now. So does you know Jesse Pinkman. We essentially up the score from 50 to 75. And what we also do is we start to decrease the scores of people who have not been appear appearing as often in in the first 10 episodes that we have seen. For example, Steve Gomez originally went from 0 to 25. And over the last 10 episodes because he's not appearing enough, we start to decrease his score and we go from 25 to five.

[00:25:10]Same with you know Skyler White goes from 50 originally to 25 to 10 and so does Mary score go from 50 to 25 to 10.

[00:25:19]Then we keep continue to watch more episodes. You get to let's say episode 21. We get all these characters. We get Saul Goodman for example and his assistant Hule and then we apply the same logic. We increase the score. First of all we add new information. So Hule was not there in any of the prior 21 episodes. He's appearing for the first time here. So we add that to our memory bank. We don't know yet how important it is. We'll only know that as his character starts to reappear in the future episodes. And what we're doing here is even you know Saul Goodman appears for the first time. So we essentially give him some score. We add him and we give him some score. And we start start also to you know downscore those characters that are not really showing up that frequently in the last 21 episodes. So we continue this. we get to the 35th episode, newer characters get added. Existing characters, the scores increase and some of the existing characters, the score starts to decrease. For example, Hu appears sparingly in the in the episodes. So, we start to decrease his score. Same with Badger and so on. So, finally, we get to let's say episode 60 and more characters get added. So, we add them to the memory bank. We adjust the score of existing characters either up or down. So we're essentially down meaning we are forgetting some of their forgetting or rather we are downsampling their importance. So you're forgetting information about them and we can we are adding newer characters but we are also up scoring certain characters that are already in our memory bank because their importance starts to show up over all the episodes. Now finally we get to the end of the series and our score essentially looks like this in our memory bank. And this is in a way like a probability distribution of how important each character is to the entire series. So we essentially take that and then we started with a memory bank of all zeros. We could essentially arrive at a memory bank that has a probability distribution of how important each of these characters is to the entire story line. So the key takeaways here are you start with a concept of a memory bank. That's how our brain functions. And then as we are watching things, we start to add those into our memory bank. And as we continuing to watch, we will remove those things that are not really relevant because we do not have infinite capacity to remember everything. We will start to remove those that are less relevant and we will incre increase the score of those things that are more relevant to the story line. So this is the the core idea behind how you process long sequences when there are several you know subplots within it there are several characters within it and so on.

[00:28:12]Now how do we go from here to actually understanding LSTM? Let's study that next. Let's start with a highle architecture first. So let's actually start with a classical recurrent neural network. In a classical recurrent neural network which we which we have seen in the prior episode, you could actually unfold the neural network and it looks like this. The sequence is processed one element at a time and you would pass you'll start with an initialized state and that gets passed along with the sequence input and so on. So when you want to take this RNN classical RNN and convert it into an LSTM, we will need a few things. Number one is that we will need this concept of a memory. Uh a long-term memory, which you know it's like a whiteboard if you think about it.

[00:28:56]By the way, an LSTM is an optimization on top of an RNN. At its core, it still has this concept of unfolding and processing, you know, an input by first tokenizing it into a sequence and then processing it individual elements of a sequence at a time. So, it's basically an LSTM is a form of an RNN, an optimized form of an RNN. Okay. So, our goal now is to take this architecture and convert it by adding a few things.

[00:29:24]Number one is that we add this concept of a long-term memory. the this context state that exists in an RNN which in an LSTM is called as a short-term memory that continues to exist. But we also need to capture this long-term memory.

[00:29:41]That is what this C stand here is what it stands for. It's called cell state in the literature. Now we unroll it just like you unroll an RNN and both the the short-term memory which is represented using H as well as the long-term memory which is represented using capital C get passed to each of the RS LSTM cells. Now we will need to have three gates in each of these cells.

[00:30:09]The first gate is it'll remove unimportant information from our memory.

[00:30:12]So it'll remove unimportant information from our long-term memory. The second one is the add gate which will add important information to our long-term memory. And then the third one is what is called as an output gate which will pass the information that is you know connecting the context that is flowing across all the words in a sentence.

[00:30:35]Okay. Now basically the same cell again because it's un unfolded it appears again and again and the long-term memory and the short-term memory are the ones that tie all these things together in a sequence. Okay. So in order to understand LSTM we actually have to understand five things five concept. One is the hidden state. This hidden state also exists in a RNN and this is the one that passes the context between adjacent words in a sentence. To that we are al we adding this new concept called as a cell state which is a long-term memory. So think about this as a whiteboard in whiteboard that is tracking key characters uh or key subplots across multiple episodes and seasons and so on. Then you'll need to have the gates. The first one as I said is a forget gate. It's a knob that controls how much unimportant information we have to forget. Okay.

[00:31:26]Then we have to have an input gate. It's a knob that controls how much information important information we actually keep in our memory bank in our cell state. And final one is the output gate which essentially tells how you pass along the hidden state the context from one word to the other word in the sentence. Okay, as you're processing that sentence. So the first thing we need to actually address here is how does an LSTM actually know what is important because we saying that the long-term memory keeps you know important information and the gates essentially modulate how much of that important so-called quote unquote the important information gets added or removed. So how does an LSTM even know what is actually important? The interesting thing is it actually does not really know what is important but it learns it through the training process.

[00:32:14]And the way it does that is first is it starts with some initial randomness.

[00:32:17]Okay you initialize the weights to some random values and we'll see what are those weight matrices shortly but it initializes it to random. So it really does not know anything about what is important. The next thing is as we are training we show that network many examples. Let's say we're doing a network that is being trained on movie reviews and to identify the sentiment that's conveyed by that review. You show it many such movie reviews and you pair that with the pre-labeled you know whether that label says the movie is good or bad and so on. So the LSTM actually looks at all that training data then it makes predictions and if its predictions are wrong then we back propagate that you know through the network so that the weights are adjusted. Okay. So this is nothing different than any of the neural networks that you have studied so far.

[00:33:04]But what will also happen is that because of the way the LSTM is designed, it'll actually start to recognize patterns. It'll basically recognize things like if there are words like let's say for example amazing and terrible consistently then it'll say okay these kind of words seem to correlate more with certain types of sentiments. If it's the word amazing exists then there is a likelihood that that review is positive. If the word terrible appears there is a likelihood that you know the review is not good. So the LSTM starts to build recognize this pattern and build things around that by recognizing these words and and their correlation with the sentiments that they convey. They also you know understand that there are certain words that can flip the meaning of you know subsequent words. For example, if it sees something like a despite or however, it'll start to learn that okay, these things, you know, are are flipping the meaning of what appeared before the these, you know, these kind of words appear. Okay, it learns that and then we have these gates, the input and the forget and the output which the LSTM learns, you know, when to open and when to close, how much to open and how much to close based on how its predictions are relative to the ground truth value in the training data set. Okay. And then you know it'll also start to look at you know what are those words which are strong indicators of a positive or a know negative or a neutral sentiment.

[00:34:32]And all this goes back to you know its prediction and the ground truth value that says you know what is the actual sentiment that's being conveyed by the movie review. So using these gates and there's long-term memory and the short-term memory and all of those things the LSTM starts to figure out what is important and what is not important. Maybe another intuitive way of looking at it is you know if it is not important okay then its prediction will be off if it is let's say let's say it is not important and the LSTM is not you know downscoring you know those words then it prediction is going to be off and if it is important and it is not it is not upscoring them even then the prediction is off so over a lot of you know back and forth through the forward progress as well as forward pass as well as the backward pass. The LSTM figures out what is important and what is not.

[00:35:30]Okay, so now let's dive into the architecture. Again, a quick recap. The LSTM looks like this. When it's unrolled, you have this long-term memory passes a cell state. The short-term memory which is denoted as H here. And each of these cells has these three gates is a forget gate, input gate, and an output gate. Okay. Now let's double click and see what are these what each of these gates do. Okay.

[00:35:56]So by the way these are the different formulas that actually express you know how these gates operate and just by looking at it you know somebody who's new to this will start to get intimidated because there's a lot of there's you know there appears that there's a lot of math here a lot of formulas and so on. But actually if you understand this intuitively first then you'll you'll see that you know this is actually not really that complicated and it actually clicks because there's a certain way in which this thing is designed that is very natural. Okay. So let me explain that. So the way I will explain this is I will go through each of these gates. Okay. And I'll explain what is really happening and why is it happening that way and why is it that these specific arrangements will actually let's say enable the LSTM to forget unimportant information or enable it to add important information into its long-term memory and so on. Okay, let's start with the forget gate. Again, the forget gate is forgetting unimportant information from our long-term memory.

[00:37:01]Okay, so we'll focus on this gate here.

[00:37:04]Okay. All right. So in the first thing so this is the token that is happen that's coming in at time step some t okay and the previous hidden state the short-term memory state is ht minus one and the whiteboard the memory that we have is expresses ct minus one okay now what we have to do in the forget gate is that we have to forget unimportant information from the long-term memory Okay. Now the first thing is how is the input actually propagated into this gate. The way it is propagated is that you take the embedding the word embedding for this token and then you concatenate it. The matrix here this matrix here you concat this matrix is the embedding word embedding for this token. You concatenate that with the the previous short-term memory state. Okay, that's what you do. So it's basically nothing but concatenating two matrices. You take XT the word embedding for this token and then you concatenate it with the short-term memory multiplied with the learnable weight matrix that we have here. This is this part is exactly like how you have it in a RNN a classical RNN. Now this is the input that goes to all the gates by the way Z. Okay, it's a concatenation of the token plus a multiplication of the previous the previous hidden state with the weight matrix for that learnable or trainable weight matrix for that. Now what you do is that the forget gate is actually calculated as this input which we'll now start calling Z. Okay, going forward this Z times another trainable or learnable weight matrix called WF which is weight matrix for the forget gate.

[00:38:55]Okay, that's what it does. And then you take this gates value which is calculated as a sigmoid of WF * this input Z and you multiply it with what is coming in through the the input here.

[00:39:10]Okay, CT minus one. Okay, so let's study this a little bit more. By the way, this when expressed fully is C forgotten is that which is basically the our long long-term memory or the whiteboard is basically what was there in the whiteboard so far times the value of the forget gate. Okay, let's double click that further. Okay, this is what it is. Now the first thing here is the information from the previous episodes.

[00:39:43]So let's say the previous episodes you had Skyler White, you had you know the Walter Jr. and you had Walter White and you had Hank and you had Marie. Okay, this is what HT minus one is. Then you information from the current episode which is in the current episode you have Walter White appearing, you have Steve Gomez appearing, you have Hankry appearing and then you have this new character which is Jesse Pinkman appearing. Okay. Now what is there in our whiteboard currently our long-term memory you had you know so far what we have seen is that Walter White and Skyler White and you know Walt Jr. and Hank and Mary all have equal scores of 50 50 and so on. Now what you're doing here is you have to forget some portions of these three characters because they don't appear in our current episode. See in this current episode only these characters appear. you have to forget this. So how do you do that mathematically? The way you do that mathematically is that if you can multiply these values with something which are which is you know a value between 0 and one you can essentially decrease their score because what is it that you're doing? Forgetting means what you're decreasing the score of these characters. One great way of doing that if you want to kind of get a number between 0 and one to multiply with these cells is sigmoid function because sigmoid takes any value and it squashes that value however big or however small it is. It squashes that value into a value between zero and one. That's what sigmoid does. Okay. Now what you could treat it as okay the sigmoid value that comes out is one then means that you don't forget anything about this person.

[00:41:32]If the sigmoid value is let's say 0.9 it means that you forget 10%. Say a number multiplied by 0.9 means that 90% of that value is still retained. 10% is forgotten correct and so on. If the sigmoid value is let's say 0.2 which means that you are retaining you know 80% sorry you're retaining 20% and you're forgetting 80% of that information because a number multiplied by 0.2 two you know essentially it will keep 20%. And the remaining 80% is thrown away. So you know that sigmoid function is a great function that can be used if you want to forget or downgrade the score of those values that already exist in our whiteboard. That is the reason why the forget gate essentially uses a sigmoid. But it is sigmoid of what it is doing a sigmoid of the input that is coming in. And that input is basically a mix of a concatenation of the previous state that's flowing through our network so far and the current you know at this time step the current value correct XT and HT minus one. So once you do that you can essentially down sample and forget the relevance of certain characters from our long-term memory. Okay. Now here if the value of let's say vault is not changing it's because you want to not forget anything about these characters because you know they reappear in our current episode. So essentially the sigmoid you should swing it more towards one both for Hank. So both for Hank as well as Walter White. Okay. And you have to swing let's say the values for you know Skyler White as well as Walt Jr. and Murray by you know 50%. So that you would essentially decrease their score. Okay. How much percentage would increase and decrease?

[00:43:34]It all comes down to you know you make some prediction. First of all you this this weight matrix here W F is the one that is influencing this correct this is the one that is influencing it. So what you have to do is that since this is randomly initialized initially you will make all wrong assumptions and you'll either you know in fact you might end up you know forgetting some portions of Walt and Hank as well which you should not and you might actually increase the weight of know or rather keep the weights the same of you know Skyler and Marie and so on. But as we compare our prediction with the ground truth value you'll realize that hey you know the way we are swinging the scores of these individual characters is incorrect and over time we will start to swing it in the right direction so that you know we forget less important information from our long-term memory. So one key thing to observe here is that we are this long-term memory is only decreasing the scores of people that it has deemed as less important as it is processing the current episode. But what happens to these new characters that are not there in the prior you know or long-term memory that's not here. And the reason it's not here is the role of a forget gate is to take what already exists which is ct minus one multiply it with what is coming in in the current episode along with what it has seen so far.

[00:45:05]Okay. So it will kind of change the values of those characters that are already in the whiteboard. There is no provision here to add anything. Correct.

[00:45:16]It's only changing the scores of characters that already exist in our whiteboard. So that is what the forget gate does. Okay. So when the forget gate is done executing it will emit a updated value of our long-term memory with scores adjusted so that less important characters their score starts to go down. Important characters their scores remains the same. Okay. We also have these two characters Jesse Pinkman and Steve Gomez appearing for the first time in our episode and but they don't appear in our whiteboard. So we have to add them to our whiteboard. How do we do that? For that we have to use the input gate. And what is the purpose of the input gate?

[00:46:01]The input gate adds important information to our long-term memory. So now we'll focus on understanding how this gate works and how the math behind it works. the intuition as well as actually what is really happening when when these gates actually process processes this information. Okay, so by the way what is the input to the input gate? The input is basically the same the concatenation of the current tokens embedding it's text for example and the previous state correct and now this input actually goes into two functions one is a sigmoid function you see I represented this using what I call as I what is what information needs to get added and the second one is I how much which is the input how much is how much of that information needs to get added by the way the literature doesn't call it out this way. But I think that expressing it like this makes it easy for us to you know kind of do a mapping be behind you know do a mapping between the math and what is really happening anyway. So the input goes through what needs to get added and how much of it needs to get added and then using some math here which we'll explore very shortly we will update our whiteboard or long-term memory. Okay. So this is the the function for the input gate. So the first thing is the previous state that is flowing through is our ht minus one.

[00:47:27]The current episode has these characters h xt. Right? Now the existing memory, the long-term memory will have these values. Remember that it has already gone through the forget gate and we have suppressed the values of these three characters because their importance is less as we are processing episode after episode. So this is going to be your existing or current man updated whiteboard. Okay. Now this is the information that you're adding. You're adding to the whiteboard here for the first time. You're adding here. Okay.

[00:47:59]You forgotten unimportant information.

[00:48:00]Now you're adding important information.

[00:48:03]Okay. Let's study this important information adding. Now the first thing is a sigmoid. So sigmoid basically which is represented as this I thing is what information needs to get added? what is important that needs to get added and again a sigmoid is a great way of taking any value and you know squashing it between 0 and one. So if it is zero it is not important if it is one it is very important. So you' essentially use that sigmoid function over the input which is Z and you'll use a trainable weight matrix W A and you'll multiply that and take a sigmoid of that that will tell us you know how important does this new information that is coming in and it'll score it between 0 and one and that's what you're going to get. Okay for example Steve and Jesse both appear for the first time. So obviously they need to get added but Jesse has 2x more screen time than Steve. So we have to amplify you know Jesse's score more.

[00:48:56]Although both of these are important information that needs to get added. We actually use the sigmoid function to swing you know a higher value to Jesse than to steal. Now you also need tanage.

[00:49:06]What the tanage does is it'll also say for the existing you know characters that are there in our whiteboard you know how much do we amplify their signal or how much do we change their signal strength basically Hank and you know Walter are already there and again they appear in this episode so we don't have to add them but we have to amplify their signal you know further and tanh is a great way to amplify it or suppress it based on you know this function of tanh which takes any number however large or however small and squashes that between a negative one one and a one okay so it's a great way to kind of modulate the strength signal strength of individual you know characters in that are there in our long-term memory so that's essentially what is happening over here so you take the multiplication of these two the I what so what is important and how much of that is important you multiply that and then you add that to the information that you have gotten from the forget gate. So you update your long-term memory based on that. Okay. So at this point what has happened? So number one you added the new characters.

[00:50:20]You added Steve and you have added Jesse. Both of them initially had a score of zero. You added them and you have made the scores appropriate based on you know like what is their importance to the plot line. Okay. So Steve is less so. So he essentially add the sigmoid essentially swings the score towards let's say 25 and Jesse is more so it gives him more score and swings it towards 50. Uh Walter White and Hank appear again. So as a result you actually swing it up swing up their score further and you essentially increase it from 50 to 75 for both of them. Okay. So at this point we have updated our long-term memory. We have taken out unimportant information using the forget gate or rather we have suppressed it and we have added you know important information which doesn't already exist to our long-term memory as well as amplified the signal strength of the those that are important using the tanh function. So now we have an updated long-term memory. Okay. Now let's get to the output gate. The output gate is this is basically the short-term memory and you need to pass that short-term memory to the next element in our sequence. So how do we do that? So we'll focus on this gate right now the output gate and this is the math for that. It's relatively much simpler than the previous two ones. So let's understand this. What is the input to the output gate? It's the same thing the concatenation of the previous short-term memory with the word embedding for the current input. That is what you pass.

[00:51:56]And then actually let's expand this.

[00:51:58]Okay. So the updated short-term memory is calculated by the output gate as following sigmoid of the input that's coming in. Again the input is a concatenation of the current input tokens embedding and the previous short-term memory and then tanh of the white whiteboard which is the long-term memory that we have calculated in the previous steps. Okay, remember one thing at this stage when you get to the output gate pretty much the long-term memory is all taken care of. We have essentially computed the changes that need to be made to the long-term memory by eliminating what is unimportant and adding what is more important. Correct?

[00:52:43]So that doesn't get changed in the output gate at all. Only thing that gets changed in the output gate is a short-term memory. So this is let us say our long-term memory that you know it's set to these values. The previous short-term memory is this. The current input is this. So what the sigmoid does is again how much do we amplify or suppress the weightage of each of these characters in the short-term memory as they are progressing from episode to episode or one element of the sequence to the other. Again, sigmoid is a great way to squash the values between 0 and one. So that's why sigmoid is used here.

[00:53:23]And then you actually multiply that with the value that is there in the long-term memory because you also want to see you know the information that has appeared a that has been accumulating so far that has appeared in the previous let's say we are in the the 40th episode. the long-term memory is carrying information that is there way back all the way from the first episode and so on. So we also want to take that signal strength for each of these characters and multiply that with the short-term memory. That's why you use tanh. Again tanh ensures that the the value is bounded between a negative 1 and a positive one. Okay, that is the reason why a tanh is used on the long-term memory and you take the multiplication of both of those things.

[00:54:09]So just to summarize our understanding of LSTM, what is happening here is in an LSTM you have this long-term memory and then you have the short-term memory and you pass this through the LSTM cells through three gates, the forget gate, the input gate and the output gate. The long-term memory basically carries accumulated information or the context over long time time periods and it uses the forget gate to suppress unimportant information. uses the input gate to add new new or amplifies you know important existing current information in the long-term memory. It has no direct role in the output gate has no direct role in manipulating uh you know the long-term memory. It has an indirect role because you know the output gate changes HT and the which is the short-term memory and the short the long-term memory also depends on the you know the short-term memory because if you look at the value Z is a function of both the short-term memory as well as the current input.

[00:55:10]There's no direct input or direct role that an output gate plays in manipulating the long-term memory.

[00:55:19]Now the summary again the short-term memory essentially carries the context that is flowing across shorter timestamps time steps. So it's more influenced by words that appear more. So the forget gate has no role to manipulate the short-term memory. The input gate has no role to manipulate the the short-term memory at least no direct role. It has an indirect role because again the short-term memory also depends on CT. CT depends on the forget and input gate. But there's no direct dependency between the manipulation of the short-term memory H with forget gate or the input gate. But the short-term memory is updated using the output gate.

[00:56:00]Okay. Now why does something like this actually address the vanishing gradient problem? Remember I told you that there is a vanishing gradient problem where if the sequence length is too long the gradient descent algorithm essentially as it is updating the weights of these matrices of all these matrices as it's progressing backwards in time through all the time steps the gradient value starts to go so small that they go to zero and as a result there is no change that is happening to these weight matrices and the reason we explained that also is Because in the classical RNN the short there there's only short-term memory and the short-term memory has a multiplicative relationship. The current value is a multiplicative relationship with the previous value. And when you take the gradient of this essentially the partial derivative of this it becomes an exponentially decaying value because you know uh a small value multiplied by another small value is a much more smaller value multiplied by another small value is much much more smaller value and so on and there is no way to add information to the short-term memory. That's where the long-term memory comes in because even in a long-term memory, you have this multiplicative effect which kind of as you're doing back propagation in time, the gradient value becomes much more smaller and tends to go towards zero and as a result you know in the previous time steps the change in the weight matrices is going to be less. However, the long-term memory also has an editive property. Because of the editive property, what will happen is regardless of how long the sequence starts to get, you'll have this addition and that addition protects itself from making the gradient so small that the weights do not change. Okay, that is the actual intuition behind why the vanishing gradient problem does not really exist to the extent that it does in a RNN. Of course if you give it a very very very long sequence let's say you give it a sequence of a thousand word sentence and things like that even there you would see that you know an LSTM will start to tr struggle because again regardless of the whether you are adding or not these values essentially because of tanh and all that they are smaller values any values usually between -1 and positive one so as you're going back in time across all these thousand time steps you'll see that the values start to diminish But thousand is unrealistically long sentence. So for sentences that are let's say 50 or 100 words which is also pretty large. What they have observed is that because of this editive you know thing that's happening within the long-term memory of a LSTM you don't have the vanishing gradient problem that an RNN has.

[00:58:57]Okay. Now let's revisit this important thing again just as a quick recap. As I said in the beginning, LSTM inherently does not know what is important. It learns this through the training process. Initially, it starts with random data. Then it trains. You give it a lot of examples and then it you know trains over that. It prop back propagates, updates its weights and so on. But what happens here is that because of this long-term memory and because the gradients are not vanishing, it will start to make correlations that you know even the words like amazing and terrible even if they appear much earlier in the sentence or much earlier in the sequence because of the erative property of the long-term memory. It will be essentially start to associate the right sentiment. How does it know what the right sentiment is? That is available as a ground truth value. write the ground truth label for let's say you know of the review sentiment that is there in our not training data. So you'll be able to ass make that association even though these words like you know the the words that influence the actual sentiment things like amazing and terrible even though they appear much earlier in the sentence. It also starts to recognize that you know there are certain things that you know you could have a very positive let's say 90% of the sentence all saying positive things and then towards the last 10% of the sentence you have words like despite and however that flips the meaning correct because of the short-term memory with you know the long-term memory it'll know that you know there are certain words that trigger a flip in the sentiment so That's another thing that the combination of the short-term and the long-term memory as the as it's architected or designed in the LSTM helps the network to learn. And of course the gates enable it to kind of figure out how much of the information to pass in, how much of the information to suppress, what gate to open, what gate to close and so on. Okay. So over time, you know, when you train the LSTM with a lot of data, it actually figures out what is important, what is not important, what words trigger flipping uh of the sentiment, what words trigger, you know, pushing the network weights in such a way that it is more aligned with the ground truth values and so on. Of course, uh our web series is not complete without actually implementing the concepts that we have learned. So let me show you how you would implement an LSTM. Obviously when you're using it in for practical production use cases, you would use library functions that have been optimized. But before you start using a library function, let's actually handroll an LSTM to understand its core concepts better. Okay, that is the best way to actually understand concepts, not just using library functions. Okay, so let's look at that.

[01:01:53]So this is our familiar picture. the LSTM cell with all the forget input and output gates with the long-term memory which is called as a cell state and the short-term memory which is the HT okay and these are the formulas that show you how these gates operate and how these values are manipulated uh so I handcoded this LSTM implementation using just numpy and it helps us actually you know figure out how would you translate this code from concepts to actual implementation Okay. So what I have is I'm I'm initializing all the weight matrices WF, WA, WB, and WO. I'm initializing them to random values. Just like in any neural network, you start off by initializing it to random values.

[01:02:38]I'm also initializing a neural network also needs to have bias values. I'm also initializing the bias values. One thing that you notice here is that in any of the formulas or any of these pictures, I don't show the bias matrices. That is only because it's much easier to you know see a picture if it's not overcrowded with all these matrices. So I've skipped that. But in reality you also need with every m weight matrix you also need to have the bias matrix. Why you need to have a bias? I have explained this very clearly in episode 1 of season 1. So if you do not know why a bias is needed, you can go back and reference that one. Okay. But anyway, so I have initializing the weights and the corresponding bias matrices all to zero.

[01:03:18]Now, all to random. The bias is always initialized to zero. I also explained this in season one episode 1. Okay. We also of course when you have to make the final prediction, you need to have an output matrix. So that is also initialized to random and the bias for the output is also initialized to zeros.

[01:03:35]Okay. All right. So across multiple training epochs, let's say I'm training across 10,000 epochs. uh I take a sentence I take a collection of sentences then I take one sentence and what we do is that we tokenize the sentence okay you tokenize the sentence and then you process the one word of the sentence at a time but before that you need to initialize your long-term and short-term memory so the short-term memory let's say is called h brief I initialize that to zero which is this one here I initialize it to zero. H0 is zero. The short long-term memory or the cell state is initialized also to zero. Okay. Now I have this I also keep a record of all the manipulations that are happening to the long-term and short-term memory in an array here. Okay. And also the predictions that are coming out of these individual LSTM cells in this array called predictions. Now for each word in our sentence I have tokenized this. say for each word I have to get the embedding because again neural networks do not understand text they only understand numbers how do you get uh a numeric representation of a text you get that using word embeddings I explained that in season 2 episode 2 you can go back and reference that if you don't know what this is okay so I get the word embedding for that word initially start at t0 which is the first element in the you know the sequence is the first word then I have to concaten ate that right with the previous short-term memory. So it's basically array concatenation of the current word embedding and the short-term memory. Now that is what I need to pass through each of the gates.

[01:05:24]How do I pass that through through the forget gate? I take the combined value multiply it with the weight value WF right combined weight combined multiply it with WF add to that the bias for the forget gate and take a sigmoid of that that's what is happening here sigmoid of Z * WF Z is concatenation of X and the short-term memory correct and then you add the bias value and take a sigmoid of that that is your forget okay what is input input is you take Z you take a sigmoid of the combined Z * W A which is this combined is Z add that to the bias and you take a sigmoid of that this is I what what are you adding to the long-term memory multiplied with how much of the information or how much of the signal strength of that information needs to be added which is done through the tanh tanh is again the same input multiplied by its own weight matrix WB added to the bias mat matrix there and you take a d of that and you multiply these two so you will get the signal strength as well as what needs to get added to the long-term memory.

[01:06:38]Now the output gates OT value is calculated as what again the combined Z value which is Z again is the combination of the word embedding at the current time step with the previous short-term memory with its own weight matrix called WO and the bias matrix and then you take a sigma that's OT okay now you have to manipulate the long-term memory how do you manipulate the long-term memory long-term memory DT is the previous long-term memory multiplied by the forget gate plus the input that is coming in. So what is forgotten or rather what information needs to be forgotten add to it what information needs to get added and that becomes your new long-term memory and your short-term memory is what is the output gate the value of the output gate times how much of the long-term memory values also influence the short-term memory that is your updated short-term memory okay from this you would make a prediction Yeah. And you store that prediction. If you're doing just, you know, sentiment analysis or you're doing, you know, predicting the next token in a sequence and all that, you really don't need to store these intermediate predictions that come out of the each time step. You only need that when you're doing language translations where each word needs to be translated into a corresponding word in a different language and all that. Only then typically you'll store the individual intermediate predictions.

[01:08:15]Otherwise, you're only interested in the final prediction. Anyway, here I'm just storing it just as a illustration. Okay?

[01:08:21]And I'm also storing the the you know short-term memory and the long-term memory as it's getting updated. Okay? So now I pass this updated short-term memory and long-term memory back to the next token in the sequence. So you go and pick up the next token or the next word in the sequence.

[01:08:40]Go through the same motions again. Okay?

[01:08:42]Then go to the next word in the sequence. go through the same motions again and again and again till you are done with all the words in the sequence.

[01:08:49]Okay. Now once you have done that you have gotten your prediction and you have to compare how you are predicted with the ground truth value and that will give you a loss. You use a loss function. Typically the loss function is a cross entropy loss function and when you get that loss you essentially back propagate that loss again back propagate in time because you have to go through first adjust all these weights WB wa wf and so on in this time step then you have to back propagate adjust it in this time step same weights again adjusted in this time step back propagate and adjust in this time step back propagate adjust in this time step and so on that's where you reverse it and you back back propagate and adjust the weights you know in reverse order. Again because of this editor function here you see that even for longer sequences this weight values essentially the gradients do not for the long-term memory do not go down to zero. So there is you do not have the vanishing gradient problem that is inherent in a classical recurrent neural network. So this is our implementation of a uh LSTM.

[01:10:03]Now where are we going to draw this boundary? Is the boundary or where you want to capture the short and long-term memory across a sentence? Is it across a paragraph and so on? So for that let me illustrate that using you know how you see this in the context of very large you know input data. So let's say that you have uh in a book series let's say the Harry Potter book series which has seven you know books in that series. Now each of those books so seven books in the series each of the books has several chapters each of the chapters have several paragraphs each of those paragraphs have several sentences each of those sentences have several tokens.

[01:10:44]So do you put do you capture the long-term context on every book or every chapter within that book, every paragraph within that chapter or where do you capture this? So the way to look at this is if you have hierarchical information like this and you would like to make sense of okay for example u what would be you're trying to understand the sentiment of a single sentence you obviously need to capture the context around each sentence correct but let's say you want to understand the sentiment of a paragraph then you obviously need to capture it at a paragraph boundary but what if you want to say hey what has really happened in this chapter then you need to capture the context around all the paragraphs across all sentences within that chapter. Similar to that, if you want to say, hey, what really happened in let's say prisoner of as a book then you'll need to capture the context at the book level. So when you have hierarchical information like this, the way you process this is essentially to have multiple LSTMs. Okay, which is what I wanted to illustrate here. So essentially I have one LSTM on a book uh for every book. Then I have you know then a chapter based LSTM. Then I have a paragraph based LSTM and a sentence based LSTM. And what we do is that you know you basically take all the book series. Then you break up each take take each book in that book series. You break it up into each book into its individual chapters. Then each of those chapters you break it into individual paragraphs.

[01:12:14]And each of those paragraphs you break it into individual sentences. Then you break each individual sentence into its own set of tokens. So the inner loop essentially then becomes processing in LSTM on a sentence boundary. So which means that you reset your long-term and short-term memory for that the sentence LSTM and what you would do is that okay you get let's say the actually let's let's look at this. So what you have done here is let's say you start with the inner loop. Okay, so innermost loop which is every sentence you're processing. Now when you're processing every sentence you would basically say okay I'm going to split every sentence into its individual tokens and I'm going to basically get the word embeddings for each of those tokens because again neural networks do not understand text.

[01:13:04]They only understand numbers. So you need to get word embeddings for each of those textual tokens. Okay. So that's what you capture in this token embeddings and that is what you pass as an input to your sentence LSTM. Okay.

[01:13:17]And those token embeddings go in and you process it like we have seen before. You go through the forget gate, input gate and so on. You process the long-term short-term memory update and you go through all the elements or all the tokens in the sentence and what you pass back is basically the the short-term memory or the context that is flowing across all these words in the uh sentence. Okay. Now that you would store for each sentence you'll store in something called as a sentence embedding because now you have to process at a paragraph level. So you concrete each sentence okay each sentence it's emitting out it's updated in a short-term memory you add that to the sentence embedding matrix and that sentence embedding matrix essentially becomes input to your paragraph level uh LSTM okay and then you run the same thing same LSTM logic across the paragraph and each individual paragraphs for each of those things the short-term memory is actually captured captured in another array called parag paragraph embeddings and that becomes an input the paragraph embeddings becomes an input to your chapter level LSTM and so on. Now one question why are we actually using the short-term memory and not the long-term memory. The reason is number one is that actually the short-term memory if you go back let's just go back here. So you see that the short-term memory is basically already capturing the information that is there in the long-term memory. So that's the reason why we use a short-term memory in propagating the context and the embeddings for the next LSTM in our hierarchy. So this is how you would process you know things that are hierarchical in nature. This is a textual example even for time series data. Let us say that you have you are capturing the system metrics CPU utilization, memory utilization, storage IOPS, network packets per second, the database queries per second, all those individual metrics. Let's say you're capturing it every second. Okay. So you can and you can say that you know I will create a sequence uh window of let's say every 60 seconds to kind of understand what's happening in my system every 60 seconds. But you would also like to sometimes make a query and say hey I want to understand what's happening in this hour boundary or minute boundary or you know day boundary or week boundary or month boundary and all that. So there's a hierarchy there. So what you can do is you can essentially structure this as multiple LSTMs. You would have one LSTM that's tracking information on a per second basis. Another LSTM that is tracking information on a minute basis.

[01:16:01]Another LSTM on an hour basis. Another LSTM on a day basis. another LSTM on a week basis, another LSTM on a month basis, another LSTM on a year basis and so on. So you have all these hierarchies. So if you want to make the right prediction what has happened let's say what is going to happen in the next minute you would use the minute LSTM.

[01:16:20]Although you captured the individual data across every second because of this way hierarchical processing you will use the minute LSTM. Let's say you want to make a query and say what's going to happen next week you would use the week LSTM. Okay, next month you'll use the month LSTM. Just like in the case of these books, if you want to understand what has happened in this chapter, you would use the chapter LSTM. What's happening in this paragraph? You'll use the paragraph LSTM. What's happening in this book? You'll use the book LSTM. So that's how you need to think about data that has hierarchy built in. Okay.

[01:16:56]Now let's see a actual coding demonstration of how you would use LSTM in your own coding projects. So for our coding project what we will do is we'll build a LSTM model that can understand news articles and once it is done processing a news article we will give it a short prompt and we will ask our LSTM to complete the sentence.

[01:17:23]Now what I'm going to do here is I'm going to import the necessary libraries.

[01:17:27]Then I'm going to be downloading a article from the internet. I'm going to be downloading this article which is basically a news report or briefing or statement that was put out by the white house when India's Prime Minister Modi had visited uh United States. This came out in February 13th of 2025. Okay, let's use that. It's a decently long article and we want to build a LSTM that can process this article so that if we give it a few keyword prompts it can complete the sentence for us. Okay. So what I will do is I'm going to download this article.

[01:18:07]Okay. And I'm going to extract just the text from it. And for that I'm using you know pre-built Python libraries like beautiful soup to do that. Okay. It's going to parse the HTML, take out all the unnecessary tags and just give me just the text that is there uh that represents this article. Okay, so this is the text right now. What I'm going to do is I'm going to break this long text into individual sentences and I'm also going to tokenize this text so that I can build my vocabulary. So I've done that and I've seen that the maximum sentence length in this article is 68 uh tokens long and the total number of unique words that I see in this article is,50 and these are the different unique words. So this is b essentially my vocabulary okay about a vocabulary of size,50. Now LSTM is an optimization on top of a recurrent neural network which means that just like an RNN you'll need to take your input and make sequences and pass those sequences to the LSTM. I showed you how that is done using various examples in the previous episode season 2 episode 3. So if you do not know how that is done, I will not repeat it here, but you can go back and reference that. But I just written a very small piece of code here that will take all this information and it will convert this into all these sentences into sequences that can be fed into my LSTM. Now let's build the By the way, when it's processed, it looks like this.

[01:19:42]0 is all padding. I'm padding it shorter sentences with pad tokens which have a token ID of zero in our vocabulary so that each of my inputs to my LSTM is of maximum sentence length which is uh 68 here. Okay, you can see this here 68.

[01:20:02]All right. So that's all I'm doing here and I'm also producing a ground truth value which is the next token in in in that particular sequence. Again I don't want to go into the details. I actually explained this using multiple examples in the previous episode. You can go back and reference that. Okay. So now that I have tokenized it and I've also converted all my sentences into sequences, I'm going to be building my LSTM neural network. So the way I do that is I will use the TensorFlow kit as you know sequential the the module and then I'm going to add an embedding layer. Remember that a neural network needs to convert text to numbers. So that is done through word embedding. So I again I show this in the episode two of season 2 as well as episode 3 of season 2. All I can do is I all I need to do is I add an embedding layer which takes the total number of words that are there in my vocabulary and whatever the embedding dimensions that I'd like to learn which is 300 year and what is the maximum sentence length that is one layer and then I create an LSTM layer. This one will essentially create the LSTM model that I showed you architecturally. And this 150 essentially says that the weight matrix that it creates is of size 150 by 150 cells. Okay, I'm predicting the next word because again what is my goal? My goal is that I want the LSTM to read all these sentences and start predicting the next word and as it and in a loop I can keep on predicting next words to form sentences. So how do you predict the next word? You predict that by soft maxing across multiple all the words in your vocabulary. This is again something that I have explained in multiple episodes. But if you don't understand this, you can go back and uh look at the previous episode on RNN and I'll it explains clearly why we do this here.

[01:22:03]Okay. Anyway, so this is this last layer is an output layer that is using soft mass as my output function. And for the loss function, I'm going to be using the cross entropy loss function with an optimizer of Adam. Okay, let me compile this. If you notice here, this is my I mean I've not really uh run this yet. So it's not really formed all the connections yet. But now let me run this. Okay. So when I run this across 100 epochs with a batch size of 30, you see that the loss is starting off very high at 6.5. Remember that the loss needs to go down to a very small number, typically 0.1 or something like that, which is when you know that your neural network is trained well. So here it starts off very high, but across multiple epochs, it's the loss is going down. and it's converging down to a smaller value which is what we want which means that our neural network is actually learning the patterns that are exhibited by the different words and sentences that are there in this news article. So at the let's say 45th epoch the loss has gone to about 0.17 something it started at what it started at six I think. Yeah six and it is going down. Yeah let it go down. Let's let it run till 100 epochs. By the way, this is a very small sample set. Ideally, you need to what you need to do is you need to give it I would say hundreds or thousands actually literally tens and thousands of u such articles so that it can you know start to make meaning out of you know these sentences and what these articles are saying and so on. Okay, so we're almost there. Let's just wait for a few more seconds and then we should be done with training our small LSTM neural network. Okay, there you go. So, we have trained it. Um, our loss has not gone all the way to down to zero, but it is low enough. It started at 6 something.

[01:24:09]It's gone down to 0.0853, which means that it has it has learned something. Actually, let's do this. So I'll do lsdm dot summary to see what is my network size.

[01:24:21]Okay. So if you can see this network has about 2.2 million parameters of which 744 of them are trainable parameters.

[01:24:32]Okay. So it's a very small network in the context of know language models.

[01:24:38]This is considered to be a very very small tiny mod model which is fine because our goal is not to go and train it across all the articles on the internet but just to get an understanding of how LSTM is used. Now before we start to use this train model to start predicting sentences or sentences let's just go and do a quick recap of what we have seen so far in terms of how the LSTM is built. You have your embedding layer attached to that an LSTM So it's basically a few characters, one line of code and you can essentially build u an LSTM model. Okay, using the TensorFlow kiraas library. Okay, now that our model is trained, let's complete the sentence. So what I'm doing here is I'm going to give it uh a prompt. Let's say I just give it one word. We'll give more word shortly, but let's say I give it one word. Modi who is the prime minister of India clearly because you know this is article is about Modi's uh visit to United States.

[01:25:35]Okay. So I want to see how it completes the sentence if I just give it one word called Modi. And I would like it to complete the sentences by uh adding 20 more words that come after Modi. Okay.

[01:25:47]And the way it is done uh is basically you start with tokenizing your seed text. Seed text is the prompt here. And then you basically go after the total number of tokens you'd like to predict which is 20. Here you would do the same thing. You would pad the sequence with maximum sentence length. Then you will pass it to the prediction model.predict model is basically the LSTM model that we built.

[01:26:13]Then it predicts the next word. We will use a vocabulary to convert the token ID into the actual word. Uh and then print that word. Then we go and add that word to our the seed text to our sequence because modi and let's say it created another word. You add that word. then you pass it back, it predicts another word, then you add that back. So it in this is called an auto reggressive mode of generating text. Okay, again I explained this in previous episodes.

[01:26:42]I'll not go into more explanation here.

[01:26:44]But now let's go and run this. So I'm giving it one prompt word Modi and then expecting it to generate the next 20 words. Okay. So it says Modi looks forward to hosting President Trump in New Delhi for quad leader summit ahead of which the leaders will. I think this is a very good prediction built on top of a very small LSTM network. Okay, let's say I want to change this to Trump. But let's copy this and then run it here. Just change it. Save it. Let's say Trump. What does it predict? Huh? Trump and Prime Minister Modi noted the importance of advancing the people to people ties between the two countries. I mean for such a small training sample set and a small network, I think it's doing a pretty darn good job of predicting the right sentences.

[01:27:44]Let's say I do one more thing. Let me create something that's more than I'll give it a longer round. I'll say what did Trump Ram and Modi agree on. Let's see what it says.

[01:28:00]what they India road map for defense industrial cooperation and recogn recognizing the rising importance of autonomous systems and the leaders and on so obviously it's not putting sentence markers like periods and all that stuff because you know the way you do that is you actually train the neural network into showing where sentence ends and all that we are not doing that so that's fine but I think it's again done a very good job so as you can see uh our small LSTM network is doing a pretty good job of um you know completing sentences. Now obviously if you train it on many more articles and you you know give it you know more training time and all that and build a maybe even a larger network you'll start to make more accurate predictions but I think it's pretty impressive even what our small toy LSTM network has done so far. So it's time now to wrap up this episode. Let's do a quick summary of what we have studied so far. A LSTM which is an optimization on top of a recurrent neural network introduces this concept of short-term memory as well as long-term memory. Short-term memory exists in an RNN. An LSDM adds long-term memory to that and it has this concept of a forget gate, an input gate, an output gate and using these two it is able to get across the limitation of an RNN where it does not understand you know longer sequences and LSTM can because of this editive property of the uh long-term memory. All right, with that, thank you very much. And if you like this content, please subscribe, like and share this content with others.