Temporal Fusion Transformers Explained | Time Series Forecasting

[00:00:00]first and foremost a warm welcome this is going to be a fun video we're going to cover the TFT model architecture so take a deep breath close your eyes and Tada here's a big shiny equation for you now I'm going to cover this step uh by step so let me break it down the first one so Y at time I so the predicted value for the if time series at time t for quantile Q at forecast Horizon to and I know that that this is tricky but again predicted value for the quantal because we have this um with Quantas in TFT it's not mandatory but it's about the probability of you know predicting something and then we have forast rizon to as we have shared when it comes to TFT you may have multiple uh time Horizon forecasting then we have the F of Q first the function used to compute the quantal forecast and this function takes several inputs to generate the prediction the first one would be to so the forecast Horizon representing how far into the future we are predicting then we have the Y at um so the historical Target values for the e um time series and this represents the past observations of the target variable then we have Zed unknown inputs also known as past covariates for the E time series from time T minus K to time T and these are inputs that are known only up to the current time step and include variables like past weather conditions past cells of stuff that we know what has happened but we don't know what will happen in the future then we have known inputs also known as future covariates for the E time series from time T minus K to time t+ to and these are inputs that are known in advance and include variables like holidays day of the week scheduled events and then we have static variat for the if time series and these are features that do not change over time and provide context for the model location type of product like anything that static does not change over time therefore when it comes to the forecast we predict for multiple uh time series forecasting Horizon so this will be the to then as well we predict in quanti so we have this confidence interval and there's historical inputs and non inputs and and non inputs and the one that was missing here the static cat and all of this is then bundled uh together now if you thought that this equation was a whammy now here comes a double Wy and PSY according to the dictionary is an event with a powerful and unpleasant effect a blow so here we go this is the actual diagram from the TFT uh model and this is the main goal for this lecture to master this framework of course again if you have any questions do hit me up this is not easy right so this is massive so we're going to take it uh step by step I'm going to try to explain everything one by one and see how it builds on each other and hopefully it will make sense if it doesn't come to me I'm here to help we're going to start by just looking at XT + 1 first and everything starts with a variable selection this is similar to any model like this is your inputs this is how it starts then the variable selection moves to the lstm decoder what does it do the lstm decoder takes in the processed information from previous steps such as past data and known future inputs to generate predictions for future time steps it receives encoded information processes it and predicts future values step by step leveraging its memory to consider both recent and long-term data finally the ad and Norm gate is crucial for maintaining the stability and efficiency of the Network's output it works by combining the output of the lstm decoder with the input from a previous layer through the residuals connections and this helps preserve information and gradients after combining the result is normalized to ensure the values remain consistent and within a manageable range this normalization prevents issues like exploding or Vanishing gradients thereby enhancing the learning efficiency and stability of the neural networker moreover the lstm decoder is excellent at handling sequential data and capturing temporal dependencies but sometimes the direct influence of certain variables is more beneficial if they are not processed sequentially by bypassing the lstm decoder the model can preserve the Integrity of these crucial features and ensure that they have a direct impact on the final prodiction this improves the model's ability to make accurate forecasts by combining the strength of both both sequential processing through the lstm and direct influence through bypassing variables this process is repeated for each subsequent uh time so x t + 1 t + 2 t + 3 Etc so t+ to the variable selection lstm decoder at and Norm gate work together to ensure that the model continuously adapts and makes accurate predictions by leveraging both path and known future inputs moreover please note that we are still talking about future known inputs which is inputs that we know will repeat over time again day of the week holidays Etc now we have been talking about the future but we also need to understand the past and this is where the lstm encoder comes into play the lstm encoder processes past inputs to capture and encode temporal dependencies into to a fixed length context vector and this context Vector summarizes the information from the entire sequence which can then be used for various Downstream tasks while the lstm decoder focuses on generating future predictions the lstm encoder is responsible for creating an informative and compact representation of past sequences and this way the model can effectively use both past and future information to make accurate forecast so in other words the lstm encoder captured past data the variable selection identifies key features and the lstm decoder predicts future values with an ad and Norm gate ensuring stability and efficiency together they form a powerful architecture for time series forecasting at the same time we need to have in mind that we don't just have one Past put but several furthermore if you recall the lstm section um it is not so different so this is where we now move into the next stage so up until now not very different from lstm but now it will be so let's get started the next component in our uh TFT model architecture is the gr or gated residual Network the G RN or k i I don't know how to which way is better I think K is fantastic but grn should be a bit better I guess so the grn is designed to enhance the Network's learning capability by managing the flow of information through gating mechanisms now here is how it works gating mechanisms so controls the flow of information allowing only important signals to pass through while filtering out irrelevant data as well residual connections these connections add the input of a layer to its output helping to preserve important information and gradients this is very similar to the concept used in the ad and Norm gate and finally um normalization this ensures that the outputs remain consistent and within a manageable range stabilizing the training process so by integrating the grn into the TFT model we can ensure that the network efficiently manages the flow of information focusing on the most important features and maintaining stability throughout the learning process now let's talk about the Mast interpretable multi-ad and tension part of the TFT model this component is absolutely key for understanding how the model captures complex dependencies and interprets them for better predictions the mass interpretable multi-ad tension and this is a mouthful this is an advanced mechanism that helps the model focus on different parts of the input data simultaneously it uses multiple heads to look at different parts of the data allowing it to capture various patterns and relationships within the data so this multi-ad is then flagged with the multi arrows here and therefore why does it have three arrows and this is because the arrows H indicate um the multiple heads used in the attention mechanism so each errow represents a different head focusing on a different part of the input data so this multi-ad approach ensures that the model can capture a wide range of dependencies and relationships and this makes the predictions more robust and more accurate but how does it work really right so the attention mechanism assigns different weights to different parts of the input data and highlighting the most relevant information while downplaying the less important parts and it's this selective focus that helps the model make better predictions by concentrating on the most critical pieces of information to sum it up the mass interpal multi-ad attention mechanism so it enhances the mold's ability to capture complex patterns and dependences within the data improving the overall accuracy and interpretability of the predictions this is a crucial part of the TFT model that allows it to effectively process and analyze large sets of Time series data now after this selective attention the processed information flows to the next add and Norm gate this gate combines the attention enhanced dat data with residual connections from previous layers ensuring that crucial information is preserved the combined data is then normalized maintaining consistent and manageable values which stabilizes the training process and enhances learning efficiency however the output of the Gated residual Network gr or grn might bypass the uh Mima so the mass interpretable multi-ad and attention and go directly to the next add the norm gate for several reasons why firstly the grn already defines and refines and as well filters the most important information making additional processing by the Mima unnecessary for these critical signals secondly skipping the Mima reduces computational complexity and speeds up the processing which is beneficial for time sensitive predictions lastly bypassing the Mima can prevent overfitting by avoiding excessive attention based adjustments thereby preserving the Integrity of crucial features and ensuring they directly impact the final prediction next up the grn uh processes the outputs further refining the information the grn ensures that only the most relevant information is passed forward by filtering out the noise and preserving the essential data through its gating mechanism and by doing this the grn maintains the quality and relevance of the data enhancing the model's ability to make accurate predictions and this refined output then moves on to the subsequent ad and Norm gate continuing the cycle of processing and refinement throughout the model now let's talk about the final stage of the DFT model architecture the dense layer and quantal forecast after passing through the multiple layers of the model the processed data reaches the den layer the den layer applies a final transformation of the data think like you know an activation function and integrating all the learn patterns and dependencies and this layer is crucial for generating the final predictions the quantal forecast represent the model's predictions for different Quant providing a range of possible future values this is especially useful for understanding the uncertainty and variability in the predictions by generating forecasts for multiple quantel the TFT model can offer a more comprehensive view of future outcomes helping to make more informed decisions based on our predicted range of values now we need to talk about static covariant encoders which also play a crucial role in the model by encoding stat static features that do not change over time such as location type of product these static features provide important context for the model helping it understand the broader characteristics of the data the static covariant encoders are connected to multiple layers of the TFT model because they provide consistent unchanging information that needs to be accessible at various stages of processing by enriching the model we think in this static context that TFT can make more informed and accurate predictions leveraging both Dynamic inputs and stable longterm features in the chart the temporal Fusion decoder is responsible for synthesizing information from past inputs non future inputs and static Co variat to produce the predictions it does this by using Advanced components like the attention mechanisms that we talked about to ensure that the most relevant information is emphasized and the model learns the best way possible the decoder combines these inputs to generate the final predictions while considering the dependencies and patterns in the data the position wise feed forward layer in the chart applies a series of linear Transformations and activation functions to each position in the input sequence independently this layer processes the dat data after it has been refined by attention mechanisms ensuring that nonlinear relationships and complex patterns are captured by applying these Transformations the model can better understand the interactions between different time steps without altering their temporal order the temporal self attention mechanism shown here in the chart allows the model to focus on different parts of the input sequence simultaneously this mechanism assigns varying weights to different time steps helping the model identify and emphasize the most relevant information again emphasizing the multiple arrows indicate the use of multiple attention heads each focusing on different parts of the sequence which enhances the model's ability to capture the long range dependencies which will hopefully then improve our accuracy then we have the static uh arrangement um if you didn't see it it's over here it's the last part of the temporal fusion um decoder and this involves incorporating the static covariates into the model so these covariates the ones that are constant over time and the static coari encoders process these features and integrate them with a dynamic temporal units and this enriches the model understanding and allowing it to make the decisions and this step ensures that both stable and and variable aspects of the data are considered the chart shows that the output from the first add an arm gate can also bypass the temporal Fusion decoder and go directly to the add an arm gate so all of this here in the end may be bypassed it's a possibility this bypass occurs because the initial processing may have been already good enough so refin enough and all the critical information and has been processed so skipping these additional layers can reduce the computational complexity speed up processing and avoid overfitting by not doing unnecessary adjustments in darts the bypassing of the temporal Fusion uh decoder in by the output from the first add anorm is handled automatically right so this is not something that we need to care about it's a possibility in that Nars does take care of it now let's sum it up we are almost at the end the TFT architecture synthesizes information from past inputs non future inputs and statical varas to produce accurate predictions using Advanced components like attention mechanisms and gated residual networks the process begins with a variable selection followed by the lstm encoder to capture uh past data lstm decoder generating future predictions the adorm gate uh ensures stability and efficiently by combining outputs and normalizing them the Gated residual network uh enhances the learning capability by managing the information flow while the Mima so the mass interpretable multi-ad detention it focuses on different parts of the input data simultaneously to capture the patterns and dependencies the den layer applies a final transformation to generate predictions and the quantal forecast provide a range of possible uh future values enhancing our predictions static variate encoders provide context and integrate consistent information throughout multiple model layers finally the TFT model allows the output from the first add an arm gate to bypass the temporal Fusion decoder and move directly to the last add Anor gate this bypass occurs when the initial processing as sufficiently refin find the critical information reducing computational complexity speeding up the process and preventing overfitting darts handles this bypassing automatically within the model architecture and now o I know it was long but I really hope that this big deep knife would be the way to go there to fully explain this TFT architecture thanks and I'll see you in the next video