Temporal Fusion Transformers For Interpretable Time Series Forecasting

[00:00:01]hello everyone the research paper that we are gonna talk today about is titled as temporal Fusion Transformers for interpretable multi-horizon time series forecasting now these terms for example interpretical multi-horizon Transformers We Will going to decode them in a very short in term of time so yeah let's hang for a moment this uh people was published by a collaboration between the University of Oxford and Google Cloud AI team Brian link for University of Oxford and these are the three guys from Google Cloud AIT collaborated to Target these time series forecasting Problems by attaching the attention mechanism to it and it provide a noble architecture which handled or tackled these time series problems very efficiently now what they did what their technicalities were what were the nobilities in their work was we're gonna see it in the next slides so yeah let's start so when we talk about this Time series casting uh we are referring to the process of predicting these future values or these patterns or between the variables based on their historical behavior in Time series a data points can be recorded in a sequential order but in a regular regular intervals they can be in hourly deity monthly or we can say in a yearly basis our goal is always to analyze the historical data and use it to make the accurate predictions about the future values but we are doing these problems for a very long time from the time we are using machine learning to tackle these issues but there are very crucial you know misunderstandings or you know see we can say uh drawbacks Associated to our approach for example there is a heterogeneous nature of available data sources our input features have a hetogenity between them we always in our traditional Frameworks and in our traditional models we always our input data as a lump it treated our n number features like a scene but our feature were always different our input variables were always different when we are giving a forecasting model or we are making a forecasting model for example uh if a model has let's say a five to six number of input variables then some of the input variables will hold a temporal dependency within it some of the input variables will hold a spatial a kind of you know disturbance or uh variability between it so these capturing these spatial and temporal dependencies with a normal or with a traditional framework was not possible it needed a very unique framework to capture these uh you know dependencies and also there was a interaction between these features these different kind of you know uh the nature between the input variables and the capturing the interaction between these different input variables was not these our traditional machine learning models and uh deep learning models were not able to you know capture these complexities between our input data so we needed a very strong or robust model to capture these dependencies now uh what happened is from very long time these deep neural networks are you know outperforming or machine learning models and uh from many years we have getting a new Frameworks which are outperforming the previous methods also but yeah the one thing we can see very surely uh is despite performing that good the interpretability always remained challenging for these models now what do you mean or what do we mean by we say this term interpretability is uh let's say our model is predicting the weather of uh you know uh of a location so if a model is interpretable we can find out okay if a model is saying that it will be raining today then on what feature or on one bit variables it is saying that this will be the output or this will be the prediction of our model so this interpretability is making our model more transparent it is you know increasing the confidence in our outputs uh it is making our model a more trustworthy more reliable so this interpretability was very important for a very long time because all our deep neural networks our machine learning models the basic criticism they face from these n number of figures was they were like a black box models and whenever our you know state agencies or national agencies are making a system or making a prediction model uh which is gonna affect a large amount of uh you know basically affecting our country on a very large scale economically or agriculturally uh for example let's say we are making the weather prediction model or a model for our Healthcare or our economy so they cannot trust the Black Box model the the model should be very transparent about it outputs and it should be uh transmit about its prediction capabilities and it's a prediction uh kind of analogies so yeah there were some you know techniques like line and shape that we are using from now four to five years now very effectively but these methods are some post hog approaches and they would lead to a very poor explanation and a very poor explainability uh why because uh first line line was a surrogate model and it basically you know both of these uh methods basically ignores the time ordering of the input features and the models which are kind of you know ignoring the time featuring of input features how they can be you know reliable in making our making our our model interpretable so interpretability is kind of a thing which should be inculcated or we must say should be added within our model within the uh we can say uh the methodology or the basic framework of our model so all these kind of issues were handled very effectively in our temporal Fusion Transformers how they do it uh but first we will see how the related work or how the previously done work helped us and kind of you know LED it to let us to find a optimal solution forward problem so let's see the work done in this field until now okay so uh when we talk about the recent works that has been uh done for you know the time handling these time series forecasting problems then all the work can be basically categorized into two categories the first was these I treated approaches and the second one was these uh direct methods now what are these we will be kind of talking them one by one first when we talk about these I treated approaches so they basically utilize these Auto aggressive models uh for example you have been using things like arima or arima based models like things now in these uh Auto regressive models uh predictions are made by recursively feeding the previous predictions into our future inputs so they will uh utilize sometimes you know uh we have heard these lstm networks or dssms which are also known as deep State space models to generate the only one step ahead predictive distributions now this one step ahead predictive distributions are provided here now which are then used to recursively generate these multi-step predictions what does it mean is uh when they are you know using their mechanism they are only producing a one step ahead prediction but these one step ahead predictions are then categorized and then reused to provide us with this multi-step ahead predictions so but the issue has uh been with these methods are uh yeah we can see that these iterative approaches have been effective in some cases but they rely on the assumption that uh the values of all the variables or all the input variables that we are using in our model uh and accept the target variable are known at the future time and I know you know this here is the issue where it is you know affecting our mechanism because this assumption basically restricts their effectiveness in the Practical scenarios because in Practical scenarios there will be numerous useful time varying input inputs are there which exist so uh with many being the known in advance so uh having this assumption basically you know is a drawback with this iterative approaches and we need to see for a good method which will be dealing uh with these time varying inputs with a more effective way then when we talk about these direct methods so in direct methods what happened is we just give a series input and got us input uh put in series also in you know in description we can say it is a Time series forecasting methods that are designed to explicitly generate the forecast for uh multiple predefined Horizons at each time step these methods employs this a sequence to sequence models uh for example we have heard these lstm encoders to summarize the past inputs and generate the future predictions so you know here is the basic distinction between these iot and these direct methods lie upon so one example of this direct methods is this multi-horizon quantile recurrent forecaster uh mq RNN all these methods that I am talking about uh the comparative study of them with our temporal Fusion Transformer uh will be shown in the result section so you should be getting the hang of them so this mq RNN which I am talking about they uses lstn or convolutional encoders to generate these context pictures uh context vectors which are then fed into multi-layer perceptrons MLPs for each forecast Horizon so these directly methods generally outperform these iterative approaches in terms of accuracy because as we know that the basic assumption that these iterative approaches were working on a work housing a serious kind of you know we can say a misunderstanding in terms of handling the inputs so we can see that how these direct methods are advancement over this is our iterative approaches and the method that was discussed in this paper that is this temporal Fusion Transformer hence there is a novelty come in this method because it explicitly considers the diversity of inputs and naturally handles both static coverage and these time varying inputs so it basically incorporates separate encoder and decoderation mechanisms for static feature and setup attention for time wearing inputs how this is being done we will see when we are discussing its architecture in detail so yeah let's move forward and see how its architecture is made and but yeah before uh dwelling into its architecture the first thing is uh from from five to ten years whenever we heard about things or time series forecasting issues we see the only thing that ruled this era was these lstms so why not we are using this uh basic LST mechanism and why we have to you know change our approach to a Transformer based model why we need attention what is so special about Transformers so yeah let's delve into that and find what's so special in this Transformers okay so uh when we talk about these things that why Transformers and what is the need of Transformers to be introduced in this section so the first thing is this architecture of Transformers that we all familiar about because when we uh seen this research paper attention is all you need in 2020 2018 from this Ashish vaswani so yeah now what was so special about his attention mechanism why we need Transformers in this context so as we know the field of deep learning continues to advance this architecture typing most as a game changer in various domains and these architectures were originally introduced for these natural language processing tasks offered unique advantages over traditional approaches so in this context we will explore the key reasons why there is a growing need for Transformer by exact packages uh focusing on their ability to model these long-term dependencies their attention mechanism their parallel processing capabilities and also create a number of benefits we will see them one by one so first uh when we see that uh when the issue comes about modeling these long term dependencies then our traditional recurrent neural networks rnns uh such as lstm or Gru they struggle to capture these long term dependencies due to their sequential nature so whenever we are you know when we have to capture these long term difference dependency one thing was clear that we cannot handle them sequentially we have to opt for a different mechanism which are you know using a better mechanism to capture these dependencies so uh in contrast to that Transformer based architecture excel at mostly modeling these long range dependencies by employing this special self attention mechanism now uh what this celebration mechanism mathematically is I will be telling you in the next slides so don't worry about that self attention basically allows each suggestion in the input sequence to attend to all other positions enabling the model to learn the context and relationship of different elements more effectively this ability to capture long-term dependencies in crucial and various applications including natural lens processing and time series forecasting etcetera and image recognitions may make our Transformers a very good choice for you know these kind of problems now the next thing is why this attention mechanism is a you know good option when it comes to better context learning so one of the fundamental components of our Transformer was this attention mechanism attention allows the model to assign different weights or importance to the various parts of the input sequence so basically we can identify the major basically input variables which are affecting our output majorly so this mechanism not only enhances the model's understanding of the input but also enables it to focus on the relevant information so by leveraging this attention Transformers architectures can learn the intricate patterns and relationship in the data leading to the improved performance and accuracy now the third benefit that we get from our uh you know from our Transformer is this parallel processing now what this parallel processing means is this was the you know uh we can say the advantage of using Transformers because uh when we talk about parallel processing then unlike our sequence models like rnns uh Transformers can process inputs in parallel making them highly efficient this parallel processing capabilities stem from self attention mechanism uh which allows the model to simultaneously attend to all positions in the input sequence so see how it makes things move fast and more effective as a result a Transformers can process the long sequences in parallel reducing training and inference times significantly disadvantage is particularly valuable in applications that involve large data sets for example in our issues when we are you know dealing with this time series forecasting problems and which involves the real time processing and we are speed is very crucial we cannot waste any time there so you know in that cases these our Transformers were very very effective now third thing was or you know the fourth thing was which are effective was this transfer learning now another remarkable aspect of over this architecture for the transfer learning wow because they are often pretend on massive data sets such as imagenet or last text copula or you know uh and they learn the journal patterns and representations now this pre-training allows this model to capture the rich feature from diverse data sources these retained Transformers can then be fine-tuned on specific tasks with comparatively smaller data sets and we can use these data sets in any other location or we can adapt them for any other you know variable inputs uh which have the similar nature Ally cards and this transfer learning approach saves computation time and data requirements enabling the model to perform when even with limited labeled data and then then yeah we also know how there is a benefit of fast training in friends and processing because due to their parallel processing nature and efficient attention mechanism Transformer offers these fast training inference and processing uh their ability to process data in parallel and capture long-term dependencies more effectively leads to quicker convergence during the training so which make them move faster and also also they can effectively process the large amount of data during inference uh making them suitable for our real-time applications and large scale deployments now it is totally clear that by Transformers were adopted uh basically how Pi Transformer based approach was targeted while we were solving this time series forecasting problems so yeah let's see the next aspect of our research paper uh okay so uh as from now on or from you know from very long time we are discussing that how this time profusion Transformers are providing our providing us with this multi-horizon forecasting sorry so uh when we talk about multiversion forecasting we have also a issue of diverse inputs so what happen is you know in multi Horizon forecasting involves predicting variables of interest at multiple future time steps uh offering users access to uh estimate across the entire forecasting horizontal uh this ability to look ahead and optimize actions at multiple steps in the future is incredibly valuable and think of retailers optimizing their inventory for an entire upcoming season or uh you know clinicians optimizing treatment plans for their patients so it has a long you know we can see application part so the application of multi Horizon forecasting at vast and impactful to ranging from retail to health fair to economics now what are the problems with this the first is our input variables are having very diversity in them for example the first thing is the first thing is the our input variables can be categorized into two categories the first are these static ones and the second are time varying for example there will be some input variables which are uh you know remaining constant throughout the time series they are not changing within the stations and there will be some inputs which are varying bedtime now the inputs which are varying with time can also be categorized into two another subparts for example one which will be unknown in future now future is the time interval we are considering uh for which we are providing a forecast that is uh time interval After Time T and there will be some time varying input features which are uh know in future also for example which will be known after the time T also so how this thing is taken care in case of our temporal Fusion Transformers so yeah first we will be knowing more about uh what about this diverse inputs and what is the importance of these diverse inputs so in Practical what happened is when when we talk about multi horizontal forecasting application now multi Horizon means we are providing you the outputs at a you know 2 at multiple time steps ahead and we are while we are dealing with multiple variables so providing with the distribution of outputs at uh you know a multi horizon mean a multiple time Step Ahead uh distributions uh outputs is more crucial so uh when we are doing these things we often encounter you know diverse data sources uh each providing unique information about the Future these sources may include known events uh for example uh upcoming holiday dates uh other exogenous time series such as historical customer food traffic and static metadata like the location of a store however the challenges lies in the fact that we have little knowledge about how these data sources interact with each other the heterogeneity of these inputs combined with the lack of information on these interactions make multi horizontal time forecasting particularly very challenging so you know multivision forecasting seems to be very kind of you say promising in future but doing it is a very challenging task as compared to you know image processing etc etc so why uh there is importance of diverse inputs so despite the challenges uh incorporating these diverse inputs is essential for us because to achieve the accurate and reliable multi-horization forecast these inputs provide the valuable context and information about the factors influencing the variables we are trying to predict by leveraging these diverse inputs we can uncover we can say uh the hidden patterns or identify correlations and capture the complex dynamics of the forecasting problem at head uh this holistic approach allows us to make the better informed decisions and optimize our actions across the entire forecasting Horizon so uh here we can see the how multi Horizon forecasting is a better way to deal with this time series forecasting issues because to tackle the challenges posed by these diverse inputs uh researchers have developed Advanced Techniques in models from which these temporary Fusion Transformers are also you know among one of them so it was a crucial we can say problem while we are dealing with this the time series forecasting issues but it allows us to optimize our actions at multiple future steps and the presence of diverse inputs including non-future events Excellence time series and static metadata poses unique challenges however by embracing these diverse inputs and developing Advanced models we can uncover we can K and cover valuable insights and perform make a better model which can now you know deal with these time series forecasting issues more effectively now how this model is composed basically what is architecture is saying to us or what's so important and novel about it uh we will talk about it in the next slide so let's move ahead okay so when we come to temporal Fusion Transformers we have some major nobilities like gating mechanism the variable selection networks static covariate encoders uh our temporal processing and our prediction intervals I will talk to you know or we'll talk to about one of them about them one by one but first let's see the architecture of our Temple Fusion Transformer so what happen is as I uh talk about the diverse inputs our temporal Fusion Transformers you know deals with it very implicitly or you know we can say uh very uh categorized and category categorically for example uh you can see static metadata inputs which were known up to the time T only and the inputs which are known up to uh the future times ahead also these three types of inputs are feeder to our Network in different ways for example you can see these color changes here this color changes says that they are encoded differently so you can see these LST encoders have also shown with a different color in these uh three different subparts so it means to providing them with different encoding or to the different parts of our architecture uh basically provides or give the information to a model that how these inputs are differentiated in time because uh these inputs are not the same they have different characteristics and they have very you know you can say very uh different nature so that they can that can be inculcated into our model or Incorporated in our model so that it can predict in a better way now what's another better things in our we can say our TFT is the first thing is the jeting mechanism so when we talk about the gating mechanism uh our model incorporates the this mechanism that provides the adaptability and the flexibility to the models architecture uh this mechanism allows us to uh dynamically skip over unused components and adjust the depth and the complexity of the network based on the specific data set and scenario basically it tells about how much of input should be given the non-linearity should be incorporated into that or not and uh this adaptive nature ensures that uh TFT can effectively handle a wide range of Time series forecasting tasks accommodating both simple and complex patterns in the data now next thing was this a variable selection Networks now when we talk about this variable selection now what this variable selection do is it does is it uh basically plays a crucial role in selecting the relevant input variable each time step by dynamically choosing the most informative features TFT focuses its attention on the inputs that have the greatest impact on the target variable these feature selection capabilities enhances the model's accuracy and efficiency and it avoids these unnecessary computation or irrelevant variables discard them and optimize the use of our computational resources so that was a major you know we can say the nobilities are kind of you know uh we can see the efficiency or efficient way of TFT to handling these diverse inputs third thing is our static covariate encoders so to incorporate the static feature into the forecasting process of a model utilizes these static covariate encoders these encoders encode the static information into context vectors which are then integrated into the network by conditioning the temporal Dynamics on these context vectors our model captures the influence of static operates on the time series data and this integration allows the model to exploit the valuable insights provided by the static features and enhances its forecasting accuracy so especially in scenarios where the static factors plays a significant role in those forecasting problems now when we talk about the static obedient encoders the next thing comes is what is this temporal processing so temporal processing basically tells that our model is basically you know better in learning both long term and short-term temporal relationship from observed and known time varying inputs for uh local pricing our model employs a sequence to sequence layer uh which captures our immediate dependencies within the time series uh additionally it has a noble interpretable multi and attention block to capture these long-term dependencies that you can see in the figure and this attention mechanism allows the model to effectively capture these temporal patterns and dependencies that span across multiples time set at time steps so by considering both local and long term temporal relationships TFT achieves a more comprehensive understanding of the data resulting in accurate and reliable forecast and now what is the next thing that was Noble to it that was the prediction intervals as I told you when we talked about the prediction intervals our uh temporary Fusion Transformers are basically providing our us the quantile forecasts that are that is uh they are basically you know in nature they are providing a distribution of our outputs they are not giving us a deterministic output but basically they are providing us with a distribution at different quantiles for example 10 15 and 19th uh you know quantiles they're providing us with the distributions so that we can get a better idea of how our model is working okay now this prediction intervals basically uh are provided through this quantile forecasts which offer the valuable information about the uncertainty associated with the predictions because of prediction intervals indicate the range within which the target variable is likely to fall at each prediction horizontal this measure of uncertainty allows the decision makers to assess the level of confidence in the forecast and make informed choices based on the associated risk the inclusion of the prediction intervals enhances the interpretability of the forecast and you can frequency that it enable users to incorporate the risk management strategies in their decision making process so it basically tells that however TFT mechanism or architecture was so Noble and what was the benefits or you know the major constituents of its mechanism uh that make it so novel so yeah now next let's move to you know the advantages of advantages of tfts that it provides us while dealing with time series forecasting problems so yeah in short if we have to basically you know summarize the temporary Fusion Transformers benefits to handle these uh time series problems uh we can come up with a number of benefits but let's summarize into them five to six topics so yeah the first thing is handling this diversity of inputs so uh our model handles the diverse inputs including both static covariates and time varying data very effectively and it incorporates its static covariate encoders to efficient basically effectively uh integrate the static features uh into the forecasting process this enables our model to leverage additional information that can have a significant impact on the predictions by considering both static and time varying inputs our model can capture a comprehensive view of the underlying patterns and Dynamics in the data now the next thing is this intreputable attention mechanism so when we talk about this interpretable attention mechanism how it works I will be you know telling that very deeply in the next slides but basically it utilizes an mechanism which allows users to gain insights and to how the model makes predictions uh the attention mechanism identify the important variables and components within the input data that contribute the significantly to the forecast this uh interpretability aspect is valuable as it provides transparency and helps users understand the underlying factors driving the predictions so uh by examining that anything patterns users can gain insights into the temporal Dynamics and the relationships between different variables so it is one of the you know selling point of this upper TFT mechanism next things what is identifying the important variables for you know prediction so yeah our model basically captures or identify the important variables for predictions by leveraging the variable selection networks and we can say that it dynamically selects the input variables at each time step this feature ensures that our model focuses on the most influential factors while disregarding our irrelevant inputs so you know when we are dealing or basically what in short we can say that uh consider encountering this time series forecasting issues we have n number of inputs that are varying for a very long time we have huge data sets so basically identifying the important variables and which are you know affecting our predictions are which is a crucial step because without that data engineering or feature engineering you cannot be you know giving you if you are not giving your model the information that is more reliable or even more important to provide the prediction then if you're not feeding your model in a better way then you cannot expect miracles from it you have to have give uh it's uh you know the input in a much understanding way because your model is like a machine and uh we use the you are giving input to it so that input is a more understandable way you have to basically liquidize it in a manner that you model can you know fade from that information and give you more reliable outputs so yeah above model has that capability but still it depends upon the user's capability to feed the model more effectively now the next thing is this capturing the persistent temporal patterns now TFT is designed to capture this both long and shortened patterns present in our time series data so is temporal processing capabilities our model can effectively learn the dependencies and the relationship between past and future observations so the interaction part that we talk about in the you know input parts we are coming into that it is handling that interactions very effectively due to this attention mechanism that is has been added into its working so what happen is it allows the model to capture these a persistent temporal patterns such as a seasonality or a lag effects which are crucial for accurate forecasting so by accounting for these patterns our model enhances its forecasting accuracy and provides us with more reliable predictions so yeah it is one of the major kind of frequency the benefits or the major we can say the important features and the Novelties that our TFT brings into our focus and yeah the other major contribution or we can say uh better thing was this gating mechanism for our irrelevant inputs our model enables it to adaptively skip over this irrelevant input and this feature enhances the model's ability to you know handle these noisy or unnecessary variables that may not contribute significantly to the forecast so by negating the impacts of these inputs of a model improves efficiency of the forecasting process and focuses his attention on the most informative components of the data so yeah so it's totally includes that why our TFT architecture is you know basically handling all the issues or all the drawbacks of our approach and our architectures that we have used in the recent times and also uh the drawbacks that we talk about our approach it is handling all of them very effectively now what is this interpretable attention and all these things we'll be talking them in the next slides so yeah let's go to that now so okay before diving into our I know interpretability and how it has been our attention mechanism have been made interpretable I have to you know I wanted to give you a slight idea of how a model is working with the inputs and how in a very concise way it's handling those inputs uh into it so yeah as you can see here upper uh y cap QT tau is equal to uh function fq uh which taking as inputs what Tau uh y ID 2 K time T then z i t 2 the K time T then x i t 2 the K times t plus Tau and s i now what these variables mean one by one let's go into that our y cap QT tau is nothing but it is about the quantile forecast that we are providing at all time steps ahead from time t and the queues refer to the keyword sample that we are providing the operation for so the Y cap is basically the prediction that we are providing from our model uh fq is basically the prediction model or the function which is taking all these inputs all these inputs and providing this with the Y cap so fq is basically our model or we can say the relationship that we have formulated from our model and uh as I told you tau is nothing but it is the number of time steps ahead uh from which uh from time T that we are providing the forecast for and why iit2 to KT is the target information within the finite look back window K this look back uh window K is basically the encoding length the time period we are giving of a model to C into behind the time so it can have this temporal attention into it or basically it can capture this temporal dependencies uh of the data is the targets or basically you know the known outputs up to time t Okay up to time T now what is more important is let's change the color so yeah this zit and x i t these both are other input variables okay these both are the input variables but what's important about them is the Z inputs that we are talking that we are talking about these are unknown in future that's why they have only that they have only you know known from the look back period of K to the time T only but when we are talking about this x these are the inputs which are known in the future also that's why they have been given a time interval from a look back period K to the t plus Tau so it tells about how it is handling these diverse inputs how this input is giving given to a model and yeah in the end SI is nothing but it is the static variables that are not changing the time as I told you in the previous slides however input variables were differentiated into the starting and time varying so yeah SI is telling us about the static variables from that slide here so yeah it basically summarizes that how a model is taking all these inputs okay and uh providing uh making uh basically incorporating a relationship over them and providing us with the quantile forecast at the Q example with a Time tab ahead that we need information uh you know up to so yeah that's all about our diverse inputs and the quantile forecast of our TFT model now what is the interpretability that I'm talking about or Time Bomb so yeah let's switch into that so yeah it is the most important aspect of this paper because the you know the very important thing or very novel thing they did was uh when we talk about Transformers the thing we know about is a multi-handed attention we all know about it but a multihead attention is and how we use it why it is so significant okay but the issue with multiple attention is it was not interpretable so when we are getting results out of multi-head attention uh that we do with our Transformer Transformer Transformer paper that's attention all you need we discussed it uh in that paper uh that was in 2018 I think so yeah that multi but I had multiple head attention is very good while it is capturing these dependencies of our data in the previous time steps or we can say the previous inputs it was very effective but the only issue was it was not so interpretable that much of interpretability it doesn't have that once we get the outputs we can backtrack which kind of very cables are affecting these uh you know our predictions or our outputs to know that or to make that attention mechanism useful and also interpretable our temporary Fusion Transformer mechanism basically make it interpretable how there does it let's see that so yeah interpretability was incorporated by a simple mathematics that it is because uh as we see from our interpretability uh yeah so uh our attention mechanism was nothing but it was you know uh as we know in our tension mechanism there are these three matrices that query key and value mattresses uh we get the dot product from query key values uh these matrices and then we multiplied with the uh relevant of these value Matrix and what this thing we get this is known our other attention head we just uh form uh you know a normalizing function which we generally use a soft Max into that uh this is for the scaling thing and this is basically a attention head that we get now uh in the basic Transformer architecture what we did was uh generally we took these attentions from multiple heads and concatenated them as we've seen that H1 H2 H3 H1 H2 H3 are nothing but just they are the we can see the pair of the query key and the value matrices now the different pair of query key and value matrices tells us about the uh different multiple aheads okay different uh different heads so when these heads uh provides us with the attention matrices and when these attention matrices are basically you know concatenated we call it laser multi ahead multi-head attention now this multi-added attention was used in Upper uh basic paper of Transformers that we talked about but as we know the issue with that was this concatenation that we are doing here was not that much of interpretable so to make it interpretable what they did was this whole term concatenation was replaced with h Delta Now versus what this H tilder was this Hilder was nothing but it is uh simply what they did in this uh you know formulation was they take a shared value Matrix okay the value Matrix was taken as a shared value so it uh and these attention matrices that is the uh you know the dot product of the query key values matrices we um despite of doing concatenation of them we did what we did was uh we did the uh additive averaging of them okay the by doing additive averaging these uh value matrices this value Matrix that we're talking about we have taken it uh you know the shared value so it will automatically come out and this additive aggregation of these attention matrices uh when it's done uh not you know concatenated and they have done in an additive aggregation of them this whole thing was more you can say interpretable as compared to this concatenation now why it is happening so you can think it off in a sense that uh let's see um we have three features uh so yeah that's what we're doing we have only two features both of them have very high attention value when we are you know comparing them with our output but just because one of the features have very you know a low value Matrix score its attention value is getting discarded so this attention value should not be kind of you know ignored or uh but you should say uh should not be getting get in vain because our attention mechanism is working so hard and we are incorporating that into our model so to resume or to keep that uh basically attention mechanism alive and basically backtrack it once we get the output we do this additive aggregation of our attention heads and not basically concatenating them so in to summarize all data basically what they did was they get this attention weights okay and then in spite of providing to the different value matrices for different features they use the shared value Matrix so these shared value Matrix comes uh outside of this additive aggregation and when we in place of providing them with the of you know concatenation when we did this additive aggregation of our whole attention mechanism was more interpretable as compared to the standard Transformer Network so yeah uh this interpretability uh into a retention mechanism was the key point and was you know more effective to identify if our model is working fine how it is giving our outputs is these outputs are reliable or not so yeah it was the major input or we can say it was the major novelty of these people that they make this attention interpretable now yeah next we will see that how this model was trained and how this validation and the comparison scores with our model source so yeah let's uh move to that now so yeah a while uh we talked about uh you know evaluating the performance of both architecture so uh to evaluate this architecture we need different type of data sets so that we know that uh what kind of uh you know the outputs of our model is giving for different kind of inputs which contain different type of variables and different type of output nature so the type of data sets used were these four the first was an electricity data set which were containing the data of you know us 30 370 us households and in this basically model what we did was uh we take the look back period that we talked in the architecture time the look back period was a week that is about 168 hours and we forecast for we give the forecast of our model for the next 24 hours uh then the second model data set that we used uh was a traffic data set which basically tells us about the occupancy owner of our road in the US and it also Aggregates on an hourly level uh as per the electricity data set with the same look back window and the forecast Horizon so these were the first two data sets then the third one was a retail so it was taken from a kaggle competition data set of a favorite uh basically uh grocery sales and they forecast these lower product sales 30 days into the future using the 90 days of the past information so you can see the variability in different setup and different different data sets is there now the fourth one was these uh volatility it was you know the most difficult of these data sets because the first two were the simpler ones and these two coffee very difficult ones you know uh when we talk about volatility it contains the daily realized volatility values of 31 stock indexes 31 stock indices and they were computed from the intraday data along with their daily returns so basically in this case what they considered was they consider the forecast over the next week uh that will be the next five business days uh using the information over the past year so the Look Back Time was the past one year and we were giving the forecast for the next five days so in this case the look back period was about 252 days and the forecast time that we are giving was the five days now what's special about these four data sets was the first two data sets that is electricity and traffic they were the universe data sets so what happened was in these data sets there was only one variable in the input or maybe you know information regarding that one variable uh should be kind of you know distributed into three to four variables and then one output variable so they were univarity in nature but these last two data sets they were kind of multivariated nature you can have you know their sources and you can see how DVD and kind of you know how they have been incorporated or kind of you know passed up or basically stacked up into a model uh in the real paper technical provider description in our video so yeah the last two data sets they were you were more complicated because they were not Universe they were multivariate in nature also the focus time and the look back period was different for them so um using these four different of data sets basically you know give us the overview of how our model is performing and how our model is comparable to other techniques that we use about okay so yeah let's see that how of a model uh basically basically performed on these four different kind of data sets and was it comparable to the known methods we have right now or not so yeah let's move to the performance over model on these uh different data sets so uh first of all we have to know that how the training was done on these different data sets and what is the procedure that we followed during the Training Method so yeah uh the most basic one there's our data sets were divided into three parts that is training uh validation and testing one so tiny data was used for you know give our model the sense of data that how it is to train its uh parameters then validation data set was used for the tuning of our model tuning for hyper parameters and then testing was to know that how a model is performing on this data set so yeah then uh first our model was trained so the thing was the of a model that we are basically providing us or basically we are we can say offering to our uh you know research domain then this model should not be that much a computational demanding that it uh basically you know outresource the researchers and we cannot lose it so to basically take care of that also uh when this testing was done this was done on a single GPU in it but for example when we did this paper and we know the researchers and did this uh you know handle the data sets with a single GPU they give a damn you know timeline of the data like that if they use this Nvidia Tesla uh V100 GPU then the uh optimal time over TFT model T8 for the electricity data set we talked earlier it takes slightly over six hours to train and that is very less when we talk about trading Transformers uh while the each effect being roughly around 52 minutes so yeah it was learning more about the data it's uh you know the temporal dependencies and everything so yeah it was a very good time to train and each Epoch was around 52 minutes that was good for a Transformation Network then uh you know after training there was also an issue of this during training there was an issue of this getting better hyper parameters because hyper parameter tuning is also a good thing or you know a required thing to do when we have a large model so but researchers did was they uh take basically a large domain of these State size dropout rate uh mini bet size and then also they give the learning rate sizes or you know all the possibilities that they can have and so that they can get a fine-tuned parameters to give as I input over to our input to our models also the number of heads uh I'm seeing here is ranging from one to four so uh one to four is the multi head attention that we talked about in the previous slide they also checked that uh how much how many number of heads would be more suitable for our uh you know optimal training of our model so yeah that's all about how the training was done and how training procedure was done uh on our different data sets uh for over this temporal Fusion Transformers let's go to the results parts now uh okay so uh when we come to the results section the thing we uh basically notices was uh first we will be seeing the electricity and the traffic data sets because as I have told you these two data sets with the simpler ones and our univariate in nature also so the first table you are seeing here the we are basically comparing the p50 losses on these different models that we have seen till now and discussed uh in the previous slides and you can refer to the original paper also and you know kind of learn about them in a you know sequential manner somewhere else but the comparison here clearly shows us that the losses for example these uh kind of you know regressive models like arima our TFT based architecture had an advantage of 180 percent a lower loss as compared to the remodels in the electricity data set basically and after that there were other these you know very popular models like uh deep ad dssm that we talked about then the sequence to sequence and mqrnn also we had a significant you know uh what we can say is advantages uh when we compare the losses the 50 losses however temporal Fusion Transformers to all these models so here we can see how a temporal Fusion Transformers performed very way better in training and after the interest um then the at the testing section they outperformed all these models how it was so significantly uh basically you know outperformed these uh previously known mechanisms to us uh then in the next slide sorry next not in the next Slide the next uh table when we compare the P90 losses uh we can also see that in this case also uh a remob was outperformed uh then ATS then this uh dpar there are all these uh deep AR and other models were totally outlown and performed by our temporal Fusion Transformers so it basically shows that all the predictions or all the things that we promised earlier that temporal season Transformers with this Noble approach of architecture and this better approach of capturing the uh you know the dependencies between the data uh were not in you know this is a fake promises they basically outperformed the you know the known parameters to us and uh known architectures to us so yeah uh it was quite promising and uh it basically we can say it uh uh yeah it gave the results that we expected out of it now let's see the other two data sets and how they performed on it so yeah uh in this case when we talk about this volume and a retail data sets uh which we which I told you earlier but the more complex data sets with a more number of variables they were multivariate in nature uh also their look back period was different their uh Focus time was different so they were you know uh given more complex kind of a data set to see that however this proposed architecture will be performed under that so you can see the results also that our temporal Fusion Transformers were very close but improved as compared to this mq rnns and also and as compared to the popular choices like deep air they had a good we can say this this was a slight or a very good Improvement when we come to these data sets like volume and Retail and these p50 losses in case of p50 losses the Improvement was more but in case of periodialysis it was slightly comparable that how they were performing so the basically you know the basic takeaway from these results and comparisons is that yeah the TFT uh basically capturing these temporal dependencies of input variables handling these diverse inputs uh it's not only promising them in vain but also delivers us with a very good results so yeah uh in summing up summing it all up we can see this architecture who was a noble approach and it is outperformed the available uh kind of architectures we had till date so yeah it was a quite a great approach and it delivered on inverted promises it will be doing okay now let's see what he had next the next thing was the results of interpretability that we were talking about from so long so in these tables the interpretive interpretability of our static variables and uh our past inputs that were known up to the time T only uh are given and all the values with uh weights of more than 0.1 are been highlighted so you can see clearly that in case of static variables uh what is a particular stored number and it is a specific item these things were given the more weightage so it makes sense because the location of a store and what is uh its location uh where it has been working from so long so these things basically affect the sales of a particular store also and when we talk about these past inputs so yeah you can see here how these national holidays and in which month we are trading also in which month we are selling products they had a very high weightage and very high sales over you know kind of found in those kind of variables or in those kind of scenarios so here we can see the interpretability results for making totally sense of what results we were getting in our forecast or in our uh what we say is uh in our quantile Focus so yeah interpretability basically uh given us uh the kind of you know the overview of how our result section is working and how our model is predicted a prediction predicting the outputs uh is it doing well or not so from these values we totally found that yeah it's um getting accustomed and kind of interpreting the uh these upper sales data set very easily now coming to the different kind of other kind of inputs so the inputs uh which were the future inputs uh the future inputs uh basically refers to the inputs which were known uh from the time t to the t plus stock that I told you earlier in the slides so in the future inputs you can also see that uh when there was uh on promotion the things were there and the particular days of the month there were some national holidays and then there are also some local holidays or how all these things affected our forecast of the model so yeah this interpretability basically you know uh tells is more uh you know the kind of transparent picture of how our model is predicting the future so in conclusion uh we can say that our temporal Fusion Transformers provide us with a powerful and versatile solution for time series forecasting by incorporating these diverse inputs uh interpretable attention mechanisms uh capturing persistent temporal patterns and utilizing gating mechanisms a TFT delivers accurate and reliable predictions across a wide range of application and next we will be seeing all the you know references that have been used in this paper so so yeah these are all the differences that were used in this paper and it will continue also yeah and you can uh find the you know a link to the original paper invitation description of my video also uh and find out more about this paper from so yeah in end I will be saying thank you for watching this video and I hope you found this exploration of temporal Fusion Transformers insightful and valuable uh stay tuned for more exciting topics in the world of machine learning and data science thank you