Graph-Based Long-Term Memory for Agentic Workflows

[00:00:23]Hello everybody. Good morning, good evening, good afternoon, and it's almost a little sad now, but this is the final notes road to nodes AI workshop we're hosting today. Um, graph-based long-term memory, how agentic workflow adapts through experiences. What what you also can see on the screen already. I'm super happy to have um Alessandro Negro uh do the workshop with us. Aleandro is the chief scientist at Graphaware. Hi, Alessandro. How are you doing? All good.

[00:00:54]What about you?

[00:00:56]>> Yeah, good. No, I'm excited. I'm happy um about these workshops. They've been very very popular. We had lots of lots of great engagements uh great excited engagement audiences throughout all these u these sessions um all of March still March for one more day. So we uh we kept that uh in in the loop and uh yeah in two weeks time almost exactly on the day right I think it's exactly on the day. I know it's it's Wednesday, so it's two weeks time still tomorrow. Um I know the AI is taking place and we'll do the full the full event. But um I'm super happy to do the road to notes as as always as a little warm-up series with all of you to get a little bit more hands-on experience on nodes uh graphs relationships and AI. Um specifically about memory for agents. I think this is a super hot topic. I heard a lot about this. People were talking about this. I was at CubeCon in Amsterdam last week and know lots of people were talking to us and say hey I heard about this memory thing how does it work context graph memory agents what's what's going on so I think this is very very timely um that that you are covering this today Alessandra gives us give us a bit of an insight in in how how things are are working and uh what from your perspective is a is a good a good practice and um for all of you the session is recorded so we will share a video link with you afterwards so you watch it in any time if something was too quick or if if you um had to skip over something. Um we will also share some links that Aleandro has prepared for us throughout the session but also in the in the email afterwards so you can look at everything again later on.

[00:02:36]And if you have any questions throughout the the the workshop today, Alexandra told me there will be some some break points. So we have some time uh to cover a little bit of questions. So if you have any please use the Q&A function.

[00:02:47]You should see that at the top right over the chat where it says um you know chat messages and Q&A. Click on Q&A, add your question there and then we'll see it pop up in the Q&A segment and then we can uh we can look at questions and answer them. Uh but obviously also uh write in chat I see people from Mexico and France who did dare to write where they're from. I think the other chat there's another chat. Please use the session chat. Bunch of people from United States. Welcome all. Um, great to see you. Um, and yeah, I think with that I can re return to the backstage and uh let Alexandra have the stage. Um, have a good good time everybody and I'll be I'll be back in for for questions.

[00:03:33]>> Cool. Thank you, Alexander. So, uh, good evening, afternoon or morning depending on where or when you are watching this workshop. So today um you will learn how to empower your agent workflow with graph-based long-term memory and evolving the playbooks. By introducing these mechanisms, you will get your AI to learn continuously while you are using it without fine tune finetuning required or at least not at the beginning. This is a going to be a mixed workshop uh with a lot of theory but also a lot of practical examples demo code and architectural uh desend details. So um in the next days as Alexander said I will also release uh the source code uh which is a GitHub repository. Consider that it's not supposed to be a real project. So I didn't spend much time on readmi or project setup but you will find all the prompts and the lang graph configuration that you will see during this uh presentation. So no need to to take screenshot because it will be available um immediately after this uh workshop.

[00:04:47]So moreover considering that uh this is quite practical and it is um a two hours workshop we have all the time. So feel free to interrupt me and ask questions if you have any. Uh there will be some moments in which I will stop on purpose and I will ask you if you have some questions but really feel free to interrupt me whenever. But uh before we uh dive in, let me uh quickly introduce myself so that you understand why I'm here in front of you. As Alexander said, um I'm the Graphawware chief scientist and at Graphawware uh we spent uh more than 13 years helping customers adopting graph technologies and in particular Neforj for mission critical uh tasks spanning from recommendation to uh data analysis to advanced search to natural language processing and in the last six plus years actually we also developed our own graph powered intelligence analytics platform called Yume that is built entirely on top of Neo forj uh that is specialized in law enforcement domain. So some of the screenshots and the demo that you will see today um come from from Yum.

[00:06:00]But uh what I'm going to present you today uh can be reproducible completely by using the code that I will give you and Neo forj of course. I'm also the author of uh two published mining books that are graph machine learning and knowledge graphs and LLM in action that are let's say um in the middle or that are spanning between graphs and machine learning natural language processing and uh so what does it mean that really I spent more than half of my career working at the intersection between graph machine learning and natural language processing in particular the last years um including LLM in the graph space. So you can imagine how much excited or glad I am to present here at Nodes AI and uh if this is not enough I also have two hours actually u that is like the paradise for for an Italian like me. So in order to help you uh to follow the content uh of the workshop, I like to give a a full picture um first including uh the problems the type of problems that we would like to to solve and the high level idea um of how we are going to solve it and then we will go in the details and I hope that this will help you to follow in a better way. So generative AI uh have become incredibly powerful as you have seen in the last years. Um the agentic approach has become a common pattern. You know these autonomous agents um combine multiple tools, reasoning steps, actions and knowledge sources to get users what they need. We've gone from uh the early stages with impressive but somehow unrelable uh results to ones that can to systems that can access specific domain knowledge through um through our knowledge graphs or rugg approaches. So the initial problems we had like illusion hallucination for example or lack of domain expertises have been tackled or heavily mitigated by these technologies and these approaches and agents played a key role in that. So now these agents can give you accurate grounded answers by pulling from structured or unstructured uh knowledge including a lot of uh amount of code they can site sources that they can give you they can let's say can navigate a complex domains and this is absolutely a huge progress but there's a critical piece missing here I don't know if you remember Dory from funding Nemo. So, she was helpful, enthusiastic, ready to assist actually, but she had no short-term memory. Every few minutes, she'd forget where she was going and uh what she'd learned. They This situation describes pretty well the main problem with vanilla agentic AI uh systems. They can access uh knowledge and tools, sure, but they can't learn from experience in in their original format, not at least without going through uh complex and expensive training processes. They can't remember what worked last time, what failed or adapt based on users feedback.

[00:09:34]They reset with every new uh conversations.

[00:09:38]They make a mistake, well, they will make it again tomorrow. Give them feedback. it's gone uh in the next section session and every iteration starts from zero. So to be truly uh intelligent and supportive then AI uh needs two things. One is the knowledge foundation and we've built a you know rug or graph rag and knowledge graphs um to solve these issues. So we consider this as sorted out but also we need the ability of this system to grow to adapt to retain memory on long-term from experience and use this experience to get better next time and this is uh what we are going to discuss in this workshop and uh here is the bonus or the you know the stake because um for mission critical uh decisions that is the u let's say domain which we work This learning should be transparent and uh explainable. Analysts needs to know um why AI is suggesting a particular approach. What experience led it there.

[00:10:48]So here's the question. What if AI agent can could learn um exactly like uh just human humans do. So that's what we are going to solve uh today. and uh trying to solve it uh we discovered that this is not an easy problem. Well actually it's quite complex. So because uh while we were solving it we faced two major challenges. So the first one is that we can't just use a full past experience as another ra data source. Sure we can store every interaction, every output, every piece of feedback. But if we treat it as a another knowledge source to retrieve from, we will be back in the uh from where we started again in the rug causity.

[00:11:42]So we needed to convert experience into something distilled and useful like patterns, strategies, lesson learned. So we need not like this is what happened 147 times but here here is what we learned from these uh 147 experience. The second challenge is that even though we store only patterns lesson learned etc. So we store smart we can't dump all of these into LLM every time that we ask the question. Imagine if every time that you had to solve a math problem, your memory will brought back historical information about Roman invasion or how to prepare the best picana ever. You know, definitely you'd lose focus completely. With LLM, we face the same problem at worse because we are limited by the context window we can use you. So this context is growing. I teach model iteration. here that okay the number of tokens you can send grew quite significantly absolutely that but this will never be uh infinite moreover even though this will happen by chance you know um passing all this information will cause LLM to lose focus and so um if we bring this relevant memory this would bury the signal in noise so we need a selective memory the right information at the right moment for the right for the right question.

[00:13:20]So, how do we solve this? Well, let me spoil a bit uh what we are going to discuss today. This is a very high level um architecture or vision of what we are going to see. So, what we are doing literally is to uh combine uh or bringing together uh two of the hottest breakthrough research areas in the AI agents right now. and uh not only of course in the agent but in general in AI and also we extended them a bit and these are the context graph and the agentic context engineering or ACE if you prefer the short version. So let me describe them first then we will see how we combine them. So first context graph this comes from a very recent research on capturing the complete story of the decision making at the early stages uh or this is generally referred to human processes you know what decision is taken why what were the other options and why they were discarded for example the more context the better because we can um get more information while we are taking the Next decision applied to aentic. The idea is elegant because it traps every decision your agentic system uh makes along with the context in which it happened, the options that were available, the reasoning around that the and most importantly the outcome including human feedback. So uh it's not just about uh what did agents do it's also what what else sorry why what else um can be done and how did it turn out so the complete picture as I said with its provenence um and this is what enables both real learning and extendability and thanks to graph databases um that extend our storage capab abilities and even offers a flexible way of storing all this information. We can literally store everything even connections among the different steps, derations etc. And this is not only useful of course as we will see for growing our knowledge but also it will it's also useful for traceability reference debugging and of course explanability as we will see. The second uh important research area that we are going to merge here is this concept of agentic context engineering. I'm I'm sure that you have seen many articles and papers on this topic including the one that I mentioned here uh from where the charts are taken. Also anthropic the cloud AI the cloudi uh let's say company you know has an interesting blog post about it. So it's the evolution of uh the prompt engineering you know because instead of having a static prompt um that is manually or automatically edited um this context is passed as an evolving a dynamic piece of content that could help uh the the system to perform better uh next time. So ACE creates living playbooks as we will see of experience that evolve as our understanding deepen as a agentic understanding deepen and these playbooks are passed as guidance to the prompt to improve the result at each iteration as as I said and uh as you can notice from these uh charts actually we've got thanks to this uh a 10% improvement in many different uh scenarios actually. So and they uh achieved this by preventing two critical problems. You know one is the brevity bias. So where summaries lose crucial uh details and the second one is the context collapse where knowledge erodess over uh over time. So coming back to our high level schema we see uh in this picture our virtuous loop. So in the context graph we track agents decisions and human feedback and we use this source for crafting and evolving playbooks that are used to provide guidance to agents so that they can do better next time. So, context duff gives us a comprehensive comprehensive auditable um experience tracking perfect for mission critical decisions as we said because we can keep track of everything. We can um know exactly why did the system do this and the living playbook that is evolving not extending all the time gives us a created actionable guidance that actually get better over time without growing infinitely. So because they are constantly um refined not extended and this allows us to really keep it as something manageable that can be passed or as we will see can be let's say use as a source to extract a proper guidance the next time and these uh two technologies actually um can solve um what researchers called the biggest bottleneck in the agentic systems right now. the memory that learns that is exactly what we what we need. But um let me show now how it works in practice.

[00:18:52]But so in order to do so uh we we need a real problem to to solve because um uh this will help not only to make the examples that I'm going to show you more concrete but also it will justify the complexity of the solution as we will see because as you will notice but as you can guess you know the the final solution will be pretty complex. So we need to tackle a complex problem so that all these uh effort will make sense. So the value delivered to our users or to our company should worth you know the should be worth the time that will be spent and the effort that we will um provide by doing this. So uh in particular um I asked myself you know what makes a good test case for me you know so I found out that there are four critical things to make it you know reasonable. So first of all, we need um a problem that requires multi-step reasoning. It's not something that you can just ask um a question and get the answer immediately. Second, it needs domain knowledge that goes beyond what generally LLMs uh know because of the pre-training process. Third, it should benefit learning and feedback of course because this is the core of what we are going to see. It should interpret human feedback and use this to get better next time. So we really need to learn from it. And of course since you likely to approach this problem in a proper way, we need um a clear metric to or clear metrics in general to know if we are actually improving. So we need also a proper data set. So by reading a few papers and and blog post a blog post I found what I needed in terms of problem I I found out that this financial document analysis or financial document intelligence has the right level of complexity because it's a real problem uh with proper uh let's say uh sides uh to make it complex um enough for our test. Um, of course it's critical because we we want to address mission critical problems and definitely this is the case because the wrong answers or wrong decisions can generate a bad decisions in terms of you know uh finance decision and so we can lose actually money or people using it can can lose money and uh this problem should learn should um get domain knowledge from the interaction with the users or from the answer and the um and question actually at each iteration and this is of course really relevant. So luckily I found also a good data set that is called finance bench that I had to extend a bit to make it even more complicated but it was definitely a good starting point. Just to give you again um an example of how much complicated is this problem. I also tried several um for several questions inside the finance bench a few uh well-known uh cloud providers of these LLMs you know and of course my favorite one and it is always very good let me say just that in this case I'm really raising the bar quite a lot is cloi so I asked a few questions and actually it was pretty accurate but not 100% accurate as we will see in this screenshot I first asked the question and then I pass the real answer and it noticed that there is an issue in one of the documents in one of the years in particular and this is happening because it struggled to find the full document and to go through the full document because it was too big um but of course it can do better with the rug etc but this is exactly what we are going to do so as I said just now I mean the problem of answering proper complex Financial question can be solved by using a rag you know um and in particular um you know financial rug is exactly what we are trying to to address here. So the problem is is clear and uh but it's complex even with uh with this uh hint because u there are a few aspects that we need to consider and here we have a few um example of the questions again to understand why these questions are complicated for example the first one is a 3M capital intensive business based on the fiscal year 2022 data so this is complex because the the word capital intensive it's something that is close to that is definitely not available in um let's say in the in any of the documents that we have and uh so it make um text search or uh even the the vector search quite complex you know so the first test that I run I noticed that there was no way for me to retrieve the right documents for this specific question so I struggled a lot to find a solution and then uh the other issue Um from the second question is that for such a large set of years let's say range of years from 2019 from to 2022 actually we need four different documents and each document can contain easily more than 65 or 70 pages. So we are talking about a sides that will go way above the grabability in terms of window. So we have these basic issues but we have also more let's say complex issues for example even though we will find the the word capital intensive you know what's the threshold for considering it if the M is capital intensive or not so actually what happens in particular for this question is that some people can consider um capital intensive the capex over revenue and the threshold could be 50%. Other might use a capex over assets with a threshold of 25%. So this is a really first of all domain specific but also it is a user specific. So different users can have literally different needs and so if we come out with a with some models maybe that this won't work for another use case. So actually it needs to follow the requirement or the the feedback of the user while the user is using is using it. So it's not only about the domain expertise but it's also about the user uh preferences and as I mentioned you know there will be multiple documents in this in this sense. So let's try again to explain better why for example a simple rug or a oneshot rug will fail for who doesn't know how the vanilla rug works here is a simple image you know here we have the question starting um you know the the process then this question is converted in a in a vector the same embedding function that has been used to um vectorize the the chunks of the documents we So by using a similarity that can be cosine or whatever you can get the most similar chunks related to your question and and then you can pass these to the answer generation and uh and then you can take all these and pass to LLM and at that point you will get the um you know the answer from LLM. So basically what happened um in this case is that as I mentioned with the capital intensive uh question you won't find that much or you will find wrong documents probably so you won't be able to answer and this is exactly what happened to my um during my uh test of course just to be clear there will be way more complex implementation some using a graph rag that is super powerful if you like to answer very complex question but still what I'm challenging here is the concept of oneshot so that you decide up front which documents to retrieve and then you pass them to the answer generation phase. So there are techniques to make it better but still the problem here is not the retrieval phase but the fact that it happens before the answer the question and it's one shot. So um you can't go back and uh you know improve the result because you don't have the the right content here to answer the the question. So and and even the brute force won't work because passing everything as we said it's impossible.

[00:27:46]So in the question that we've seen before passing four um 10k document is absolutely a nogo. None of the current um models will support such a large um token size or token window you know. So static retrieval strategies can handle the dynamic reasoning task. So that's why this is a perfect problem for using an agentic approach and uh in this case since it is applied to to rug this is called agentic uh rug. So basically um we com flip completely the way in which it works because the paradigm so instead of retrieving uh everything up front it's retrieve think decide what's next or think retrieve decide I mean it can be uh organized I mean structured in a in different way so in a gentic rag the AI becomes an active participant in The retrieval process it's it decides what tool to use. For example, if it's vector search for conceptual query or keyword search for exact terms, structured query for more complex but organized data, document navigation to hope between sections etc. It makes this decision during the reasoning process not up front. So here's the power. It can adapt. uh the agent retrieves a document evalate it or many documents actually and think okay is this set useful um for computing for example capital intensiveness if not it can come back and retrieve other type of information so if uh the vector search return nothing because of the capital intensive issue then it will try with a keyword approach but more interestingly doesn't have only the the question you know it can ask for specific type of information inside the uh the documents. So this gives us a focused context. We only pull what we need when it's needed. So no brute force, no drawing in irrelevant documents and crucially we can use a different tools for different type of questions or for different subtasks. So the right tool for the right job. But how can we orchestrate this is the real question because agent is a generic concept then we needed to implement it in a real um let's say architecture.

[00:30:22]So aent as I said is a generic concept and then we will see how we are implementing it. Let me clarify that this is just one of the many possible uh implementation.

[00:30:35]Maybe some of you already used the identity rug in the past and they've seen a different version of it and it will be perfectly uh fine. Remember that the scope here is to have a complex problem that we would like to solve with an agentic AI system. So the problem has been identified that is the financial document intelligence. Now we need to identify the agentic architecture to solve it. We identify the rug as the problem or let's say a sort of solution and in particular agentic rug that is exactly what we would like to use as a as an example but what we are going to discuss later in this presentation can be applied to any other type of agentic um configuration for different type of of scopes definitely you know so just to clarify where we are here so we are to using this aentic rag implementation as a as a base but it can be applied to whatever you it. So let me explain this implementation in particular. Again the starting point and the end point is exactly the same as before. So we have the question entering the system and we would like to get the answer. So the problem let's say outcome doesn't change but what happens in the middle is the measure issue. So here what I did was to divide the problem in two major let's say subtasks. On one side we have the retrieval phase and on the other side we have the answering phase. So the purpose of the ritual phase as you can guess is to get the right set of documents to answer the question. But this is not done by a single element. It's done by these three elements. Actually let me say that this is the explosion of what is commonly called react um agent. But uh instead of using a simple React agent where I have literally no control over, I prefer to split it so that I can keep track of every single decision that I can implement it as I uh as I wish. And in particular in the React agent, I have only one prompt that I can use for the entire retrieval process. Here I can have as we will see two different prompts. One for the retrieval strategy and one for the evalation. But let's see how it works. So the retrieval strategy is the element that literally decide what is the the next strategy to take um the right information out of the documents. Then once the tools are identified with the parameters etc. there are the actual calls to these uh through these tools and the documents are retrieved and this can happen in parallel. So actually there to strategy node can decided to run multiple calls in in parallel. Then there is another let's say element that actually does two things on one side evaluate the quality of the retrieval. So it take it takes the question the result documents and evalate okay are these documents these chants enough to answer the question and then it if this is the case produces um a so-called let's say growing knowledge.

[00:33:38]So it extract from these documents what is necessary to answer this question. If at this point the extracted knowledge is enough then it will move to the next phase that we will discuss in a second.

[00:33:49]But if this is not enough, you see there is this feedback loop that will come back to the retrieval strategy with the feedback to say okay this is what we have gone we what you gave me this is what is missing please focus on what is missing and and extracted these are from the tools uh from I mean via the tools or from the documents and this is exactly what the retrieval strateg will do in the next iteration we'll actually take this into account and make a different call to extract only the missing pieces so as I said If everything will be fine after one or multiple iterations, we moved to the answer generation. Let me say and stress once more that here we are not passing all the documents because we grew up this knowledge that is all the answer generation node needs to answer the question. So there is no documents passed anymore. At this point, the answer generation will answer the question and at this point we have um let's say LLM as a judge whose role is to evaluate the answer and again take in this case not true but three type of decisions. The simplest one is okay the answer is fine the computation is fine let's move to the end and provide the answer to the user. The second option is that there are some computational I mean error errors but still the data provided by the retrieval phase is correct is enough so let's go back to the answer providing some um feedback to the answer generation but there could be that the answer evalation will come back and say there are some missing items in the uh generated uh knowledge please retrieve other information and again pass the some retrieval feedback to the uh retrieval strategy and this will again extract the missing information and the iteration will continue. So that's how it works at um higher level. So um as you can imagine that the process is complex you know of course as we will see the accuracy will be enormously higher than the oneshot rug. But of course we moved from one single LLM call to actually four different calls if everything will go well. Otherwise this can be literally uh you know a very long number of iterations you know and actually this means a lot of LLM calls to the um you know that can generate more or higher costs for the for the company. But you know the the return is that the answer are um answers are accurate. So this is not um let's say um a talk around rug. Uh I spent already a lot of time discussing how it works. But again I needed you to understand what kind of aentic system we are putting together. Here in the next three slides you can find um some key difference between the oneshot and the aentic rug that justifies you know why you will need an agentic rug or when you will need an agentic rug. What I would like to stress on because this will be useful later is that the pre-processing requirement is totally different because in the um oneshot drug you have to spend a bit more time on the let's say pre-processing because for example in the case of graph rag you need to decompose the text in the entities in the uh relationships and so on so forth or you to you summarize the content of some documents or create hierarchies etc. In the case of a gentic rug, as you will see, this will be way simpler. So, because we moved the complexity to the retrieval part, not to do the uh pre-processing part. And of course, the complexity as you can guess is completely different because the uh in complex aentic rug you have a very complex structure as as you have seen.

[00:37:44]But the latency of course will be higher and on the way back of course the answer quality will be higher. error recovery is in place while in a classical oneshot drug is no recovery or at least not in the standard way and it can adapt things that one shot cannot cannot do and of course in our system in particular we will see how the full trace will help to to keep track of every single uh decision here. So the problem of keeping track of what's going on in the in the agents or in a conversation is definitely um not new uh in the sense that this is a very well recognized problem. So there are out there a few other solutions that might be you have seen uh before. I would like to mention these two because in my opinion there are the more production ready and actually they have a lot of interesting use cases. So it's important to mention them to compare um what I'm going to propose you to what is available in the in the market you know as other other options. One is ME zero.

[00:38:54]So M0ero is the most production ready um let's say memory system available. Its focus um is on uh keeping track of conversations um and its core insights is that you don't need the full conversation but you need a compressed to I mean the duplicated set of important facts. So it has uh this mechanism through which after each step or after each iteration you know you get some um manipulation some merging of the of the content and this is important because you will see something similar even in um in my approach you know so you have that at the end of the process you will get some addition some update some deletion or some I mean skip or some no operation um you know once you it received a new message of course in terms of latency It is one of the best tool because it reduces a lot of the latency and actually it has this 7K concept which means that it is able to keep the the full uh let's say memory around 7K tokens. Um and uh the problem that we have here compared with what we are going to see is that uh no answer um let's say quality will come after or thanks to feedback loop you know. So the memory updates just based on conversation content and unless you implement it there is no natural way to get feedback from from the user or at least not so immediately. Then the other uh platform or tool that I would like to mention is Zap or Zap. the um the reason is because it is in my opinion the the most u let's say ambitious uh memory system available and it's also interesting because of course we are at nodes AI and it uses a heavily temporal knowledge graph uh as a structure of the memory you know of course it needs let's say richer structure because um you know it stores entities relationships and entire uh timeline. So in these three tiers which is a bit you know complex and um is quite a huge overhead. Uh so the benchmark um provides um strong let's say results but this must be justified by uh the uh the reason why you are doing it because the overhead is higher so the latency compared with me zero is higher but of course um there are scenarios in which you don't care about latency as we will see even in our case but you care about accuracy and so definitely zap is one options that you should should take into account but again here there is not that much related to feedback loop etc etc so I will stop here uh for a second because we reached the point um where I gave you the the full picture of what problems we are going to solve what is the uh aentic uh let's say system that we are um let's say addressing to um to prove that what comes next um can improve the quality of this of this system and and now we also have a better understanding of how critical are the the agents in the um generative AI field. Um and I think that it's clear how long-term memory can help these AI to to learn. What is missing actually is how we are going to to implement this. So we have our multi-gen system. So we can start by discussing about the uh the details of how to implement it. What are the different components that we need to to implement?

[00:42:47]I barely um discussed about the pre-processing when I was comparing raes one shot raic rug. As I said already, the pre-processing is pretty sim simple.

[00:42:59]Um, and in this case, what I'm using in the system that you will see live is a very say simple preprocessing step.

[00:43:10]Simple because differently than what happens as I mentioned with graph rag.

[00:43:14]I'm not extracting entities. I'm not extracting relationship. I'm using only one summary, not a summary etc etc. So there are all these simplification and actually I'm using only one call to LLM per document which is definitely way lower than what you will get if you're using graph rag. So the processor is described in this picture. So we have of course the first OCR because the documents are all in PDF but this is up to you. I'm using Dockling uh on my computer, but um we also tested Azure um the old form recognizer now called Azure document intelligence that allow you to extract what um you know you need from the text actually. So the content even quite well. Then on top of these we apply some semantic chunking. So instead of using just a fixed site chunking as we will see in the demo in a few slides, what we do is to organize the content so that the the content per se is self-contained and we have also a description of these of this content because this is easier to read and to understand and actually the the let's say the content helps us to to find what we need in a better way and definitely this is proven in our test to more effective on the on the long run but also for understanding the the final results. What we also tested was to in particular for tables or for images to pair this previous OCR step with some um large language models or let's say multimodel language models post-processing that literally take the result of the OCR and semantic chunking take also the picture literally of the image and extract or fix the content you know because it can happen that of course basic OCR cannot extract relevant information for images or in particular for tables they cannot be structured in a in a proper way. So um these post processor can help a lot to get better results fixing a lot of issues during the uh the process. At that point we have um a process that involves LLM uh that extract because the postprocessing is optional because before I said that I have only one this is true because in most of the cases we are not doing the postprocessing. If you have this of course it will be um that you have two calls to LLM but in general we are not using that much for our use cases but you could need it. In our case, as I said, we are using LM for um extracting titles, summaries and tags. Um actually, we are passing the first couple of pages just to get for our um 10K or quarterly documents the title because we don't have it in the PDF metadata. um the summary so that we will be uh able to you know better add some header to each chunk and some tags to improve the extraction. Then we embed the chunks um and the summaries etc. And we um literally index these uh as a vector but also as a as a text. And in this case, Neo forj helps us quite a lot because with the same structure, we can with the same data structure, we can store the embeddings and and the text. In terms of uh let's say structure of the graph database, this is a very minimalist. Um we have the the documents that have pages and these pages have chunks. We also shortcut a bit the documents with chunks and we included also some aggregation in this process because we notice that with semantic chunking it can happen that you will get a very small um chunks. So what we did was to store both small chunks and big chunks you know and uh we let vector search or text search to find what is what is better or more accurate during the uh the retrieval phase. So this why we have the chunks including other chunks and of course as I mentioned we have also the the tags for um you know f retrieval even though to be honest in this at this stage we are not using it at all. So in terms of tools even here we were very uh basic I I would say we have and you will see uh soon three basic tools. One is as you can guess vector search. So we use similarity in particular cosine for semantic search. Um we use a nej vector indexes as I mentioned before and this is extremely uh fast. Then we can apply some filters on on documents or on other aspects but u you know this is optional and this is a decision that the retrieval strategy agent will uh will take. We have the second uh tool that is a full text search tool that actually use these lucine based queries again provided by Neforj um through the full text search indexes for extracting specific terms. I would like to spend a few seconds here because in my opinion you know using the retrieval strategy um say node would with this full text search actually we don't need real entities extracted because as I said you know we simplified a lot the pre-processing thanks to the agents and actually one of the reason is because of that because as you will see if you you know in a few examples but also in the if you run it is that the strategy agent will be able to decide which entity to use if it is a company, if it is a specific term, if it is a person, etc., etc. So, it will recognize that this is an entity and will search for this entity in the text without having a specific node containing it and with the understanding that it will get about lucine query actually it will be able to get exactly what it is even though it's not stored as a as a single node in the in the graph. That's why this simplifies a lot of the the prep-processing phase.

[00:49:38]And then I noticed that at some point the system was struggling to find what it needs. So I I said okay we should mimic what financial advisor analysts do. They literally read through the documents. So there is this tool that is the page search tool that literally get the document name and the page actually the list of pages and actually it goes through these pages looking for information. So when there are I mean the two before failed actually it has this fallback that literally goes through pages of these documents to find exactly what they need. But of course it is never the first one because it needs at least to know what is the document um file you know the the name of the file to get through this and of course these are just three tools available among many others that could be because we can have more complex structure. You can definitely use a graph rag if you like if you like here and then you will have an agentic graph rag in which you can use the graph structure to get a better queries or you can have also your article summaries you can do whatever you want but here the point is have a simple implementation. So it's time now for a quick uh demo actually. So let me um switch to this tab.

[00:51:00]Okay. Okay. Okay. So, um, >> do we want to answer a couple of questions before we do a demo?

[00:51:09]>> Yes, absolutely.

[00:51:12]because we had a couple um coming in um and um I'll I probably won't be have able have the time for all of them but maybe we can we can start with Joey asked pretty much in the beginning um how would you distinguish between short and long-term memory if you you know duplicate the long-term memory and the short-term memory or do you dynamically link it maybe you show this in the demo as well but that's probably a a good good starter to to distinguish what what is what And how how does what become long or short-term memory?

[00:51:46]>> Okay, that's a definitely a good question. H actually the the reason why I called it a long-term memory and I even didn't mention short-term memory is because uh I I think that we refer to short-term memory to what ME zero or Zep are doing. you know, they keep track of what happened in the uh let's say in the specific conversation, you know. So, to me, um these tools are focusing on a short-term memory because they keep track of the um of what's going on in the conversation. Of course, they store it and this automatically will become long-term memory. But here um you know what I meant for long-term memory instead is that actually we are keeping track of everything is happening in the system as we will see forever literally.

[00:52:36]So how you will see um in the few next slides actually what we are doing is to keep track of every single decision and this will be tracked forever and from this memory we are extracting from this long-term memory we are extracting our um let's say playbooks or our evolving playbooks. So that's the the distinction in my system specifically I'm keeping track of every single um decision which you know if we look at it as a a single question answer is short term but the fact that I'm using it you know and you can use it literally by condensing them by aggregating them for example uh it's immediately becoming long-term memory because it's not connected to a specific session so answer and question and actually I'm not using it during the the session But I'm using after the session to extract knowledge. That's why I distinguished between these two um terms.

[00:53:33]>> Okay, makes sense. Um Aisha asks when you talked about the the agentic loop and with the LLM as a jud judge segment how the feedback implicit or explicit um is is is that stored in the graph to refine future actions? um and how how how that makes um you know a step practice maybe.

[00:53:56]>> Yes, I mean uh absolutely uh these will be visible in the examples that um we will show later you know once we will go through the the concrete examples of how this loop is closed with um you know inside the graph and also inside the the prompts that we are um you know using to answer this question. So it will be visible in a few minutes.

[00:54:16]>> Okay. Super. And then uh Jay asks um because you talked about that you had the best results with Zap um if you have any benchmark data used or how how do I mean was this just personal experience or how did you decide or had you come to the conclusion?

[00:54:32]>> Well I mean just to be clear I didn't say that I I had the best result with Zap. what I said is that I like it the most because uh as I mentioned at the beginning I'm a great fan of graphs and knowledge graphs you know and um and so the basic idea is that zap I mean is entirely based on the concept of the temporal knowledge graph so that's why I like it you know I didn't test yet on these infrastructure but this is something that I have in the in the road map to compare these results with what I will get by using the zap and Men zero and I will publish something in the in the next months about around this but again I didn't test it I just went through how it works and I checked you know how this could solve my problem and then I decided to go my own way and I implemented it but really zap is in my mind what's the best solution you know because I love um knowledge graphs and it is based on knowledge graphs >> cool yeah and then always um you know make your own tests try try something what works with one use case or one specific um you know stack doesn't work with another or is maybe not as as well performing with another. So even if we show something here you know take your own uh you know time and uh and try something play something explore different options and tools and and see what works for you how you feel you know results wise but also you know what's more fun to work with what's what's easier to to do and and to build and um and and you know just just go with that.

[00:56:05]Totally agree. Yes, this is absolutely true. You know, and that's why I would like to make a comparison for this specific um problem and I'm sure that there will be tons of other problems in which men zero, for example, will be the best solution and others in which ZAP will be the best solution and a few others in which what I'm presenting today could be instead the the optimal solution for what you need. So, absolutely test everything and use the right tool for the right um problems, you know, for the right job.

[00:56:36]Yeah. And then maybe maybe one more question. Mark Ori asks, "Do you also cons or did you also consider more traditional tools for pre-processing like spacey?" I mean they would probably not uh he says probably not they would not be able to to summarize but but still is that is that still an option or are they outperformed by by all the other um more AIdriven things that are currently available? Well yeah I mean uh let let's say that u um if you go through my first book uh all the natural language processing part was done by using spacy and actually at graph we have used it for several scenarios many years ago I have to say that the reason why I don't see it useful in generic terms is because it struggles when we come to specific domains you know as I said we operate in the law enforcement space but we also operated in healthcare for example and um we struggled a lot not to recognize people for example where spacy can excel but when we started saying okay we would like to recognize the specific um you know terms or specific um how do you say relationships unfortunately all these tools failed in that sense LLMs can do better you know because you can easily um let's say tweak the prompt to extract what you need so in this sense I would always go for LLN unless of course unless the the data you have is so um secretive you know so you have to protect the privacy of people that you can't use public lms in that case of course I will go for spacy but again honestly um it works really in specific cases we also had in yum way many years ago actually a tool for annotating documents to customize py to extract what you need but it was so tedious. So personally I would be far from that. I tested we we also promoted in a historical part of our life and uh actually we decided not to go for that because it's too complex you know when you come to something that is different than people organization location.

[00:58:51]>> Yeah. Yeah. Makes sense. Uh, all right.

[00:58:55]Uh, maybe we do one more because I think it's also interesting from Orlando. How quickly do you do we expect storage needs to grow with continuous memory ingestions? Do you think that that could become a problem um with with this uh pipeline?

[00:59:11]>> Well, um, another very good question.

[00:59:13]So, let me say that now um I'm storing everything. So, every iteration I'm I'm storing uh all the details. But let's say that the way which it works and you will see in a few minutes is that once I you know extracted the uh you know the track of what happened to the uh question answering phase and also the entire process I have a step that will literally extract the evolving knowledge or update the the playbook according to the to what happened in that specific session. in that specific moment in theory you can completely remove the uh the trace. You don't need it anymore because you extracted already what you need and so you can keep the playbook and the playbook as you will see is very very small. So it's very very limited.

[01:00:01]Of course, at this stage of the project, I'm even not thinking about removing it because I need to keep track of every single decision to verify if I'm literally improving thanks to the guidance. And we will see uh as a part of this demo how much information I can extract thanks to to this um history track completely. But of course in production I agree, you know, you can't keep everything. But for that you can decide to go for okay obsolions. So after I don't know one week you can remove um all the uh question older than that or you can keep this by daily and so you can remove maybe relevant question or you can decide a different policy. Okay if it's an interesting question for whatever reason because done by an interesting user or because it generated a longer path than usual then you can retain it and remove the rest. actually you can put a um you know a cleanup strategy or a teddy up strategy after and this won't affect what you are going to see today.

[01:01:04]>> Okay, >> cool. Um I won't keep you longer from seeing it actually in in action. So um yeah, let's let's let's continue with that and um we'll have we'll have some time for a couple more questions later on, I'm sure.

[01:01:18]>> Sure. Yeah. So um actually it was good to have a moment to answer some questions so that we can keep in any case this part uh I mean shorter because this is just about a few details of how the um you know the the documents are stored in the uh in the graph. So let me show you this um screen. So here um I can uh you know perform some um search uh to to show you um for example how these documents are are stored. Um and then I will show you an example of so as I said that this is the document that has chunk that and the sorry that has page that has chunks and then we have the um the bigger chunks know then we can easily filter for example for some document like starts with 3M okay and then we can create a preview then we can expand so that you can see the structure. So here we have the the documents where we have document I mean document summary we have the the source file we have the uh you know the tags actually and we have I mean just basic information like the number of pages then we have the pages and then we have the the chunks let me show you the chunks here you can't see the the embeddings because they are filtered uh in the view but they are there I would like to show you an interesting thing here Because here we have for each chunk two text one is the text as it is extracted from the uh let's say the semantic chunking. So this is exactly what is extracted and we have also an extended text because what I noticed at some point was that some of the chunks didn't contain for example the name of the company or the the year of reference of the document. So actually I u since I extracted the title and the summary I have a small portion of it that is a an header of the real text. So the real text is what you see here you know in the text this is what has been extracted as a result of the chunking process. But in order to improve the quality actually I had to put this error up front and this dramatically improved the quality.

[01:03:42]This is just a trick that I wanted to share. Also uh what I didn't mention is that the embedding has a the embedding function or the the model has a great impact on the final quality um of this uh process. You know in this case I'm using a BGM3 and I'm using it through a lama just to save some cost on embeddings. Uh but you know I tested many others and the quality literally changed the final results according to the type of embedding that I'm I'm using. Another thing that I would like to show you quickly is uh is this one.

[01:04:18]So uh let me show you this uh this document if I can get it. Okay. U this document is an example of 3M fillings for the uh 2018.

[01:04:32]Yes, because it is u yeah 2018.

[01:04:36]So actually you see that these are documents with a complex structure and very interestingly they have a lot of tables to to process. I have to say that document did an amazing job. Um Azure document intelligence did even a better job but they are I'd say comparable.

[01:04:54]What I would like to show you is how tables can be stored in chunks uh to make a better retrieval. So you see here you you have a table then you can decide to store it as a um let's say um meta table so actually with the same structure of the columns etc using um you know the common structure but you can actually store it in the way that I'm going to to show you. So let me get some some chunks for three um company 2018.

[01:05:30]So if I go to page 11 or eight material you see that the title is always useful of okay exe executive officers and you see that instead of storing the table you know um as a as a table actually it is stored in this way where we store facts you know so this tooling guy has age 65 this tooling guy is has the current position executive chairman of the board So the I mean dockling was able to process this table and actually extract the information in a way that is better for retrieval for example and this again improved a lot the the quality but again I don't want to discuss too much about the prep-processing because definitely not part of this but if you like to replicate these um these results you need to consider also these aspect otherwise you will get worse results and say I some did something weird and uh make the result fake But in reality is because I spent significant time to make it better and better. So that's where we are. Um so at this point um we have our um multi- uh step agentic rug that we described and this is exactly um what we would like to um to improve and uh as some of you ask in order to to get the full picture and see where the um you know the the feedback comes into play where the guidance comes into play we um we need to see a bit of the of the code actually. So, let me show you just a a bit of code to show you where you will be able to find the uh the prompts.

[01:07:19]Here we go. So, um actually this is the the project uh you know that I will share after this uh this event. The entry point for me is this evolate rug implementation that actually runs through the um this bench benchmark and run all the all the tests.

[01:07:39]Um and it can use a different type of um configurations. So changing the rug type you can switch from aentic ra to classic rug and other configurations. Um and uh and then you can also switch between different embedding models. As I mentioned, I'm using a BGM3.

[01:07:59]I'm also using different tests, different databases. Actually, as you can see now, while we are speaking, I'm still running the last tests as ablation because I would like to be sure that the quality is improving. Um, but what I would like to show you at this stage is the agentic part. So, this uh is um the the description of the the workflow. As you can see, I'm using land graph and uh I'm sorry just for the names because I didn't have the time to match the the slides name with the name of the agent node here. But uh you know I will fix it before giving you the the final version.

[01:08:42]So here we have the retriever agent. Of course we are logging the question but this is just a log. So we have the retriever strategent the retriever that is the the element of making the calls to the tools. Then we have these great documents that is what you've seen as a retrieval evalation and growing knowledge. Then we have the question answering and the reflection. And then of course we log the the final answer as independent node. Then of course we have this uh let's say description of the of the graph structure where we have you know all the edges plus a couple of conditional edges that are the uh you know the the squares that you have seen there where we decide where to go if we have to go back to the strategy agent or we are good to continue to the questioning answering or like in this case we have the three options that I mentioned before.

[01:09:31]ending up and providing the answer to the user or go back to the retrieval strategy or go back to the answer and generate a new answer. So this is a easy to describe in lang graph that's why I love it because what you have um you know on the schema you can easily represent in in langraph. So if you are curious about a few of the of the prompts, I can show you the retrieval agent prompt. Here you have the prompt.

[01:09:59]So you are a financial research analyst blah blah blah. Your task is to retrieve uh to use the retriever tools available to extract the most relevant part. So here there are critical rules something that I had to fix because of some misunderstanding of the tool calls. Um and here are some suggestions on how to use it properly to avoid to go through the same step again and again. So here I keep track of the iteration the question here we have the guidance block that we will see soon and here we have the feedback that can come from the um you know from the step inside the retrieval phase or from the um evolution at the end of the of the process. I have of course the growing knowledge the list of tools available. I'm also passing the list of previous attempts so that I mean the the prompt and LLM know what to avoid because it didn't work or it worked but this is already tested. So we avoided to do the same things again and again. And what is really relevant for this is that I'm also taking keeping track of the pinking strategy. So I'm explicitly as you can see here um you know asking to always provide thinking with this structure because this is what I need later for taking lesson learned and actually we will see a few examples in this direction and again if this was not enough I'm also repeating again you always must provide thinking and this way I was able to provide to get the thinking all the all the time. Um if I go down I have the retrieval to call. Of course it just make the calls and then we you can define here the uh hold on the prompt for the um for the system uh that evaluate the documents retrieved and the increase the growing knowledge. So again here we have the description of what is going to happen the document sorry the question the material documents the growing knowledge because this is iterative so we need to keep track of what has been extracted already because just to be clear we are not passing again and again the same documents. So these documents are the only one retrieved by the previous step the previous node not all the previous iterations and etc etc. So we have the current strategy thinking.

[01:12:23]So we also pass what we've got from the previous strategy agent node so that this element this node has the full picture of what happened and what not and of course the guidance that we will see in a while and then a clear guidance on the explanation. The answer is the same you know structure we ask to answer the question using the knowledge and these um evolution is a llm as a judge in which we provide some specific let me show this in particular um the examples oh it is here actually so here we have the you know the the answering node because this is a different phase so you will find in another in another file okay I don't want to spend too much time here because you will get access to these documents at some point. So, let me share my screen again with the workshop. There we go. Okay. So, so far so good. So, at this point uh we can start discussing how to store all these uh information and actually this is um the the schema of our context graph.

[01:13:35]Again this is exactly um matching I mean this is matching exactly what we have in the um architecture. So you can imagine easily that we have the question that is our entry point somehow. Then we have the two phases actually the retrieval phase and the answering phase. Then we model the the iterations. So we have multiple um retrieval phase iterations and we have the let's say the answering phases all of them have these AI step of course it is of different type but I created just a simple one to to have a single model a single sorry node type and then we have the answer feedback because I need to track the the feedback provided by the um let's say by the user and interestingly We have the opportunity also to come back to the uh to the chunk because we are storing in the same graph database both the data that we needed to answer uh you know the the documents that we just processed and on the other side we also have the let's say the the trace of the execution and we can connect them easily because they are literally in the same graph database and again this is a a great power for for us to analyze really how things are going you know in the in the system. So this structure matches perfectly what we just described and here you you can see a few a few example of the thinking that I was just describing. In this case the the retrieval evalation and knowledge builder said no. So this is not a success with this reason because some documents have not been found. This is a uh the question is exactly what you have seen you know the 3M capital intensivity um and it explain exactly what is what is missing and provides suggestion on what to do next. Here we have the thinking again because even the retrieval provided some thinking about why it says no what were the options and what are the recommendations for the next steps here you can find them and then you can um see the the growing knowledge that is exactly what has been extracted and you will see a few examples in this uh in this sense um so now I I'd like to show you that uh some concrete uh examples of um you how we are storing this data inside the the graph. So let's go for some good example like the 3M since we mentioned it too many times. So let me zoom it a bit. So here we have you know that as I mentioned we have the question with all the details. So here we have you know the uh the UID that uniquely identified that specific instance of the question.

[01:16:30]Alandro Alandro, sorry, we still see.

[01:16:33]Ah, now it comes. Okay, here it is.

[01:16:34]>> No, no, sorry, sorry. I forgot to switch to this one. Um, so um sorry, I um was showing this. I mean this is a real result actually because we I asked the question is a 3M capital intensive um business based on of course financial fiscal year 2022 and this is the uh actual execution that you know generated this graph in which we have the ritual phase and the answering phase and we have that the ritual phase required two get the answer And then we needed just one um uh iteration of the answering phase. Then we have the answer and we have the answering in this case since this is green it means that it completed successfully. So the feedback was positive. Um I was just showing here I mean minor things but I can show you the the full trace to understand how it works. So here we have the question this is the answer provided. So the answer is clearly no. 3M does not appear to be a C intensive for this and that reason with all the computation that happened and interestingly we have a direct trace back to the source. So this is literally the ID of the chunk from where this information has been extracted. This is the chunk of the uh from where this other information has been extracted and so on so forth. So in this way of course for mission critical tasks this is mandatory. So if we you know say something we need to trace it back to the source of this of this information and this is the case for whatever is reported here. And again the the fact that we are splitting the retrieval phase with the growing knowledge with all the details about the the sources with the answering phase allows us to keep the focus. So we have that for each piece of information before the calculation we have the trace of the source and then we have that the answer just have a limited amount of information with the related sources. So it's easy to you know provide exact uh precise information about the the actual source. So here we have the feedback. So the gold answer was no with a bit of information and actually um you know the the evolation is not done by humans in this case. So the answer is provided by humans. But the evalation if the answer is correct of course cannot be done for 200 times and for all the tests that have done auto I mean manually. So I actually asked the LLM to evalate it and to explain why this is correct and actually we see here that the the computation was correct the threshold was correct and that's why it worked out. So this is a you know a better form of what what happened. So in this case this is the first iteration. We see that we don't have any retriever slide yet.

[01:19:31]So here the system is saying okay my approach missing this is what is missing my approach is to use tools. So we are tracking here the decision the thinking the reasoning that we need to get lesson learned etc etc and here is the the case for the first iteration but unfortunately we are missing some uh information here. So and uh here is clearly stated sorry um so this is the query that you run we run vector as you can see that since we have the strategy agent we don't have the initial question actually this has been changed and actually it is a dedicated let's say sentence that could increase the chance the chances of getting the right documents and actually we've got 10 documents but still this is not sufficient because Um you see current retrieval found of flow measures blah blah blah but um here you see that it is using some playbook rules as we will see later um and it saying exactly what is missing because of the playbook that we passed u but still you didn't see how we created this playbook so ignore for a second but just keep in mind that the retrieval feedback is driven also by the the guidance that we provided in these cases that are dynamic of course and we pass through um you know this first iteration then we go to the second one in which you see now we know exactly what is the the document we need the name actually because we asked for fiscal year 2022. So we have exactly the the right document and now the system is going page by page to extract what is necessary actually and thanks to that we've got what we need and we've got our answer that is exactly what we were expecting and the evaluation is again uh done by LLM as a judges. So the answer is correct. Complete and follows the playbook again. You see that the playbook will come back again and again because um you know it learned how to evalate and trust me that this was one of the most complex question to to answer because without this playbook the threshold was not known and um you know the concept of capital intensive it was not clear to the basic LLM without proper guidance in this case but thanks to that we have got the the proper answer. Okay, I have other examples but I I think that we will not have um time to uh go through all of them but I think that the concept is pretty pretty clear.

[01:22:10]So I will stop here again for a second just to summarize a bit what we um learned so far and then we will move to the to the playbook that is the other relevant part of this of this talk. So we've seen that we um prefer to have a simpler uh prep-processing step because the the use of tools the use of um these agent doesn't require to have complex structure behind. So we simplified a lot of the pre-processing in favor or you know but on the other side we have let's say a more complex uh retrieval part you know that actually um compensate a simpler pre-processing but still it will deliver better quality in the uh in the end. So uh the context graph matches perfectly the structure that we have in the agent. So as a suggestion if you have a different agent you should do uh something very similar because this will help you in modeling the the graph you know but also will help you to understand what's going on and understand and interpret the results. So it increases the traceability. Um, interestingly, uh, and let me stress again, we use the same graph, literally the same instance to store both the documents, so the document graph and the context graph. And we will see even the uh the playbook will coexist with that for for a reason. And tools, I showed you three, but you can plug and play uh tools. So, you can add as many as you as you wish, actually. So, okay. Now we have the um agentic rug implemented. We have the context that is keeping track of the um you know what's happening in the um agent and it's also keeping track of the feedback provided by the human.

[01:24:07]Okay. So how can we learn from this? So what you have seen so far in this schema is here in the ACE paper this part even though it's done by a simple LLM call is generally called generator. So I put it into parenthesis because you know if you read the paper you will find this generator mentioned many times. In my case it's not a simple LLM call. It's an entire um agent system agentic system you know uh but it works exactly in the same way. So we keep track of the decisions. This is the important part and the reason why I decompose the problem in multiple smaller steps is because I would like to keep track of more and more. I would like to provide more guidance tailored for specific type of tasks as we will see. So this is where we are. We have an agentic rug. We have the answer provided to the user that will provide feedback and everything is now inside our context graph that we have just seen in one example. But if we have time I can will show you more. Then what happens is in my opinion very very cool. So we have two other agents somehow because this is what um the ace paper suggested and this is actually something that I like from from this approach. They have two specific and different scopes. So the first one is the reflector. So the refactor um actually extracts or creates lesson learned from the context graph.

[01:25:37]So imagine that you have just completed um a question. I mean you got the uh you've got the feedback from the user at that point you have all the information you need all the iterations what work doesn't work all the reasoning all the feedback etc and these are in the context graph. So the refactor extract all these information and create lesson learned. So just describe what has been learned by this execution in a clear way. Uh what pattern worked, what tools was working and actually it also extract what of the existing uh playbook items was useful were usable actually to uh to get to the good result or what was harmful for for that.

[01:26:26]Again this is extracted from the context graph. At this point, reflector passed the ball to the creator. The name creator comes from the fact that its role is to create the playbooks. As I said, the playbooks are not growing indefinitely. In this specific case, I have as a fixed size that 30 actually.

[01:26:46]So I don't want to get more than 30 uh playbook uh items actually. So what it does is that it takes the output from refactor. It takes the current playbook and actually say reason around them and say okay this is what I learned from this this is what I knew already is there something that I can do to improve the quality of the playbook you know what should I remove because this was harmful what I should improve because this can be clearer the next time what I should merge because maybe these two items can be merged in a single item and make it clear etc etc so the name creator comes from this it really uh improve the quality of the playbook at each iteration. This is what happens here. I would like to make also another point actually you can have as many playbooks as you wish in the current implementation. The last one that you will find in the um in the code there are many playbooks and you will see a few examples today because first of all what I did was to split between the phases. So I had a playbook for the retrieval phase and the playbook for the uh answering phase. At the beginning I had only one but this didn't work because they had very specific needs. So I decided to split them and I said okay let's create one for the answering and one for the tribal. And actually this improved a lot to the final quality.

[01:28:12]Then I also said okay maybe that with different type of questions or with the evolution of these tools I could have different type of needs. So I need a different type of playbooks. So what I did just an experiment and actually the results are quite better. I divided this per type of question. So now I have a a classifier that will take the current list of uh classification items and let's say type of questions and decide if the new question can be one of the ex can be classified as one of the existing otherwise we'll create a new a new version a new class actually and then I have a playbook I mean actually two playbooks one for the retrieval and one for the answering for uh each of these class and uh in this way as we will See we have like the class of capital intensitivity that has specific rules for answering specific rules for retrieval because actually retrieving specific elements for um capital intensive is important because we need specific tables we need specific elements inside the the system. So just to give you the full picture what happens after is that I have two options here. I can pass the entire playbook. It can be done. You know, you can pass because it's limited. You can keep it even shorter. It will never exceed the sides of the context window. So, it's fine. But what I noticed is that it can be destructed as I told you about the examples of the Romanian invasion, you know. So what I did as was to in include another element that actually will take according to the type of course of the question the current playbooks and generate five very focus at the guideline five for the um answering uh and five for the retrieval and pass this information that are not generic anymore but they are again specific for the type of um problem that you're going to solve. So you will see in the in the examples that these are very focused on the type of question because as you see here it takes the guide the the question the current playbook and creates these five guidelines for the retrieval and five guidelines for the answer that will be feeded as a part of the prompt as we have as we have seen and now we will see some uh some examples before doing that I'd like you to to see again how we are extending our graph model to include also this uh this part.

[01:30:47]So actually what we have here is that each question is classified in a category and actually if for this category we have a set of playbooks playbook items actually and um actually we have an entire playbook with the playbook items and one I mean is for the one playbook is for the retrieval and one playbook is for the answer you know and these are connected to the specific type of questions. So you always have the um you know the right set of um items from the playbooks that you need according to the category of these question and this is literally traced exactly the same way in the uh in the question and what is interesting is that we have we can keep track of the um let's say the items that were relevant to build the guidance. So what we passed here is also tra traced back to what were the items that we needed to create this guidance. So this is tracked here relevant for building a guidance and you see they are both for um the uh the content and uh and the thing and uh and then we have um also the uh consequence of the execution of the question. So we know uh exactly what are the um you know the items in the playbook that we changed because of these of these questions. generated a new one or deleted an old one or updated an existing one, you know. So everything is traced and again here is the power of of the graph you know we again have another entire area completely traced in this model and that flexibility of the graph allows us to to say literally what we um what happened and um and now um so that's exactly what we needed from this graph model and everything is is here.

[01:32:47]So with a single graph we keep track of everything from uh tracing to answer to feedback to um let's say playbook. Okay.

[01:32:57]So let me show you a few a few examples of the of the playbook and the guidance to better understand um you know the the value of of these results. So here are just a few again here we have the question class. We are taking always the same examples just to keep track of the entire process. So if we the 3M capital intensive uh business has been um let's say traced has been categorized as a um question uh sorry as a capital intensivity assessment and we have a few interesting section in the retrieval phase. So for example here we are saying that when extracting GAP canonical metrics across multiple years targeted the most recent annual filling with a doc scope extract phase etc etc. Um other examples are perform one discovery call um do not run in parallel or not declare retrieval success when required matrix for the users requested calculation etc. So you see that we have a clear u let's say indication clear items about how to use it. What is interesting is that even though this has been extracted from specific questions regarding specific years specific companies etc. I ask explicitly that these items these playbook entries are absolutely generic. There is no indication about the um let's say the year, no indication about the the company and actually these are always generic comments. Let me show you another example from another uh question class. Again, here we are in the retrieval phase um playbook. Again the playbook here is completely different because you know it suggesting something else is speaking about a vector retrieval template include the filling type and reporting period plus the key statement labels. So here is literally telling the uh the strategy agent how to generate the question or the sentence that will be then you know embedded and used for uh searching for the relevant element. So again here we are not searching for inventory we are searching for consolidated statement of earning that is what we need for searching for inventories and inventories of course is also is also there so um when doc name is known issue a page retrieval as I said you know this has been clearly stated and also the indication about plus minus one page has been also mentioned this is for the retrieval so a few examples from the retrieval now a few examples from the uh answer phase.

[01:35:37]So again same class task as before but here you can see what we needed from this system oper for example in this case it is explaining exactly as a domain knowledge what we need for capital intensive uh computation capex over revenue and capex over deviation and also some threshold indicated explicitly this is what the system learned I I I didn't know and I didn't write anywhere in the prompt about this specific thing but the system learned by mistakes probably because the first iterations the system made mistakes and there are many questions about capital intensivity is not the only one that I showed you and it learned what is the threshold for the user to consider that business capital intensive or not and what were the uh let's say the the values and the computation that it needs to do to get this third shot done and again you know there are also indication for the reflector because of these guidelines are provided and not only in the process of answering but also in the process of evaluating the the results.

[01:36:42]So there are even specific indications coming from the trace that um clearly stated must not accept draft what that compare capex with something else. The same is for inventory turnover. Same issue as uh as before. So um I have uh just maning the time let me complete this part and then I will run um a unique demo for for both things. I would like to be sure that we have enough time to comment on the on the guidelines. So as I said the playbooks are generic and they are stored um as a result of their creation process and uh then there is these guideline builder that will take even the question and we customize these playbooks playbook items based on the specific question. So you will see the total how this will be totally different because if the first um you know the playbook items were generic with no indication this will become very specific. So let me give you some concrete some concrete examples you know here we have these 3M capital intensive. So for the retrieval so we see that immediately we can recognize a specific years that were not mentioned in the generic one. So before declaring to complete um you know this is not generally but for example here always include a fiscal year token for example fiscal year 2022 that is exactly what is coming from the question and this was not there you know um if we look at other answer related here we have something that came directly from the uh from the items but also there are some specific elements of the um that are coming exactly from the question. So among the 30 items available in the in the playbook for answering they took these five top and of course the first one is exactly describing what is required to compute the uh you know the capital intensive ratio or parameters you know to get it right. The same is for the other question. In this case again in the retrieval we are indicating explicitly what are the period because it is converting a generic item in a more specific one and the same is here.

[01:39:04]So let me show you then a bit the the code for the refractor and corator and for the uh guidance builder. So hold on for a second because I need to go to the other screen.

[01:39:23]Graph benchmark. Okay. So they are going in another area. So here you see the the process described. So actually again I'm using a lang graph as you can guess there is a this load question data node that literally will perform the the query uh to uh extract the the history.

[01:39:45]So here I mean you will be able to see it but the basic idea is exactly that we start from the question that we are processing right now as a um you know as a next step after the user provided the the feedback and we actually are extracting the entire path we are extracting all the information we need like for the retrieval strategy we need the feedback the thinking process the generation feedback if any then we have for the retrieval call we need the the tool calls we We need the uh the number of documents returned. We we needed also a portion of the of the chunks.

[01:40:22]Actually, we are not returning the entire chunks but just a small one. As you can see here, there is a substring somewhere. Yes, here. So, we have just the the initial part. So, we are passing a lot of information and we are doing the same for the retrieval data. So here is the retrieval part and for the answering part if I go back here then I'm loading the answering question the answering details and again here we have the generate answer step and the other step in which again we keep track of the thinking the reflection feedback um and the answer actually provided by the system. So again we are treating this as a two parallel processes. So let's see what happens with these with this data.

[01:41:06]This is converted in JSON and passed to the um to the prompt. So let's see an example of the prompt that will happen for example here in the refactor.

[01:41:18]Let me show you the quickly the the prompt. So here so the first thing that we are doing because this improved quite a lot of the quality is to pass the the final signal. So I said okay this worked this didn't but as a first thing because as you know LLMs tend to forget what is in the middle. So as a first thing and the last thing we generally try to put what is very relevant the rest also is relevant but not as the the part that you put in the um at the beginning and at the end. So you're an expert retrieval strategist and your goal is literally to you know um extract to carefully analyze the full iteration trace and you know abstract what worked and what doesn't with particular regard with the term that generated the results and what didn't you know because this is exactly what we needed the input is the question the human feedback the um agent final answer you know and the uh the playbook ids. Oh, sorry. This is the creator the refactor. Yes, the playbook ID because we we need to keep track of what guidance worked and what didn't. So, actually we are passing this and say okay this is what happened. this is what we passed to the system. Tell me what work and what doesn't in terms of you know what guidance was very useful what instead was harmful for this for this process because in the thinking is explicitly said that I use this playbook suggestion to do this so if it's wrong it will be marketed as harmful etc. Then we have the creator as you notice here we are passing the entire um retrieval playbook. If before we are passing just a portion of it here we are passing the entire um the current playbook and the role is to keep the you know the number of items below the 30 entries and here are the the instruction that you will be able to to check within the um the system. Then just quickly the retriever that is the the node the part that generates the the guidance. So let me show you the the retrieval playbook for example.

[01:43:35]No sorry this is the playbook.

[01:43:38]So the this is loading and at some point here we go. So this is the playbook guidance prompt template.

[01:43:46]In this case here I'm asking explicity to come up with five clear guidance item that then can be used by the system to um accomplish the retrieval part or the um answering part and this the result of this that is actually um what is returned is exactly what will be passed to the um to the let's say to the agents inside the agentic rack. So and as you can see here we have the playbook we have the agentic um sorry the the query actually and some specific guidance that are specific because we are using a single prompt for uh both answering and retrieval. So here there is a some differences between what are the retrieval let's say critical rules and what are the answering the critical rules that are used for for the entire process. So um with that in mind um I would like to come back to the few examples to uh because we saved some time now and it's time for me to show you a few examples.

[01:44:57]So let me share the screen here.

[01:45:02]So here we are uh we are back. So let me show you a few more examples of what happened. For example, coming back I mean to a different um scenario. We have here uh the same question done twice.

[01:45:17]And the reason why I wanted to show you this is because it's not only about the the quality of the uh of the answer, you know, is also about the uh the the entire process. In this case specifically, what happened is that in the first iteration that is this one, we've got a wrong um let's say a wrong answer. So let me show you the the full process here. So we are >> you need you need to switch. We still see the demo.

[01:45:47]>> Thank you. Thank you. I will never there. So actually these are um the question is this one. Um so who are the primary customers of Boeing as of fiscal year 2022?

[01:46:01]Then we have this one that was run almost at the early stages without a proper guidance and actually it failed.

[01:46:09]So I was going to show you the the process. So this is the question. This is the answer and as you can see the answer was wrong. And the interesting thing is that here we see the the prompt. So, you know, we see what I showed you and actually we have no guidance for the retrieval and we have no guidance for the um for the answer. Let me show you. Sorry, prompt blah blah blah. Okay, this is the the content of the text provided and there is no guidance as you can see here.

[01:46:51]there are no guidance and that's why in the end the answer was wrong because it didn't help to find what was necessary in the second iteration which was longer because I needed more information to answer properly to the question but we have got a proper answer as you can see from here because the previous answer was wrong because it didn't count on a government part actually so let me show you again the reason why it failed So here you see the reason why it failed but omits the key gold fact that the US government accounted for almost 40% of Boeing um revenue. So this was why the answer was wrong. So partially it was correct you know the rest was was fine but this part was missing. So I'm I'm expecting that the guidance provided here will help to do better. And now you see the guidance you know and here you see a specific indication about Boing use vector retrieval as the first one with a short semantic discovery query example measure customers disclosure in Android filling etc etc uh for precise document swatch apply lucine etc. So you see that the first goal exactly as indicated by the guidance was a vector with disclosure primary customers. You know you see measure customer disclosure. So actually this was exactly what the guidance suggested here and actually regardless of this the retrieval was not successful because there some other information is missing. So here again we are saying that is missing customer concentration that was a reason why the previous version failed. So let's see now what is the reasoning in the retrieval strategy again.

[01:48:46]So here we have you know that the specific body customer concentration is missing because this is what the retrial feedback came from and um again here we have no measure indication because of the feedback is enough. We are using text search retrieval this time with specific measure customer one customer accounted for etc etc. Again, this was not enough. And as you can expect, the system switched to the page retrieval.

[01:49:17]So now it's getting the first 40 pages almost and got exactly what it needed in this case. And now the answer was correct. Again here in the prompt, we can see that the um let me show you the guidance from playbook is present here.

[01:49:37]And this is taken into account both in the answering phase and in the uh thinking around the um the evaluator um and uh guess identification etc. So actually this um was correct because you know it answered all the the items required for the evalator and this is why it was considered as a final and then it is actually final uh and correct because of this. Okay. So um a few other things that I would like to show you and then I think that we have some space for questions is an example of the uh retrieval playbook for example for whatever turnover ratio and the fact that we are keeping track of how much helpful some of these pitfall or pitfalls or strategic rules are. So you see for example this uh content was helpful six times and natural one time we have some harmful one um this one was able was helpful seven times uh five times this one etc. So actually these uh strategies and rules and and tools called were were actually useful in many different occasions actually sometimes they are also harmful to be honest I checked for a few of them and they were harmful.

[01:51:06]Okay so with that in mind let me switch to the final part of the presentation very quickly. I um run a few a few tests actually and the one was running while we were speaking. So I will maybe update the um the the presentation after this talk before giving to you. Um you can see uh few interesting things. So first of all this was initial quality. So we were able to answer to only 17th uh questions actually correctly. So with a percentage of 68.5% with the major issues happening here you know for the multi-document because one shot ra cannot do this and uh this for the domain relevant as soon as we introduce the vanilla agentic rug we immediately got a jump so we actually um jumped to 74% overall but what is interesting is that we improved the a bit the domain relevant type of questions but In particular, as expected, we improved the multi-document type of questions part and once we introduced the agentic uh the learning let's say loop actually we've got a dramatic improvement in the quality for domain relevant because of course we are learning by user feedback and also we improved a bit in the multi- document overall consider that the quality was already quite high. So what we improved significantly was you know these 72 that became 84 and these 54 that became 64.

[01:52:45]So we've got quite a um you know a big a big improvement but we are still evolating and uh as I said while I'm speaking uh there are still some tests running for completing this this view because I'm testing after one two three more iterations with some changes in the questions etc etc. So with that it's time to um to close this workshop. Um so um what are the the key takeaways that I would like you to bring home after this uh this presentation. So first of all the role of agentic AI systems in the uh generative AI space because they are literally shaping the future of generative AI as we have seen and we've seen just an example of a gentic rug but in reality there are tons of other use cases code generations accomplishing complex tasks etc. So definitely the word is now agentic for sure. We've seen how memory and learning on top of memory are key um features to improve AI capabilities and improve the traceability of course in particular for for memory. Um we have seen the example of a gent um that can deal with a very complex question and um early results are definitely very uh promising. And what emerged also from this conversation in a few um examples that I showed you is that having the graph offered the the flexibility that I needed for keeping track of the docu I mean for storing the documents for keeping track of the context graph so the the decisions and the outcome storing the playbook storing the guidance etc etc. So actually the flexibility of the nodes and relationship model helped me to have everything in a single um data source and actually Neo forj in particular um helped on the technical side because it offered cipher query language of course for structured type of query like pager but also others that are not mentioned in this talk. The vector index helped for um you know searching the I mean for semantic type of queries and the text full text search helped for let's say item based or keyword based type of search and as I mentioned that the edit tests are very promising I will share more results later on and but what I would like also you to remember is that all these is a complex infrastructure you know um you know the tests that I'm running are taking almost 24 hours to complete you know because all the iterations and such. So it's a complex scenario. So you need to have a complex problem with a high value high stack you know to justify the complexity of this implementation. So don't get bazooka for killing a mosquito. Of course you need really to use the right tool for the for the right problem. And with uh this sentence uh I would like to thank you for listening today and I'd love to hear some of your questions. We have a few a few minutes actually.

[01:55:57]>> Super um amazing session Alessandro.

[01:56:00]Thank you very much. Uh you got also high praise in the chat. So really really appreciate your your time and uh and the the work that went into this workshop today. Uh it's very it's very lot it was a lot but it was very um very very precise very good overview and it also went in into the into the depth of of how how it's done and and and what you can recommend as as somebody that built this yourself. So um I was I was very impressed. Uh the chat uh you know reflects that as as I can see right now with lots of thumbs ups and um and thank you. So really good. But like you said, we have a couple I wanted to uh wanted to dive in and see if we can cover a couple. Um, one comes um about the creator and you know if if the creator won't mess up your your whole pipeline when when he determines something um you know to revising a playbook create new ones. um changing data. Uh there's always the issue, you know, breaking breaking the playbook by adding something new or is is is possible to, you know, how does a retest work with that? How how do they how do they, you know, manage with with this this this this potential issue of of you know, improving something that breaks the 1990s?

[01:57:29]Well, well, definitely um let me say two things. First of all, you can definitely uh explore other type of models and uh extend these models by uh you know keeping track of the history in this version. Um I also keep track of the entire history. So whenever there is a change in something, I'm keeping the old version. this is still uh connected with the uh the questions that were you where this guidance or this playbook was useful. So this is always uh traced uh so the entire history is there with all the details. So you can always come back and see okay for this question I used this playbook why this is not working.

[01:58:14]So you can always run a postmortem analysis if you like. Um also the second aspect that you need to consider is that the prompt is structured so that the the the changes are always justified or are always limited. So what I'm asking to the creator is to take into account the current playbook and decide what really needs to change. Not I mean I'm forcing it not to make a stupid change in in the structure or the of the sentence etc. But something that concretely he has let's say a verific verifiable reason why these need to change. Of course this won't solve the problem but the combination of the full history for a postmortem analysis and the um let's say a proper prompting for the creator can help it. Plus the fact that you keep isolation between answering and retrieval and between different type of questions improves the uh you know the the safety of these overall scenario because of course even though these will affect a specific type of questions all the others will not be affected. So these isolation will help you a lot in this in this sense.

[01:59:25]>> Okay cool. Um, another question I saw from a couple of people either in chat also in the Q&A function is about you know how how do they cope with um the playbooks but you know generally the pipeline how how how do they cope with dynamically changing environments? So you know imagine you have a supply chain and then you know one supplier is not available anymore or roots change through you know reasons um and then all of a sudden the the whole you know everything that that used to work by now is no longer working because of of sudden sudden variations and and and changes in in flow of you know let's let's say supply chain management here transportation issues or something like that how >> yeah I mean consider that these playbook is evolving over time, you know. So, I'm expecting that if there is a a major disruption in the let's say in the context in which these agent operates, I'm expecting that at the beginning there will be some issues because it assumes something that is not real anymore. But I I noticed that the the capability to recover is quite um let's say fast. Let me say that based on my experience I noticed that the system learns more from failures than from um you know right path because it learns exactly what doesn't because the fact that something works doesn't mean that it work forever or for every case but if something doesn't work actually is a very good indication so what I notice is that the system is capable to learn from from it quite quite well.

[02:00:59]>> Yeah. And then maybe maybe another point that's interesting in in this point is you know obviously you made your playbooks as generic as possible. So and through the power of the graphs you you have the the possibility to say okay I sticking with the supply chain management I want to know from you know location A to location zed what's the best way and then through changes in the graph maybe I know okay this this route is no longer possible then another path becomes available or becomes the next preferred way because it maybe it might might have been not the preferred way because it was slower but because the other was no long is no longer available then the other one switches in. So we have the the dynamic concept and the the interconnectability there. You you don't lo you know this don't doesn't go away.

[02:01:43]This still will be available and then dynamically can can make it work.

[02:01:49]>> Yeah. Well, it also depends on the type of tools and how much let's say flexibility you provide to the to the agent to the main agent you know because in theory the problem that you just mentioned can be solved by the agent flexibility to understand that there is a better path that suddenly you know jump it out from the from the test. So the guidance should be very generic in some way you know and provide enough context but never you know force the agent to avoid to explore more or at least have some some fallback mechanisms that will help you know to also explore other ways after some appear to be not available in war or just to explore other options.

[02:02:31]>> Cool. And let's maybe do one or two more questions. Um here's one from Chris. Um, and I think I saw this from other people's as well. So that's why I wanted to to touch about this. And Chris, are the playbooks similar to skills that can be updated by the agent?

[02:02:49]>> Well, that's definitely something that we we could explore. You know, the concept is very is very similar. The idea of the playbooks here, the name comes from this concept that they are um let's say dynamically evolving, you know, uh inside the system. the skills somehow are made for being shared between you know a provider of the skill and the uh and another company. But that's a good idea. You know you can easily convert the playbook that you created by using the system as a in a skill that you can then translate or move to another scenario or to another agent. For example, if for example there are some commonalities that's absolutely a good a good idea. But in my mind the skills are generic and across let's say multiple agents playbook they are generated as a result of the um experience uh direct experience of the agent. So the agent and also consider that the the playbook matches very well with the with the prompt. So they understand how that's why I'm passing the prompt because I would like the system to say okay this is the prompt where you fit these uh piece of playbook you know so it consider also what is before and after so it's very very specific for that agent skills are made for genetic reasons you know for genetic purposes so without considering the actual history or the experience made by the system.

[02:04:18]>> Cool. And then just close it out one more question from Ayala. Um the the full pipeline latency is that acceptable for a chat experience or do you think this is more like a research kind of uh a thing you let run a little bit in the background and you know no real time or close to real-time answers given yes absolutely this is not for realtime answers uh you know these full process takes minutes sometimes what I can suggest and this is something that I didn't add because I I could miss the you know The real um improvement is that I would like to give it a sort of quick shot because with a proper embedding and a proper uh let's say um uh let's say um approach to the pre-processing you can get these uh questions to be answered very quickly.

[02:05:11]So as you've seen for 117 over 200 that it is more than 50% you don't need all this complexity you know.

[02:05:20]So in theory what we can do is that you can have either a system that will see the question and immediately decide okay let's go for the simple solution first and maybe it will answer in a few seconds because it goes for the simple one shot or then we say no no no this is a complex question will require more time so in this way you can balance between >> uh you know fast way and the slowest way I don't know if you remember I mean but in many of the current tools you have a sentence coming out sometimes this is difficult let me think it through, you know, and I think that this is exactly what they do. If the answer cannot be provided immediately, they switch to a different reasoning model and they will give you a better answer, but it will take more time. So, they ask you to be patient. We can do the same.

[02:06:04]>> Cool. All right. Super.

[02:06:08]Lots more question, but uh let's let's continue the conversations um uh elsewhere. Thank you very much again for for watching today. Thank you for for your questions, your your contributions um throughout the day. Um thank you very much Aleandro for last and foremost obviously >> uh for for this amazing workshop. Um if you have um you know any questions um people can can reach you as you shared um in in the at the end of the presentation. Um you can also um you know you'll get an email with the video link. We got an email with the GitHub and the slides. So you can also do um do some some commenting there on on this part and we hope to to get that out over to you before before the weekend. So you will you'll you'll see that coming in and um just briefly before everybody uh goes away I wanted to share that next stream is taking place uh tomorrow actually with Luan um who is talking about Node AI agenda with Aura agents.

[02:07:05]So we have just launched Aura agents um a couple of weeks ago and um we will build a um AI agentic um companion that runs on Aura that helps you structure the the notes AI agenda that is taking place on 15th of April as you all know.

[02:07:22]So um if you're interested in that join us tomorrow um in on YouTube or on LinkedIn and um yeah NOSAI is coming up.

[02:07:31]This was the last road to nodes workshop. I hope to see you all on the 15th of April to discuss um more AI and graphs on a full day um with lots of interesting sessions, lots of interesting tracks uh conversations um here about use cases applications um from Neo forj folks from community but also from um um you know implementers uh that build uh stuff with with Neo Forj and uh and AI. So yeah, looking forward to see you in the 15th and um yeah, until then um have a good time. Uh see you soon. Uh join the community, join um Graph Academy or run an AJ or a free instance if you like to get involved yourself and uh I see lots of people who really want to dive into this Alessandro. So we'll we'll definitely provide the material as soon as we can.

[02:08:24]>> Yeah. And with that >> absolutely I will do it shortly.

[02:08:27]>> Thank you.

[02:08:28]>> Thank you very much. Thank you, Aleandro. It was a pleasure. It was an amazing session, a great finale to a great series this this season.

[02:08:35]>> And um yeah, see you soon next time. Um take care everybody. And uh yeah, bye-bye.

[02:08:41]>> Take care. Bye-bye.