Agentic LLMs & RAG: Tools, ReAct, and Retrieval (Lecture 7) - Stanford CME295

[00:00:05]Hello everyone and uh welcome to lecture 7 of CME 295.

[00:00:12]So today we're going to uh focus on practical techniques to let our LLM interact with the outside world with other systems because um up until now our LLM was purely on its own. We've trained it. We've seen how it can reason on problem math, coding math, coding problems. And uh now what we want to do is to use our LLM in the context of other systems. So today's class we'll focus on rag that you may have heard tool calling and agents.

[00:00:55]But before we start as usual I'm going to recap what we did uh last time. So if you remember last time we focused on reasoning models and we saw the differences between reasoning model and what we call the vanilla LLM and in particular up until the lecture before last lecture. What we saw was we fed a prompt to the LLM and it gave us directly a response. But what we saw last time was that if we let the LLM reason before outputting the response, then we can gain some performance when it comes to reasoning tasks such as math and coding. And so in particular reasoning models what they do is they take a prompt as input and then what they output is both a reasoning chain which is typically hidden from the user and then a response.

[00:02:02]So with that we saw how we could train a model to be more of a reasoning model.

[00:02:10]And in particular we saw a core RL algorithm called GRPO which stands for group relative policy optimization.

[00:02:20]And we saw that this algorithm had some differences compared to the ones that we saw previously. And in particular one notable aspect is that it does not have it does not train a value function.

[00:02:38]So here this uh illustration shows a little bit how gRPO is trained. So it takes a query as input and then it computes an advantage for each output by um computing the rewards for different completions of a same prompt and then computing a quantity which is the advantage that is relative to the other rewards of that group of completions.

[00:03:09]And then we saw that if we applied GRPO with carefully chosen rewards which is one rewarding the model for outputting a reasoning chain and then second rewarding the model to for producing a good response. What we saw is that as the RL training progresses, we have an improvement of the model on these reasoning tasks. And so we saw one of the tasks being uh math problems. So here on the left graph you see the evolution of the performance of the model on the AIM data set which is a kind of a challenging math uh problem.

[00:03:58]And we saw that but we also saw that the model kept on input outputting responses that were longer and longer.

[00:04:11]And in particular we saw that even though you know towards the end of the graph above uh that the performance was kind of plateauing we saw that the output length was still increasing.

[00:04:27]So then what we did was go back to the loss formulation that is used by gRPO and realize that there is a term that makes the contribution of a token different if it is in a short response or a long response.

[00:04:49]So this is a phenomenon called length bias and we saw some mitigation strategies that were explored by some papers that came out uh in the past few months. So one was DAPo which um had a a normalization factor that was not dependent on where the token was located on which sentence it was located. And the other one was uh this paper called GRPO done right which actually just removed the normalization term.

[00:05:27]I'll go down that.

[00:05:29]Cool. So this was last time and last time what we said right before starting the reasoning class was to enumerate the strength and the weaknesses of vanilla LLMs.

[00:05:45]So last lecture was all about focusing on how we can improve the limited reasoning capabilities of vanilla LLMs.

[00:05:57]And in this lecture what we will do is two things. So the first one is see how we can connect our LLM to the ever evolving knowledge base and in particular see how we can um have access to the latest information.

[00:06:20]And then the second one is how our LLM can help us perform actions and we will see this with Shervin uh with things like tool calling and agentic workflows.

[00:06:34]Cool. So with that let's start with the first one and let's start with this method called rag that you may have heard.

[00:06:44]So let's suppose you have a model that you have trained uh but the problem is that the pre-training data on which you have trained your model is let's say a month a month ago let's suppose now let's suppose you want to prompt your model about the winner of the elections that happened a couple of weeks ago. Well, your model will not be able to respond to you or it will output the incorrect answer because up until now our LLM does not have any link to outside sources. It only relies on the knowledge that it has acquired during training. So the response that it will give us will only be based on the data that has been trained up until the cutoff which is a month ago in this example.

[00:07:45]And so we have this big limitation which is our LLM only knows about things that is it has been trained on. And you will see that all the models out there. So here I have an example with OpenAI GPT5.

[00:08:02]So if you look at their model cards, they always have these knowledge cutoff dates that is written somewhere. And in the case for instance of GPT5, the knowledge cut off date is September 30th, 2024.

[00:08:19]Which means that if you ask it in a very naive way, anything that happen after that like the base model will not be able to answer you as is.

[00:08:30]Well, you might you may tell me why not just uh continue training your model uh on you know data that happened after that. Well, the problem with that actually there are several problems with this. So the first problem is that it's very tricky to change the knowledge of an LLM without causing regression on other things. So this is typically a task that people they try to avoid doing. And the second thing is it's not very practical because you may very well have use cases that require you to fine-tune this model.

[00:09:12]So let's suppose you have a use case one. You fine-tune from this model and then somehow you want to update the weights of your model to inject some knowledge. Well, you somehow will have to do that for all the use cases that you are doing which basically adds a lot of overhead for you and just uh you know adds a lot of maintenance.

[00:09:38]So people they typically prefer to not do additional training to inject knowledge. So one idea can be to somehow take your prompt and just add anything that happens after the cut off date as a way for your model to just know what happened.

[00:10:02]Well, the problem with that naive approach is that as you know context length length is limited and so typically models have on the order of magnitude of oh of hundreds of thousands of tokens in context length.

[00:10:23]Do you know what that is roughly like what is this roughly equal to?

[00:10:34]Yes. So one token is equal to four characters. So like using this like rough approximation uh hundreds of thousands of tokens is roughly like hundreds of pages like something like a very big book. So so it's not it's not I mean it's it's big but it's not enough for us to go in that very naive route. So um so again going back to GPT5 uh so if you go to the model card you have the knowledge cutoff date which is uh September 2024. You also have the context window and in this case it's 400,000 tokens.

[00:11:14]U okay so let's suppose you actually uh context is not a problem. It's actually unlimited.

[00:11:22]Let's imagine we actually put everything in the context. Well, the problem then is that people noticed that if you feed a lot of irrelevant information to your LLM, the performance of the LLM will actually degrade.

[00:11:41]Meaning that if for instance you ask it about I guess who was the winner of the last elections and then you you feed it a bunch of information that are not relevant your LLM will tend to be confused and so people have run this test that's called the needle in a haststack test where the idea is you give a big prompt to your LLM which is your haststack and you place a fact in the prompt and you ask your model what that fact was.

[00:12:26]So the idea is for your LLM to know I guess among that huge prompt where is the relevant information which is the needle. And so when people have tried doing this for several uh length of prompts and tried different positions of where to put the fact, they have seen that the length of the prompt and the position at which you put the fact are both important.

[00:12:59]So here uh on the slide we have a heat map um that was performed for GPT4 which was I guess one or two years ago and uh what the person did was place the fact at different places in the document. So this is document depth and the x-axis is the length of your prompt.

[00:13:22]And what we saw was that for prompts that exceeded a certain amount of tokens, the LLM actually had trouble retrieving the correct piece of information.

[00:13:34]Um, and in particular, it had trouble doing so when the fact was somewhere in the first half of the prompt.

[00:13:43]Um so this just tells us that you know even if [snorts] like let's say our context length was unlimited we would still have a problem by just going through that naive approach.

[00:13:57]So that's another reason. Okay. So now let's suppose context length is unlimited. Let's suppose the problem that I mentioned is not a problem. Well the other problem is that you pay So in particular, these calls, these LLM calls, they are per token.

[00:14:20]So the bigger your input prompt, the more you will pay. So you have an incentive to not put too much in your prompt just from that standpoint.

[00:14:33]And so for instance again going back to GP5 uh order of magnitude is somewhere around a dollar per million token. So I guess it's not that expensive but it can add up if you do that for all your prompts.

[00:14:50]So for all these reasons, I hope I convinced you that we need a more clever approach where instead of putting all the new information all at once in the prompt, what we do is we only somehow find the relevant information and put that in the prompt.

[00:15:13]So that is the idea behind rag. Rag stands for retrieval augmented generation.

[00:15:22]And the idea here is to augment the prompt with relevant information.

[00:15:30]And here I put uh relevant in both uh and this is the I guess the core part of this technique is how can we get only the relevant part in the prompt. So we will see that in a second. So just at a very high level. So you have let's say a question as input. So in this case who was the winner of let's say the local election. The idea here is to somehow fetch the correct or the relevant piece of information and then augment that here in order to output your answer.

[00:16:11]So that's the rough idea.

[00:16:14]So does this uh method make sense so far?

[00:16:20]Yeah. Okay. Cool. So that is the idea behind rag and now we're going to go into more details.

[00:16:28]Um so what I mentioned is the rough idea. Um and here I just want to emphasize on the three main steps of rag. So first one is you have your prompt and you somehow want to retrieve a relevant piece of information that will help you in answering your prompt.

[00:16:48]So here the first step is to retrieve relevant documents and so you can think of your prompt as being one entity and then you can have some other uh space which maybe like I don't know knowledge base where all your documents live and so the idea is to somehow fetch the relevant documents.

[00:17:12]So this is the retrieve step. The second step is once you have fetched the relevant information, you augment your prompt. So you take that retrieved info, you just put it in your prompt and then uh ask the question. So in the local election example, it's as if I was saying who is the winner of this election and then I retrieve the relevant piece of information and now the prompt becomes um who is the winner of this election and by the way uh this election was held blah blah blah and this was the winner and this is what we're feeding to our LLM.

[00:17:53]So in other words, we're giving the answer in the prompt and the third step is to feed that prompt to the LM to generate the response. Yeah.

[00:18:20]>> Yeah, exactly. So the question is uh you may very well somehow do a bad job at retrieval stage. So yes um so this is why the retrieval stage is so important and we're going to focus on what we can do to make sure that one part like does well.

[00:18:38]Um so yeah uh we'll see how we can evaluate um I guess our our setup and different methods. But when we talk about rag, we're mainly focusing on making the retrieval part as good as it can.

[00:18:58]Cool. And I just want to emphasize once again on why it's called rag. So you have retrieve, augment, generate rag.

[00:19:13]Cool. And as you pointed out the first step which is the retrieval step is very important which is why we'll spend a little bit of time over there.

[00:19:25]So I guess the first step is for us to somehow clean the set of documents that we may need.

[00:19:37]So I said you know uh we may um want to look into outside information but we need to somehow uh sort that order that put that somewhere and this whole thing is usually called a knowledge base.

[00:19:54]So in order to form our knowledge base what we do is typically collect the set of documents that are or may be useful and once we do that what we do is we divide them into what we call chunks.

[00:20:12]So a chunk is you can think of it as a subset of the document which has a given maximum length which is uh measured in number of tokens which is typically on the order of hundreds of tokens.

[00:20:29]And the idea here is you know whenever you hear retrieval you should think about embeddings.

[00:20:37]And here what we do is we compute embeddings corresponding to each of these chunks.

[00:20:46]Now when you create your knowledge base there are a few hyperparameters that you need to tweak.

[00:20:54]So the first one uh obviously is the size of the embedding. So typically you would want a bigger size if let's say your documents are maybe more nuanced, more complex.

[00:21:07]But then if you have like a higher size uh maybe it will take more space, maybe you'll have more computation at inference time. So I guess it's a trade-off. You don't necessarily want too big of a of an embedding size. So here typical embedding sizes are on the order of thousands. So for instance like 1,500 something like this.

[00:21:30]So then you have the chunk size. Chunk size is how big your little pieces here are. So you don't want them to be too small because otherwise the text may be out of context. You don't want it to be too large because maybe the embedding will not represent in a meaningful way what is inside. So again, it's a trade-off. But typically people they choose a chunk size of around 500 tokens like on the order of hundreds of tokens.

[00:22:05]And then you also have a oh yeah you have a question. Yeah.

[00:22:10]So the question is do you train an embedding model for this? So you have two choices. Either you can use a pre-trained embedding model which people typically do or you can train your own.

[00:22:23]We will see that in a bit more detail in a few slides.

[00:22:34]So the question is what is the purpose of the embedded model? So we will see this in a second but long story short it tries to represent chunks such that it achieves your end goal which is to fetch relevant documents.

[00:22:52]So we will see a little bit how they're trained but this is uh the general idea.

[00:22:58]Cool. Um so that's this and then we have a third hyperparameter which is how much overlap you want to have in between your chunks.

[00:23:11]So here um when you do the division uh you know you uh like in a very naive way you have everything be like independent no overlap in between but typically you have some part that is from the previous chunk that is relevant to understand the current chunk which is why we want to have some overlap and which is why people they typically also have that. So it's typically in the low hundreds of tokens.

[00:23:42]Cool. So let's suppose you have your knowledge base. Now the question is given a prompt how can you retrieve relevant documents and the answer to that is we typically proceed in two steps.

[00:24:00]So I'm not sure if any of you has background in recommendation systems or search does any Yeah. So the methods we're seeing here are very similar to that space. So I guess people in the LLM community they have borrowed ideas and just leverage some techniques that we have over there. And this is typically a setting that we'll also have for recommendation problems.

[00:24:27]So we have two stages.

[00:24:29]So the first stage is typically called candidate retrieval.

[00:24:35]And the goal here is to go from a set of many many many chunks and filter it down to a much smaller set of potentially relevant candidates.

[00:24:49]So during that stage, what we're trying to do is to somehow maximize recall.

[00:24:55]just do a rough operation so that we get as many potentially relevant candidates as possible.

[00:25:04]And then we have a a second stage which is sometimes optional, but this stage is to really make sure we have the top documents being the really the relevant ones. And this one is called ranking.

[00:25:21]So the idea here is based on the list of potentially relevant documents to really rank them in a way that really the relevant ones come at the top and so on.

[00:25:36]And typically during that stage, we're going to use a model, a method that's going to be a bit more compute inensive because we have a much smaller set of candidates to rank compared to the first one.

[00:25:52]So going back to your question on how do we want our embeddings to be. So here it will really impact uh the first stage and we will see that in a second, but the second stage is also quite important.

[00:26:08]Cool. So far so good. Is everyone uh clear with the two-stage approach approach?

[00:26:15]Yep. Yep.

[00:26:18]Yeah.

[00:26:27]Very good question. So the question is do we chunk things in a naive way as in we just go with the number of tokens regardless of what happens. So it's a great question and the answer is that we will see some extensions that will mitigate the problem of when you chunk it in a way that does not make sense in a naive way you want to somehow put that into context and we will see a method that does that. So in a few slides we will see that.

[00:26:56]Um but I think your question is also a great question because depending on the kind of document that we have like for instance if we have uh I don't know like a JSON file or a markdown or like depending on the kind of file that you need to chunk you also need to be aware of the structure that is within those files. So there is also some nuance there that we will not go into details but I just want to call that out but yeah great question.

[00:27:26]Any other questions?

[00:27:29]Okay cool. So now that we're clear on the two main stages of retrieval we're going to focus on each one of these steps.

[00:27:40]So as I mentioned the first step is candidate retrieval.

[00:27:46]So here what we want is among that potentially huge knowledge base to somehow filter it down to let's say over 100 potentially relevant candidates.

[00:28:02]So here what we do is well we will leverage the embeddings that we um I guess computed during the knowledge base initialization and we will try to fetch potentially relevant candidates by doing a semantic similarity search.

[00:28:25]So do you recall how we compare embeddings?

[00:28:31]Yes. Yes. So cosign similarity is typically one way one good way to compare embeddings. So the idea here is to represent our query with an embedding.

[00:28:43]We already have embeddings of all our chunks. So the idea here is to somehow find the most relevant chunks by doing this similarity search and filtering out the ones that come at the top.

[00:29:03]So the idea here is you have your query, you have your chunk both of them, you find an embedding and then you perform a similarity operation which is most of the time cosign similarity and you obtain a similarity score.

[00:29:20]So the idea here is you just just keep top I don't know 100 and you go with that.

[00:29:28]So I just want to call out that there is some complexity in that stage because your knowledge base can potentially be huge.

[00:29:40]So what people do is typically use what we call approximate nearest neighbor methods.

[00:29:48]So you may have heard of some libraries [music] that do that. So typically this is something that will be relevant here.

[00:29:54]We're not going to go into details, but I just want to call that out. So here, the idea here is that when you build your knowledge base, you somehow partition the embeddings in a way that will avoid like make you avoid doing like just a naive linear search.

[00:30:18]So that's the idea. But uh you you may see some techniques uh like ANN techniques approximate near nearest nearest neighbor techniques and these are typically happening here.

[00:30:30]So another thing that I want to point out is a name of the architecture that we typically use here that you may also hear and for that we need to recall that these embeddings they're actually obtained by passing them through a model. So typically encoder only.

[00:30:53]So you may hear the term by encoder and this one refers to the fact that we are passing the query through an encoder and then passing the chunk through an encoder. So both of them are independent and we're comparing the embeddings and uh yeah so this is another question I wanted to ask you but I guess I didn't get the chance to but um so if you remember I think lecture two or three we had seen the birds model and so typically you would have something like a birdlike model that you would use to um encode these documents.

[00:31:41]So going back to your question, how do you compute these embeddings? So there is a a paper that I highly recommend reading actually that's called sentence BERT.

[00:31:52]And so that paper explains so it's first of all it's an extension of BERT as the name suggests and it's an extension that allows you to compute an embedding per let's say sequence for your query for your document that is tailored to be used for similarity search purposes.

[00:32:18]So the idea here is to have a loss function that will incentivize having a high cosine similarity for relevant entities and low cosine similarity for entities that are not relevant.

[00:32:34]So yeah, so feel free to check that paper out uh if you know birds which I I know right now you do. Uh it's quite easy to read. So yeah, highly recommend.

[00:32:45]So far so good.

[00:32:48]Yeah. Yep.

[00:32:54]So the question is what is the default way to compute the similarity? So yes it's cosign similarity but again you will see in different implementations that people can use other distances and I would encourage you to think about how they relate to one another. So you will see for instance the L2 distance but then if everything has a norm of one there's a lot of simplifications that can happen. So you may see some variance but I would say they're all more or less cosign similarities.

[00:33:29]Yeah great question.

[00:33:32]Cool. So we're still at the candidate retrieval stage and what we saw was one way of retrieving documents from a similarity sorry from a semantic similarity standpoint. So by the way what does semantic similarity mean? It means finding documents or finding entities that have the same meaning or that are relevant.

[00:34:01]But in the way that we compute these embeddings, we're not enforcing any kind of uh keyword match.

[00:34:12]Like when we re when we retrieve documents in this way, it can very well be that the documents that are matched, they do not have any word in common, but they mean the same.

[00:34:26]Well, sometimes you want to ensure that your what you're looking for, what you're searching for is exactly containing the keywords that is in your prompt.

[00:34:38]And in that case you would want to have a second way of doing things. So you may have seen BM25 out there. So BF20 BM25 is a relevant score that is actually a huristic score.

[00:34:56]It is based on some function of the overlap between what is in your query and what is in your document.

[00:35:09]And so that one is actually quite handy for cases where you have a query where you absolutely want to have documents that contain keywords of this query.

[00:35:24]So here I have an example that I actually passed super briefly for the previous one and we will come back to it. Uh but let's suppose we have let's say two teddy bears. One is named cuddly and the other one is named huggy. So what you want is to figure out where is cuddly. So this is your query.

[00:35:47]So if you use BM25 well the answers that you you're going to get are by definition going to contain some overlap of words that were in your query. And so here you will have let's say uh documents that contain let's say where cuddly is. But if let's say you you only used this semantic similarity search you would not have that guarantee.

[00:36:21]you would only have documents that are kind of semantically similar and those are not they are not guaranteed to contain keywords of your prompt and so just just to illustrate that. So here huggy and cuddly they they can be thought of you know semantically similar. So you you will probably not have cuddly. I mean you may not necessarily have cuddly in there just to illustrate that and that is the reason why nowadays what people do is to look at the use cases that they have and think about whether having some heristic as well in the relevant score is useful for their use case. So some people they go with the hybrid combination of this embedding based search and the huristic based search. So some combination of uh embeddings and BM25 in which case you may have even more relevant documents depending on your use case.

[00:37:32]Does that make sense? Yeah.

[00:37:36]So now I'll come back to what you mentioned about whether cutting chunks in a naive way will necessarily lead you to things that are coherent. Well, you're completely right. Sometimes you will not. Um but before we answer this question, we actually are going to address another concern which is that typically when people want to ask about something in their LLM, the query that they input is of a different nature compared to what is in the knowledge base. So your query is typically going to be maybe something uh short, maybe a question, but what is in your documents is typically going to be, you know, longer.

[00:38:26]These are like sentences and sentences.

[00:38:28]So if you really think about it, if you use the same encoder to embed your query and to embed your documents, well, these two embeddings, they're not super comparable because one is for a question and the other one is for a document.

[00:38:50]So there's one extension that tries to mitigate that issue and uh I linked the paper down there. It's called the height.

[00:38:59]So what it does is instead of computing the embedding related to the prompt, it will first generate a fake document. So it's just an LM call, a fake document based on that prompt and then embeds that fake document to find relevant chunks.

[00:39:26]So it may or may not work. It's not used all the time by everyone. So I would say just something that is good to try to see if that that works. But this is one way of mitigating this. Another way it could be to simply have encoders that are specifically trained to encode the query on one side and encode the documents on the other side.

[00:39:52]So in other words to not use the same encoder.

[00:39:56]People typically don't do that just because of maintenance purposes but this could also be another solution.

[00:40:04]Okay. So now finally going to your your question regarding how we can make sense of these chunks. If they are taken out of context they may not make sense.

[00:40:14]And so here the idea here is to prepend some piece of text that just sums up what you need to know in order to understand that chunk.

[00:40:30]So here the idea is that you have all your documents. Let's say you have one document that you divide it into n chunks.

[00:40:39]The idea here is instead of considering these chunks separately, you're going to compute some kind of context that is relevant to each chunk and that is based on the whole document.

[00:40:59]So how are you going to do that? So well it's again an LLM call. So what you do is typically uh have let's say the whole document and then you have the chunk that you want to contextualize and you ask your model well please give me a short 16 context to just make sense of that chunk and now you may tell me well that's a lot of LM calls you have potentially a lot of chunks and that's just going to be very pricey. Well, there's one strategy to make this less expensive.

[00:41:37]And I'm not sure if you've heard that uh option. It's called prompt caching.

[00:41:44]So now that you know very well how LLM work, you know like typically these are decoder only and so on. So you know that if you use the same prefix for all your prompts, well it's going to be the same computations that you just do again and again and again.

[00:42:10]So the idea here is you just do it once and you save all the relevant activations and instead of computing them again you're just going to look them up just do a lookup and then decode the rest.

[00:42:29]Does that does that make sense?

[00:42:32]>> Yeah.

[00:42:35]>> Um the question is activation from a language model. Yes. Because when you feed a prompt to your model and you ask it to generate a response, what the model needs to do is well to take all this like input and then compute the activations of all the layers and then have uh you know for the generation process to have uh this attention across all these other components. Well, given that it's decoder only, meaning it's only left to right, the thing that you input, if it's the same, then it will lead to the same activations.

[00:43:14]Yeah.

[00:43:20]The question is, what if you have a closed model? where prompt caching and I'm going to just talk about this in just one slide is an option that these uh like closed models or providers offer. And what they tell you is well this is the same prefix for all your prompts. So what we're going to do is we're just going to make it cheaper for you.

[00:43:47]So if you look at the model pricing page, you will see that there is a price for regular inputs. So inputs that are not cached and then you have the price per cached input token. And here you see for the let's say open model it's one10enth of the price. So I guess what do I want to tell you by this? Well, just try to be smart with the prompts and try to gather all the things that are likely to be repeated across prompts in the beginning so that you can uh leverage this nice uh uh percentage of that make sense.

[00:44:35]Okay, cool.

[00:44:38]Great. So up until now we have seen how we can go from potentially thousands or even like let's say millions of chunks up to uh or down to uh let's say hundreds of potentially relevant chunks.

[00:44:55]Now what we want to do is to sort them in a more meaningful way. And the second part is more optional because maybe sometimes this first cut that we've done may be good enough. But I guess this second step is about being more intentional in how we give the final score to be able to really like select the final let's say top K chunks.

[00:45:24]So this second stage is called ranking or even reranking.

[00:45:30]reranking because I guess with the first step you already have some kind of ranking. So we're reranking and what we're doing is instead of using this very quick operation similarity operation between embeddings that we've computed, we're going to use something that is maybe a bit more sophisticated.

[00:45:53]So instead of considering the query and the chunk separately, what we're going to do is to actually put them both in the encoder, both of them, and have a relevant score out of that.

[00:46:11]So the reason why it's maybe a little bit more meaningful to do it this way is that you have a model that takes a look at both your query and your chunk at the same time and gives you gives you a score.

[00:46:27]Whereas in the first step you had one embedding for the query and one embedding for the chunk which didn't have that like interaction that a model could capture.

[00:46:40]And you will also see out there that this setup is called cross encoder setup because you have both your inputs fed to your encoder. So there's like some cross interactions.

[00:46:55]So if you remember the first approach is a by encoder kind of setup and this one is a cross encoder.

[00:47:09]Yeah, the question is you will actually compute the attention between the two.

[00:47:13]Yes, absolutely.

[00:47:17]And uh here I mean um sentence bird there they have a lot of uh good documents. So I highly recommend just reading their docs there at the bottom of the slide.

[00:47:28]Cool. Well, you do that uh on all your uh potentially relevant chunks. So you have uh this uh score that is computed for each of these chunk with the prompts and then you finally obtain the ranking.

[00:47:46]And now the question is are you happy with the ranking?

[00:47:51]And in order to answer that question, you need to have a way to quantify your performance.

[00:47:58]And that's why we're going to see in the next five minutes what are the metrics that we typically use to do that. So again this is very similar to you know if you do search or recommendation. So in case you have a background there you will see some commonalities.

[00:48:18]So here's the setup.

[00:48:20]You have a bunch of chunks.

[00:48:23]You do this first and second step and at the end of the day you will have k chunks that will come at the top that you will qualify as being relevant and you want to compare that with respect to actually relevant chunks. So you can think of it as you know you have label like same as in binary classification.

[00:48:50]So you have relevant and not relevant and you predict some that are relevant and you want to know how well you're doing.

[00:48:58]So this is the setup.

[00:49:00]Well, when it comes to ranking, you need to somehow incorporate this information of how high in the ranking you've put stuff.

[00:49:12]So here let's suppose that you have ranked these n chunks from most important to least important. So let's suppose you have like first, second, uh third and so on and so forth.

[00:49:27]And you only care about the first K because in the rack setting you typically retrieve the top K that are relevant and you put all these top K in your prompt.

[00:49:41]Well, the first metric that you will likely use is called NDCG.

[00:49:49]So, it's a lot of letters. I'm going to just explain what that means. So, NDCG tries to quantify how good your ranking is by taking into consideration where you ranked relevant documents.

[00:50:08]So you have this formula that may seem scary, but it's actually quite simple what it's trying to do. So it's trying to incentivize the score to be higher if you're ranking the relevant documents closer to the first position.

[00:50:31]So what it does is it is a sum over the first k positions that you've ranked and it is looking at whether what you've ranked for each of these positions is relevant or not.

[00:50:47]So for instance it checks the first position. Okay, first position is it relevant or not? So relevance if it's relevant it's one. So it's one over some quantity that is a function of the rank and it does that for all the ranks.

[00:51:05]So this score score will be higher if you rank relevant documents high.

[00:51:13]So that's basically the goal of this metric. Um so this part is the discounted cumulative gain. So it's cumulative grain because you're looking at basically if you had relevant documents in your first k positions which is the cumulative part it's discounted because it's better for you to have a relevant document in position one let's say than position k and you can see that in the denominator.

[00:51:48]Now why do we say NDCG like what is the normalization part? Well, this metric can take a lot of different values depending on how many relevant documents there are. So what people do is they compute the quote unquote ideal or optimal or upper bound DCG that you can get for a given query.

[00:52:13]And they call that ideal DCG.

[00:52:17]and they just normalize DCG over IDCG.

[00:52:23]So the reason why they do that is they want you to score a score of one if you are matching the optimal ranking.

[00:52:33]They want they basically want to make the score meaningful.

[00:52:39]So does this make sense?

[00:52:42]Yeah.

[00:52:51]So, so the question is how do they compute the relevance uh score? So, you can think of step one and step two as being a two-step process for you to say which documents you're saying are relevant. So at the end of this two two-step stage, the ones that you say are relevant are here.

[00:53:16]Now you typically have a score for each retrieved chunk. So you're going to sort these chunks.

[00:53:25]You're going to sort them by the score in a descending order.

[00:53:30]And the relevance here that you're going to use is the actual label. So is the chunk actually relevant or not? So you're going to look at your k retrieved chunks and you're going to ask yourself, okay, is the first chunk actually relevant? So you have a label? You know which ones are relevant, which ones are not? And these ones are going to be the ones you will use in the formula.

[00:53:58]Yes, that's the ground truth. Exactly.

[00:54:00]Yeah.

[00:54:04]Cool. Does that make sense?

[00:54:07]Yeah. Okay, great. Uh, so you have a bunch of other metrics. You have another one that's called the reciprocal rank.

[00:54:16]So this one is much simpler. Uh, it takes the uh inverse of the rank of the highest rank of all the relevant documents. So if let's suppose in your first in your top K documents let's suppose the first relevant document let's say comes at rank number two then rank will be equal to two. So it basically does not care about uh any relevant documents that are passed that first relevant document. So this is just like a simpler metric that typically correlates well. So that's why people use it.

[00:54:59]And then of course I mean you're familiar with the classic classification metrics recall and precision. So if you remember if you have two classes you have the positive class you have the negative class. So recall what it does is it takes a look at all the actually positive observations and it tries to ask itself among all these actually positive samples which are the ones that you actually predicted positive that's a recall that you all know and there is a I guess a ranking equivalent which is out of all the documents that are relevant actually relevant. So this is like your positive class which are the ones that you actually predicted as being relevant.

[00:55:55]So this is basically which ones are in the top K.

[00:56:01]And similarly you also have the precision equivalent. So if you remember precision is out of all the ones that you have predicted to be positives.

[00:56:13]How many of them are actually positive?

[00:56:17]So this is the equivalent here. So which are the ones that you've predicted to be positive? So which are the ones that you have selected in your top k and among those ones which ones are actually positive. So which one are actually relevant?

[00:56:35]Does that make sense?

[00:56:38]So I guess uh these four metrics so NDCG, MRR, precision at K, recall at K, you will see this see this in a bunch of papers. So I just highly recommend you just get familiar with their ideas maybe with the formula and uh these would be the ones that you would use to quantify whether your retriever is doing a good job or not. So you have a bunch of benchmarks out there. So there is one that is actually quite popular. It's called massive text embedding benchmark.

[00:57:10]So if you want to test your retriever if uh it performs well or not, you would typically uh take it and then just evaluate it on that benchmark and then have all these metrics computed and then if you have like different solutions you would typically compare this metric.

[00:57:30]And with that, I guess we uh I guess hopefully have a better sense of how to build a rack system and specifically how to have a good retriever. So we have just [snorts] maybe one more minute. Uh is there any questions on that first part?

[00:57:48]>> Yep.

[00:57:58]So the question is so for the reranking we're using an encoder that will output the relevant score. So we will typically train a model that does that. Um so there are typically I don't know I believe there are some pre-trained ones but out there you you can very well have your custom one. Um, so yes, so that model will typically be a bit more sophisticated compared to the first step because here you can afford to spend more time to produce that score because you're operating out of let's say of hundreds possible candidates as opposed to let's say O of much more like millions or hundreds of thousands. So yeah, so that's the idea.

[00:58:41]Yep.

[00:58:46]So the question is uh how about training with contrastive loss? So here this is a detail we will not uh I guess um cover here but uh yeah in order to train these models you'll have a bunch of different uh types of loss function. So I highly recommend you read the espert paper because in that paper there are several loss functions that the paper tries to compare and this is one of them. So uh yeah, so that's a great question. I highly recommend reading the expert paper for that.

[00:59:17]Cool. With that, I'll give it to Shervin. Thank you.

[00:59:24]So thanks a lot for covering uh the rag methodology. So now we arrive at my favorite part of the lecture. Um, we're going to see tool calling and the agentic world and we're going to see how much more powerful your LLMs are going to become just in a second.

[00:59:42]So, what Ashin just mentioned is how you would deal with incorporating data that is not structured as part of your uh prompts to the LLM. And now we're going to see what we could do more in the case the data that we want to inject is structured. So in the case of rag, you have documents with words and words and words and you just want to fetch the relevant documents to answer your prompt. But here let's suppose that you have some structure that determines uh input outputs in uh in your data. So typically you could represent it maybe as a table. So you have uh separate columns and then depending on the value of uh given columns you have a given output.

[01:00:31]So we could probably reframe that setup into a function setup. So we c we mentioned tool calling and then this is this rephrasing is called a function calling. So let's um suppose that you get this relationship between input and output through a function for the rest rest rest rest rest rest rest rest rest rest rest rest rest rest rest rest rest rest rest rest rest rest rest rest rest rest rest rest rest rest rest rest rest rest rest rest rest rest rest rest rest rest of this part and then here if I had to transpose what the result of a given ID and field and other arguments would look like you could interpret it as a function with these as arguments and the output would be simply what you have as an output to the function And um in the world of tool calling and function calling very often times you're going to see that LLMs tend to use Python as a language just because it's so simple to read. Uh so this is what we're going to use as an example as well. But there is nothing that ties us to Python necessarily. So you could well have tool calling in other languages. Uh but yeah just as a note. Any questions on the setup?

[01:01:48]Okay, awesome.

[01:01:50]Um, so there is nothing controversial about tool calling, but I'm going to still state a full definition to make sure that we're on the same page.

[01:02:00]So I was browsing here and there and then try to find an authorative source and this website uh like on IBM there is an article that defines that tries to define what tool calling is. So I'm going to anchor on their definition from now on. Uh so I'm going to read out loud. So tool calling allows autonomous systems to complete complex tasks by dynamically accessing and may act upon external resources.

[01:02:30]So the things that I want you to get from that are two things. So first you have the notion of completing some task.

[01:02:39]So you're given an input and you you have to complete some task and then uh the reliance potentially on external resources.

[01:02:49]So it doesn't have to be external. You don't have to rely on external resources, but this is one potential property that can help you um fill the gap that Afin was mentioning at the beginning of the lecture regarding filling uh the knowledge gap that your pre-trained LLM has. So, we're going to see examples in a few minutes regarding what that could mean. But, uh yeah, this is one u like magical part of it.

[01:03:22]Okay, great.

[01:03:25]And just to make these uh statements very grounded in a real life applications, let's just walk through what would a tool call give us uh in the case of a very specific example. So let's suppose you love teddy bears.

[01:03:44]You're currently, you know, here at Stanford and you want a teddy bear near you. You know what? If you pull your phone out and you just ask, you know, find a teddy bear near me. Well, your current LLM without any tools would not know uh you know some real life or like real time update of uh of you know availability of teddy bears near you. So it would probably answer you something in the flavor of you know I don't know or you know not sure. And I just want to say that with the use of tools, uh, let's see how we could get to a stage where we can inject the information that is necessary for the LLM to know how to respond to your query.

[01:04:31]And the goal of the next few minutes is going to be for us to figure out both what we could do uh like what we could inject in the preamble of the LLM and what would be the steps that we could go through in order to complete such a request.

[01:04:52]Does that make sense so far?

[01:04:56]Yep.

[01:05:02]So the question is are those function APIs premputed? So that's a great question. So yes, you define them beforehand. You have some API and we're going to see that in a second. They're not LLM generated on the fly.

[01:05:17]Y Okay, great. Any other questions on the setup?

[01:05:23]So I know it will be a lot lot to take in. So yep. Okay, great. So in order to be grounded in real life, let's take a full example of what a function definition could be. So in the case of finding a teddy bear, you could imagine a function definition that is called find teddy bear and depending on your location calls some API and retrieves potential candidates.

[01:05:58]So I'm going to go through the uh main characteristics of what a function call contains and link it to the definition.

[01:06:08]So you have okay so first of all when we want to display such an API to the model you need to document it uh its input and output in order for the model to know what this function is for. So typically uh the description that you have in example of Python under a given function would be crucial for the model to know what it is about.

[01:06:34]And then um we saw in the definition of tool call that we uh that we just gave that we have the ability to make backend calls.

[01:06:43]And this is exactly what we would need to do in the case of finding uh like teddy bears around you. you would need to query some API to retrieve available teddy bears and based on your location return the nearest ones. So this is exactly what this um what this function implementation is doing and uh and as um like it returns something that is well structured. So you see uh maybe it's it's small from where where you are but you have some uh class definition that puts some structure into the output and makes it interpretable.

[01:07:24]And we're going to see very soon that this output is going to be what the model will anchor on in order to give its final response.

[01:07:36]Okay, great.

[01:07:38]And one other thing I will say is all of this that we see here is what you have implemented.

[01:07:47]You will not see all of this. Uh you know if you are an LLM, you will not see all of this. All you care about as an LLM is the function API, the input and output as well as the main lines of documentations. So all these implementation details, you're going to have it on your uh codebase, but the LLM is not going to see it.

[01:08:12]Okay, great. So now let's go u step by step into how we could make make this work.

[01:08:20]So uh the first stage is uh as you ask a question that is related to your function, you would insert at the beginning of your preamble the function API and as I mentioned without its implementation. So you just have u like the function itself and then a full documentation.

[01:08:43]So here uh you would want the LLM based on your user query to feed the right arguments to your function. So to your point the goal of the LLM here is not going to be to infer any of the functions implementations rather only what arguments we should put to it. Yep.

[01:09:14]Yep.

[01:09:18]Yep.

[01:09:20]So the question is how do you even train this? So we're going to see uh that in just a few slides. Yeah. So great point.

[01:09:25]That's the next question u you know we need to ask ourselves. Yeah.

[01:09:33]Yeah. So we're going to see how we train this. Um so but here um let's say it has been trained uh the LLM would have from its context probably our localization because let's say you have activated your uh you know location permissions and your LLM knows where you are so it knows you are at Stanford and these would be the coordinates being fed to your LLM and then the second stage is to actually do that function call. So nothing it has nothing to do with LLMs. You just take your function your argument and you execute that and you get some answer and the answer as we mentioned is structured in a way that's understandable. So in that function implementation you return some object that informs characteristics about the return teddy bear. So for example you have its name, maybe location and so on.

[01:10:35]and you feed that response back to the LLM in order to get a final response. So when you ask to the LLM a given question, you don't want to have this uh you know JSON like response but rather a response in natural language. And this exactly is the motivation for that last stage.

[01:10:57]So, I'm going to pause here for a second and check that this three-stage mechanism makes sense to everyone.

[01:11:08]Yeah. Okay. Great.

[01:11:12]And and exactly what you were mentioning here is going to be the next focus. How do you even train that? So, okay. So, let me ask you this question.

[01:11:22]If you were to train an LLM to use this tool, what steps would you need to focus on?

[01:11:36]Right? So the answer is you're going to feed the API implementation. Yes.

[01:11:42]So you would need the first LLM call to be somehow recognizing the pattern of the function implementation and the query and link it to the um the arguments that you would put in your function. So yeah, great. So tool prediction and do you need a second set of SFT pairs?

[01:12:11]So you have this last stage that is still LLM driven where you have your tool answer and you need to output a final response.

[01:12:21]So you might say, okay, hey, the LLM has seen a bunch of structured data and knows how to put into words things that it sees, which is a fair point, but usually you might want your responses to be formatted a given way. So you might want to also have SFT pairs that do this mapping the way you want it to.

[01:12:44]So this is why you typically have these two SFT pairs. And if I have to be a bit more precise, the second pair isn't just mapping the JSON response to the final response. It's actually linking all the conversation history so far. So that it knows that the initial query was someone in search of a teddy bear. It knows there has been a tool call and it knows that the results correspond to that tool call. So it would be a slightly longer input uh in this SFT pair.

[01:13:18]Does that make sense?

[01:13:22]Okay, great. Yep.

[01:13:29]Yeah. Uh so great point. So the question is what if we have more tools? Do we need like some tool selection or like more examples? So this is a great topic.

[01:13:38]We're going to see it uh like in in in a few slides. Yeah. So it's slightly more complex way of doing things. But if you want a very quick answer, if you go the SFT way, you could show multi- uh tool kinds of inputs. So you could include all of that in your SFT data sets. But we're going to see the topic of tool selection very soon.

[01:14:03]Great. Any other questions?

[01:14:08]Okay. Awesome. So uh so this is what I just mentioned. Yeah, the conversation history so far is always what you get as part of the inputs and then the output is what you would want the LLM to predict at that given stage. And since you're doing SFT here, you don't have just one but multiple such examples and you would want your examples to be varied and representing the typical user distribution.

[01:14:36]So I was asking find a bear near me and this example showed to the model how you could ground the information of location based on the user's location even though you didn't give it. And in these other examples, you could give uh the model other kinds of instructions like directly saying uh I want a bear at that location to teach the model to look at like to ground the argument at different places. And you could expand this set by also uh varying the kind of input so it doesn't have to be worried that way. You could do it multi-turn maybe in the middle of a conversation you ask for a bear and so on. Yeah.

[01:15:21]Okay, great. But this is not the only way of uh of training a model to do so.

[01:15:28]So these days, LLMs become more and more powerful in their reasoning. And the kind of data they are trained at pre-training and initial instruction tuning is typically these kinds of code data. So at the end of it, they know how to manipulate Python codes very well. So you might ask yourself, is it really needed for me to teach the model how to map a query to a function to a function call? And that is a very uh interesting observation.

[01:16:04]And you see these days that you can forgo specific SFT training and try to get around it with only training.

[01:16:15]So here instead of um you know writing SFT and then retraining the model, we're going to see that you could actually replace it with only an explanation.

[01:16:30]And does anyone have an idea of how we could even come up with such an explanation?

[01:16:38]So let's say I have a new tool. I have the API. I want my model to use it.

[01:16:45]How would you go around it?

[01:16:52]Yep, there's a great point. So, one method could be fshot learning. You just uh show in the context window um samples of input output. This this is a great point. You could definitely do that. And this is typically one accepted practice.

[01:17:08]Um but if I if I told you that fshot learning has um challenges when it comes to generalization because you would need to give specific points as input output.

[01:17:21]So it might fit to some cases and not necessarily generalize to the whole span of human language. Um you know is there another way you you could go around it?

[01:17:39]Yeah. So the the answer is ask it to do reasoning. You know, great points and writing a prompt that does the reasoning in a way that makes sense is very hard.

[01:17:52]So in practice, you wouldn't write it yourself. You would take these SFT pairs. Okay, so these are the behavior that you want to enforce and you could use it as some evaluation set. So you could say okay if I do if I ask this question I want that tool call and you have a set of pairs and you could run whatever you have so far in terms of explanation.

[01:18:22]You evaluate these prompts against the evaluation set. So you have some wins and losses. Maybe find a bear in Paris doesn't work well. Some other prompts do. So you have a list of um of like each sample with like a score and you could feed it back to a reasoning model typically to do the explanation writing for you. So yeah, this is a trick to you know help you avoid doing that hard work. basically showing the model, you know, hey, with the current prompt, here's what we get. What would you have changed in order to make the evaluation results better? So, you get with that process an iteration on the detailed explanation. And if you have to see results in practice, you would be surprised at how well it does the explanation for you. So, the takeaway here is I do not recommend you write it end to end. maybe just a draft and you let some very powerful model with excellent knowledge of uh logic do it for you.

[01:19:36]Yep.

[01:19:51]>> Yep. So the question is what you just mentioned is it training or inference.

[01:19:55]So it's training. So you do it at the very beginning. You want a fixed prompt that explains to the LLM how to use it, how to use that function. So you would typically do that offline. offline you iterate on the explanation that says exactly how you use the prompt uh how you use the function and then at inference time you would put that fixed explanation alongside the function API such that for any query it knows what to do does that make sense okay great any other questions here okay amazing So I just mentioned an example with teddy bears that would be in the category of maybeformational.

[01:20:46]So you have um so you have a question and you want to ask some external API to retrieve that and you have in reality a lot more use cases that you might see.

[01:20:57]So when I've mentioned you have some cutoff dates and let's say you ask your LLM about news of the day. So you typically wouldn't have anything that comes from the model itself. You would have an API that fetches it from a tool called maybe search, but you have also other kinds of tools available in in the information category. So for example, weather, stocks, and um let's see um you know, but you also have other categories. For example, if you want to ask the model to do some calculation for you, you could let the model figure it out with some reasoning chain. But one way to get around it is to transform the query into code, execute that code and then read out the answer. So this is why uh like tool call has a whole span of applications in the computation.

[01:21:59]um area. So you have calculations and uh you have another category that we can site here. So you can take actions on behalf of the user. So let's say you have a tool that sends emails. You could ask the model to send an email for you and it could um put the right things into uh the header and the message body and even hit send for you because you have that component that interacts with the external world as part of the as part of the tool API.

[01:22:33]Um yeah so these are just a few examples but I just want to say that uh the field of tool calling is so powerful you can do anything you want and in practice exactly as you mentioned you wouldn't have just a single tool API as part of the uh context you would have several ones because your LLM is not just an LLM for finding bears right you might want to do other things with your LM maybe you want to hug a teddy bear or check a teddy bear's moods, send teddy bears a gift, you know, and other things. So, you have a lot more functions that might be relevant to be added as a preamble just because you don't know what function would need to be used. You just need to put all of that just in case.

[01:23:26]So the setup is exactly the one that you mentioned where you don't have just one API but multiple ones.

[01:23:35]Um does anyone see any issues by do with doing so?

[01:23:43]Yeah. So the suggestion is that that's a lot of tools. Yeah. And then you're right. So typically if you have too many tools you have the problem that Afin mentioned at the beginning of a needle in a haststack where maybe some tool API will get lost in the context and you don't really know what to use you have a lot of conflicting APIs maybe and we're going to see very soon how to um kind of overcome that issue.

[01:24:11]Okay, great.

[01:24:13]So I want to take a pause here and summarize how far we've come and some of the drawbacks that we that we have and see together what we could do to remedy these drawbacks.

[01:24:27]So, first we saw that we came from an LLM that just responded back at you with a normal words to an LLM that can actually interact with the outside world, fetch real-time information or even extend computing capabilities um and uh and exactly like overcomes the issue that Afin was mentioning the knowledge cut off one in a different way that than what rag would do. So you could see these two methods as being complimentary and yeah both try to do the same thing in some sense but uh exactly as someone mentioned here uh if you have more tools then your context window might have things that it doesn't needs and um and then if you try to support so many cases you might end up being mediocre at all of them.

[01:25:27]So this is an issue and even if that wasn't an issue, you still have the context window that is finite. So let's say you have uh hundreds of millions of users that use your LLM and that want to you to do a lot of things. You cannot support everyone's use cases at once just because you couldn't possibly fit all of such tools in your context.

[01:25:54]Yeah.

[01:25:55]And then uh last when I mention these tools, these are things I hand wrote and maybe each LLM has their own way of defining and using tools. We're going to see if there is a way to standardize maybe the way we define and use tools uh later on. So do these drawbacks make sense?

[01:26:19]Yeah. Okay, great.

[01:26:22]So let's move on to tool selection. Uh so let's see what we could do in order to make um tool use more scalable. So I'm going to site uh here um a technical paper from Google deep minds uh which uses a tool selector system. So it functions in two steps. So first you have your query and you have a list of tools that can be as big as you want and that list of tools only contains the API name and maybe one or two words about what it does.

[01:27:03]So you have uh like this this prompt and the list of tools and we ask the LLM to pick the tools that might be relevant.

[01:27:15]So this is why the technical paper calls this system a tool selector. You could even see the term router um like being pronounced in the literature. So tool selection routing are similar concepts.

[01:27:31]And the goal here is to restrict the number of tools to only those that might be useful. And in the second stage, you take all the selected tool APIs and feed only those to the context alongside your query. And this is uh one way you could take to overcome this um this this issue of having too many tools at once.

[01:28:01]Is everyone convinced that tool selection could be a good way to to fix the problem here?

[01:28:08]Yep.

[01:28:14]Yeah. So, the question is, is that basically rag? You could do it with rag, you know, that's a great point, but it doesn't necessarily have to. So, you could uh have uh an LLM that does this job just uh picking the right the right tools and you ask some instructions to the LLM to output the right answer. You could definitely do it with drag. U that that is just a great point. Yeah.

[01:28:40]Yep. So, does it make sense to everyone?

[01:28:44]Okay, great. And I see time is flying.

[01:28:47]So, we're going to go very quickly on the standardization uh issue. So as I mentioned every tools implementation like the way you specify a tool implementation can be bespoke to a given LLM and you wouldn't want that because for every LLM you might need to implement always the like these tools like over and over again and duplication is not what you want. This is why there is a standard called MCP that I know um no know a lot of people talk about it.

[01:29:24]Uh so so it's something from the anthropic team that's uh like standardizes the way tools are exposed to models. So it's a protocol. It's called the model context protocol and it defines um you know standard way of presenting these tools. So I'm going just to say the very uh big lines of it where uh so you have this kind of vocabulary as part of MCP. So you have u an MCP server which is the instance that serves tools. Okay. And then tools are implementations of the functions that you want people to use. Prompts are sorts of templates that could show the user how to use these tools. And then all of these can anchor on resources which are external um sort of databases that you can use to complete the task. And then an MCP server when you use it has a onetoone connection with a piece of infrastructure in your LLM host that is called MCP clients. But these are maybe infrastructure details. Uh and I just want to ground what I just mentioned in reality. So we know our teddy bear loves to read poetry. Um so in the case of recommending a poetry book to to our teddy bear, you could think of a MCP server that provides tools that are linked to uh you know books. So your LLM host here. So let's just take the example of cloud since you know MCP is from anthropic. So cloud would be your LLM host. Your MCP server would be probably implemented by your book provider because typically they are the most experts as at serving such content.

[01:31:20]And we could assume that the book provider MCP server has tools regarding finding books or recommending them. And then the prompts here could be a ways to show you how to find a given title or maybe recommend it with respect to like some flavor maybe with respect to the users test taste and some example of resources here could be the teddy bears personal collection or maybe top books that people buy.

[01:31:52]Yeah, just just as an example.

[01:31:55]Okay, awesome. So now I'm going to come to potentially the most exciting part of today today's lecture which is agents.

[01:32:05]So we saw how much more powerful LLMs could become with tools.

[01:32:11]Agents could be seen as one layer up from it. So I'm going to give first uh definition just so that we agree uh of like what agent means here. So it's a system that autonomously pursues goal and completes tasks on a user's behalf.

[01:32:32]So compared to tools, you not only can perform like tasks uh but you have also some reasoning uh involved in it. So you could have like multiple loops of iterations and this is what different differentiates an agent from a tool. um in plain terms when usually people talk about agents it has that uh recurrence or like higher level of reasoning baked into it.

[01:33:04]And so I'm going to contrast like the agentic world with respect to the world we have seen where um you know you may have multiple calls to tools and this new agentic framework is not necessarily disjoint from the ones that we presented before. You could well have reasoning chains in inside of it. So it's like a potentially overlapping but the structure is uh consists of like tool calls and iterations.

[01:33:41]Okay, great. And now I'm going to talk about a hallmark paper called um react. So reason plus act which decomposes possible loops into different stages.

[01:33:58]So very often times when you have a query and you want to pursue a goal, you cannot do it one shot. You need somehow to decompose the goal into actionable substeps, perform each of these substeps and then come up with the answer. And this is exactly what React is about. It's about decomposing um complex tasks into uh into loops of things that can be done atomically.

[01:34:31]So here I just want to say one thing. We decompose these steps into observe, plan and act. Uh but it doesn't necessarily have to be called that way or be in that order. For example, the react paper I think introduces terms of think, observe, act. So yeah, you might see the language changing a bit from paper to paper, but the high level intuition remains.

[01:35:02]So let's just ground on a very specific example. So to be in the theme of these days weather uh starts becoming a bit cold and our teddy bear might be cold in its home. So let's just have that as an input. Uh you know my teddy bear is called please do something and let's see what uh an agentic workflow could do about it.

[01:35:28]So first you have this observe stage which translates the user query into an actionable formulation. So when you say that your teddy bear is cold and you ask it to do something uh like here in this observe stage you would link it back to the notion of temperature. Uh you know the user teddy bear is cold which may be due to the current temperature of the room which is currently unknown. So right off the bat, you now know that you need to do something with temperatures.

[01:36:03]So this brings us to the next step called plan where you know that something is unknown and you need to find it. Plan will spell that out for you. So here you want to determine the temperature of the room. And luckily among your tools you might have something that does something in these lines and this is where you use it. So the act stage is using all these APIs that might be in the context. So for example here if you have get current room temperature this is the way to use it. And as I mentioned these tools are useful when you to the LLM when you output information back to it that it can interpret. So this tool returns the temperature and you need now to interpret what the temperature means. So this is why you go back to the observe stage. You you do um like you you describe what the world looks like and what you should do about it. So here uh you can read that the temperature of the room is cold. 65 fah and the observe stage might indicate that it's colder than expected. So this is why you need to plan something and then at the plan stage you're like you know this this was colder than expected so I need to increase the temperature and then you go back to the act stage.

[01:37:32]Okay, luckily I know how to increase the temperature and it has a a function that can have uh the temperature increase arguments inputed to it. So you can adjust the temperature and then uh once it's done the observe stage um concludes that the temperature is not set at the like to the correct temperature and this is the time where you can exit the loop and return the response to the user.

[01:38:06]So here we have increased the temperature by 5° and then the output reads that back to the user in the hope that it has fulfilled the user's query.

[01:38:18]So what I described within the loop is what makes a workflow an agentic one. So you have some initial query, you have some actions that you can perform and then at each stage the LLM tries to see if it has reached the goal yet. Uh and then if it has then it will go to the output stage otherwise it may have more reasoning loop where uh where it does uh more work.

[01:38:51]So does the definition of agent that I propose here make sense?

[01:39:00]Okay, great. So now taking this a bit further, you could think of way more than one agent. You could have an agent for setting the thermostat. Uh but maybe in your home you want to manage how energy is distributed or uh you know air quality if you have some some settings to tune. So you can have different agents and one interesting use case is to have the user say something to an agent and potentially have all of them communicate together.

[01:39:33]So um this prompts uh potential needs for also a standardization of communications with uh between agents similarly to what we have seen between LLM hosts and tools and this is what prompted Google to release the agentto agent protocol earlier this year. So uh yeah, I highly recommend taking a look at their uh document specification but just to give the main lines around it um you you have some standardization of what an agent can expose. So it typically exposes a set of skills. So it can do things and um and then you know it gives examples about it so that other agents are aware of it and what you need to do as a developer would be to define these skills and also another key thing is to define how um the agent executes a given request. So for example when you execute a given query what is the status that you emit to other agents or there is a cancel method as well where let's say an agent says you know stop what you're doing what is the process to cancel an action. So yeah these are uh kind of some of the main functions that are that that people like the agentto agent protocol asks people to fill out.

[01:41:04]Yep.

[01:41:14]So the the question is uh is each agent an LLM with some context to do some tasks. So you could definitely you know imagine it you know be the case. Yeah.

[01:41:25]So yeah typically in the example that I mentioned here. Yes.

[01:41:29]And the agents operate independently. So they have their own reasoning loops and all the other agency is the input and output.

[01:41:52]>> Yep. So the kind of so the remark here is maybe your token budget can go you know uh all over the place. So you can have some uh budget restrictions you know put in place you know this is a good point but each of these agents they would not eat on each other's budget you know it might eat on your money budget but you know yeah this is indeed the concern okay so I'm going to move uh on to the topic of safety which I I have not mentioned so far but that is very important uh because with these new capabilities comes a whole string of new potential issues and you have seen um these models they have now the ability to execute actions for you. So you could think of some harmful actor doing things that you wouldn't want for you and I give one example here that could be a concern data exfiltration.

[01:42:51]So let's say you have access to a tool that can write in a public visible way um you know data. So let's say you have I don't know some email agent.

[01:43:04]Um so if you have a prompt that for example says you know write my password which potentially the tool could have access to to an email to that address you could exfiltrate data that is that belongs to the user out of it. So this is one typically one risk that you might have. You can have other safety risks and I linked a paper here that goes through some of them. So a tool sword.

[01:43:32]So I recommend a read and uh you know now you might ask you know what could you do to get around it.

[01:43:39]So you have typically two classes of remediations. one that might come at the training stage where um if you recall uh the training process of R1 or even like other um other models you have this harmlessness component. So you typically have data as part of the data mixtures that you train during SFT and and uh reinforcement learning that could cover uh safety. So this is where you could uh you know remedy these issues. You have you know another uh option let's say you have a query that has gone through your lines of defenses your training lines of defenses. You can also have inference uh safeguards for example a safety classifier that looks at the conversation so far and that judges whether the output of the LLM is going to be safe or not. So you have even that as a safeguard and uh just as a pointer I'm going to uh you know talk about like agent safety bench which uh summarizes the span of possible safety hazard hazards and offers a benchmark a full suit for it. So yeah people can refer to these benchmarks to know whether their LLM is safe.

[01:45:07]Yeah. And it's a very important topic and just yesterday there was um like anthropic revealed that they were the um like victims of a large scale cyber attack launched from cloud. So this is an issue that is uh like so it was also using like tools and agent capabilities and they published super detailed report saying exactly what the uh attackers did and went step by step through possible lines of remediations. And this is just to emphasize how important safety is in this more capable world. And both attackers and the line of defense can be you know more and more sophisticated. So it's not a lost battle. It's just that the you know tools we have for defending against um like tool you know tool based attacks you know need to be so we need to have like such measures in place and with these measures maybe it's going to be fine. So I think that was like the overall mindset of that article which I strongly recommend to read.

[01:46:19]So I'm going to say a last few words. So when you talk about agents, you have um you know this this risk that at every step of your uh thought process, you might diverge to something that just doesn't work. So that is a huge problem.

[01:46:37]Let's say if the model doesn't ground to the output properly or maybe does a mistake into that argument prediction of a tool call. So this is one big issue.

[01:46:49]uh which is the reason why you don't see uh you know large scale agents ruling the world right now because uh it's it's really like we are limited by these consequences.

[01:47:00]Um and then uh I'm going to talk about like the developing capabilities of models that enable you to have this agentic capabilities. So this is something that you can uh like like fix with SFT but uh ideally you wouldn't want to use SFT to fix reasoning uh gaps and use the model itself.

[01:47:24]And we're going to see uh next week uh what the evaluation landscape looks like. So I'm going to reserve it to next week. And just a few words of advice regarding building tools or building agents. Always start small on a very simple case. For example, find the nearest pair. Try to see if the your current implementation and prompt just works and then start from there. So start small and then start smart. So take the the like most capable model first so that you know the headroom of where you can go with the current models and then try to uh gain on latency capabilities and so on. So yeah, start correct, start small and then you can optimize later.

[01:48:15]And uh when it comes to debugabilility, these LLMs, they output chains of reasoning. So it's always good to look at them to see what what is going wrong.

[01:48:25]And I will close this lecture on a note.

[01:48:29]You know what's you know my favorite use case of using agents right now is uh as AI assistant is assistant coding. So this is something I strongly recommend in case you have a projects that require you to do you know complex piping of things you can free your mental load by delegating some of these tasks and this is something that is you're going to see in the real world very much used now uh but with the caveat that you know please make sure to learn the foundations of code know how to code right because your taste is going to matter most from now on you know generating code is cheap, but judging whether a code is correct and does the right thing. This is the hard part. And with that, thank you.