RAG Evaluation with RAGAS: Advanced Retrieval Techniques

[00:00:00]welcome back to our next session on rag evaluation everybody asks how do I evaluate rag systems here's your answer so today we're going to align our aim to evaluate rag systems using best practice open source tooling and improve rag systems quantitatively using some Advanced retrieval processes quick overview of what we're going through we're going to review rag build a meta rag assess some rag take R to the next level in rag we're going to do rag yes couple of links for you guys you can check these out right now bit. le amrag tinyurl.com eval rag these are the things that uh we've already shown some of today we're going to keep showing the rest right now so just recall open source rag what we're talking about query embedding model Vector DB prompt template go ahead and put that context in there everything gets shoved into the chat model boom you got your answer okay cool now we have this if we focus on the models we're using for open source we talked about leveraging BG base and we've seen how we can leverage new researches LOM 7B chat model from hugging face by doing a little bit of setup work at the beginning so we basically have the setup BG base llama 2 from new research chat model 7B on hugging face so we're going to set a quick set up a quick flow where we're going to do rag on rag AKA meta rag we're going to search for the top five papers on rag we're going to convert those papers to embeddings put them in a vector store ask specific questions related to the content and return answers we've already seen this before looks something like this but we want to actually be able to ask now what is Rag and we want the rag system to answer it so with that let's set up our metag over to you Chris all right we are going to meta rag so this is where we're going to rag on rag as Greg said uh straightforward enough it's exactly what we've seen before but using some uh you know different abstractions so first we're going to grab our dependencies next we are going to provide our open AI uh API key and then we are going to load our archive loader from our document loaders we've seen this before it lets us query archive for papers that match this qu uh this query by relevance we're going to load five of them and there we can see we have these five papers then we're going to create a naive index where we chunk those papers by with the recursive character Tech splitter with chunk size 500 we're going to split them there and then we're going to store them in a chroma Vector DB using open AI embeddings so you can see that we have 1,000 chunks uh from our five papers that are stored in our index then we're going to create a QA from retrieval chain which is going to be this retrieval QA from chain type we're going to give our primary QA llm which is just going to be gbt 3.5 turbo 16k we're going to use our chroma DB Vector store as a retriever we're going to pass in these search quars which is going to let us select our K to be three which means that every query query will retrieve the top three of our 1,61 documents and we're going to have them return Source documents then we can ask the question what is rag we can send that in as our query and we'll get a result rag stands for retrieval augmented generation it is a framework that combines retrieval based models with generation but I know goes on and on to explain ra rag that's all we have to do um you know this is the this is the whole thing uh you can access the notebook with the bit. Le that Greg mentioned or you can check it out here uh but yes that's all we need to do in order to do rag on rag we'll kick it back to Greg nice rag on rag what's next well how did we do let's talk about how we can evaluate this rag system now when we think about evaluation we should be thinking about evaluation feedback loops we should be thinking thinking about how we're actually going to improve our system as we continue to do more W and as we sit there now that we have it instrumented what should we be measuring what should we be trying how should we be tinkering and tuning this thing as we go well the answer isn't clear for any given application space the method for baselining how good your system system is at least relative to how good it might be after you start changing stuff that is more clear when we do rag evaluation with the ragus or rag assessment framework we want to think about it by breaking it up into four different pieces the first is the question that's the query what are we asking this thing the second is the answer what's a given back the third is the retrieved context that's the context that we pulled out of the vector DB and we passed in to the llm through the prompt finally the ground truth here is very important the ground truth is just as the name would imply by the ground truth answer now it's important to note here that when we talk about ground truth answers those are generally going to truly be something that probably should come from a human what should the right answer be that is good correct or enough for this particular query given all of this context what would a human respond with given question and access to all this data that might be the real ground truth but for us and because nobody wants to create these painstakingly terrible ground truth data sets they want a system that's automated to do this for them so we picked the best model out there we pick gbt 4 let's assume gbt 4 knows the truth so we can get a handle on the rag evaluation pipeline ideally though humans know the truth better of course we have retrieval and generation in a rag Pipeline and we can break it up into thinking about retrieval and generation retrieval breaks down into two different metrics the first metric is called context Precision answers the question question how relevant is the context to the question right context recall at answers the question is the retriever able to retrieve all relevant context and you might notice we're dealing with precision and recall classic ml practitioners should probably feel pretty comfortable with this now on the generation side there's a couple of other metrics we'll take a closer look at each of these four metrics in the subsequent slides during generation we're going to compute two things the first is answer relevancy answers the question how relevant is the answer to the question right better be pretty relevant because otherwise is definitely wrong and the faithfulness answer this is the fake news metric right this is is the answer fact checkable is this thing hallucinating is this thing confabulating is this thing something I can trust no fake news is it faithful right sticking with faithfulness here's an example to make it more Concrete in your mind and this is how faithfulness score is calculated number of claims that can be inferred from given context divided by total number of claims in the generated answer so here's an example question where and where and where was Einstein born I believe that was a typo in documentation from the ragus guys where and when was Einstein born context Albert Einstein born 14 March 1879 was a German born theoretical physicist widely held to be one of the greatest great thank you great context High faithfulness answer Einstein was born in Germany got it 14 March 1879 got it low faithfulness it lies to us right easy enough of course this is an open source tool and you know um more help with docs is something that we'd love all of our community members to support for all of these great tools that come out and that uh certainly the small teams that create them need help with so the answer relevancy metric here we're going to measure how relevant the the generated answer is to the input to the prompt and it's important to understand that it doesn't consider factuality per se but instead it penalizes cases where the answer lacks completeness or contains redundant details so where is France and what is its capital low relevance France is in Western Europe right it's like talking to one of those people that's like answers half of your question kind of like we did to Manny earlier LOL those are some epic questions though High relevance answer answers both pieces France is in Western Europe and Paris is its capital right easy enough moving on at a high level here context Precision is measuring the relevancy of the retrieved context to The Prompt now check this out we've got Precision at K this is true positive over true positive plus false positive classic Precision where context Precision is now divided by total number of relevant items in the top K results so it's a metric that evaluates whether all all of the ground truth relevant items are presented in the contexts and they're ranked accordingly high if they're more and more relevant so ideally all the relevant chunks are in the top ranks right the more relevant the higher it should be ranked makes sense it's exactly the way you would write a paper you would start it off with the most relevant things at the front end okay what about context recall context recall is measuring recall of the retrieved context here it's the ground truth sentences that can be attributed to the context divided by the total number of sentences in the ground truth right so the idea is if it's super true we definitely want it in the ground truth and we want to be able to retrieve it during retrieval of reference material question where is France and what is its capital ground truth France is in Western Europe and its capital is Paris a high context recall might look like France and Western Europe and compasses medieval cities Alpine Villages and Mediterranean beaches Paris its capital is famed for its fashion houses classical art museums including the Lou and monuments like the Eiffel Tower low context what are we doing again here France and Western Europe encompasses medieval cities blah BL BL Alpine Villages good good good the country is also renowned B BL BL alasow again it's blowing off the capital right it's not giving me a necessarily relevant sentence here it's providing context but it's not necessarily super relevant context and so we want to see only again the most relevant context so this is measuring the extent to which the retrieved context aligns with the annotated answer aka the ground truth and context Precision is going to measure the relevancy it used to be called context relevancy but they change the name to context precision and this context we already covered context Precision this is a duplicate slide so from context precision and context [Applause] recall yeah we're going to go ahead and mute that one context Precision context recall our retriever Y Man and faithfulness and answer relevancy are generator metrics so the reval system overall you might consider as being measurable by context precision and context recall the generator metrics you might use to sort of get an idea of how your Generations are improving how much they're hallucinating but realistically what's the whole big idea of rag it's get references find relevant ones shove them into the prompt context and improve generation so so it's probably more valuable to try to improve improve your retriever to thus improve Generations so if you're going to aim at something we would recommend aiming at the retriever which is what we'll show you how to do today some examples here of context this is in their docs as well context relevancy is now context Precision you can see these numbers um how they look associated with any given type of question answer contexts and ground truth so you can get some idea of the way that this kind of Might improve the answer with better contexts you might be able to get yeah I guess we're gonna have to remove person um all right so in the rag evaluation pipeline you're essentially talking about four different metrics context Precision context recall answer relevancy faithfulness for this particular demonstration we've got today we're using gbt 3.5 turbo to generate an answer and we're using GPT for to generate the ground truth and this whole setup we're going to show you the numbers before we close up for the day but we're going to show you how to get this set up and how to get some initial Baseline numbers which is going to be the most important first step to take uh right now as we do rag on rag with ragus Chris over to you man that's right right we are we are doing like rag nesting dolls here so we're going to do ragus on rag with rag it's going to be crazy first step though uh we need to you know create a a data set of some kind so what we're going to do is we're going to use our uh response schema output parser as well as our structured output parser from Lang chain basically this is just going to get us a question in an object with the key question the description is going to be a question about the context and then we're going to save that into our list of question schemas you'll notice that we have this structured output parser and then we have a uh get format instructions now what this is going to do is it's going to pass gbt 4 a prompt that's going to encourage it to Output our response in the above requested schema so this is a useful pattern like outside of even uh you know this particular case if you want your responses to conform to a specific schema now uh we're going to pass in this prompt along with some context as our QA template we're going to say you're a university Professor creating a test for advanced students you know create some questions the idea here is that in order to evaluate we need to provide these Triplets of questions answers and contexts and so we're gonna have the llm generate our questions from our contexts and then provide answers to the questions from those contexts so we'll start the process by doing this uh right here making this chat template adding those format instructions and then sending that to the llm notice that we have the question what is the main focus of the paper a survey on retrieval augmented text generation and the context is a blurb from the paper that explains and contains the abstract of that title so the idea is we have a piece of context and now we have a question that should be answerable given that context we're going to do this for 10 prompts because uh to do this for it's going to cost many tokens to do this so please be aware of costs uh do it on a small subset is going to be an okay place to get started for uh evaluation uh if you want to take this further obviously you'll have to talk to your organization what I can tell you is that it will cost a bit but it will cost much less than getting humans to do a similar task so uh that's always great next we repeat this exact same process but for answers you'll notice you we have now you are a university Professor creating a test for advanced students for each question in context create an answer we're going to pass in our users questions our uh our context that we're used to create those questions and then we're going to get the llm to produce an answer built from the question and context pair again we're going to uh look at an example the main focus of the paper a survey and retrieval augumented text orations contct a survey but retrieval Aug it goes on and on but the idea is that this answer is an answer to the question we created earlier with the provided context and that's all we're doing we again create 10 answers to our questions and then we wrap this all into a data set now that we have the data set we are able to start thinking about using ragus so the way that we're going to do this is by having this create ragus data set option because remember our answers that we have are considered the ground truth answers but we need to evaluate our lm's answers to these questions right so we have our ground truth we need to have our llm produce the actual answers so we're going to use the uh create ragus data set to do this you'll see the rag pipeline is going to take the query at question and then we're going to add the answer in our data set uh from our actual llm so this is the response generated from the model that we're evaluating basic idea here we're going to have our model get the same context and the same question and see how it does we're going to compare the two results at the end to get our ragus metrics so here's our evaluate ragus data set you'll see here that we have uh a few metrics we care about that Greg has discussed with us context Precision faithfulness answer relevancy and context recall also one thing to notice we're passing in the context that we obtained so we're not just evaluating the model's ability to generate correct or factual responses we're also evaluating the performance of our actual retrieval pipeline is it retrieving the correct contexts are the contexts relevant to the answer and the query and so on so let's uh lo you know we just create that data set it's got 10 rows and then we save it for later and then we evaluate how it did it goes through a whole process it does take a while uh it's got to answer 10 questions and then evaluate four metrics for each of them and you'll see that we get this basic QA result which has a full ragus score of 0.22 a context relevancy score of 0.07 a faithfulness score of 0.6 second an answer relevancy of 0.99 that's high that's great and then a context recall of 0.866 7 so we have a few different things that we can shift around here but we want to focus on improving you know kind of a metric at a time and the metric that we want to improve is we want to corre collect better context for our model so we're going to use some uh different retrievers and see how that impacts our ragus score so this retriever is just going to be our uh kind of like our QA chain Factory right we're going to pass in our chat open AI model we're going to hold the llm to be the same and then the only thing we're going to change is this retriever between the different pipelines right so we can plug in new retrievers but everything else should remain constant since we're trying to test our retriever specifically the first kind of like fancy retriever we're going to use is the parent document retriever now the way this works is imagine that you have a page and on that page there are five pair paragraphs the way the parent document retriever is going to work is it will search through each paragraph present in your Corpus but it's going to return the full page so why might this be useful well let's say for instance this is an example that someone from our uh llmo Cort uh worked on right which is this idea of like an equation right if I search for Bern's equation and I just rely on the text I'm not going to get br's equation I'm going to get a paragraph that describes that br's equation is either coming up or we just saw it so the way that we improve that is we expand the window around what we find in order to in order to better capture relevant information so when we search for br's equation along with the paragraph that explains that we either just saw it or are about to see it we also capture the equation and we're able to ensure that we capture more information and obviously this extends to many other examples but that's just a you know one that's that's uh top of mind so you'll notice that our parent uh documents are in chunk size of 2000s and our uh child documents are in chunk sizes of 400 these are hyperparameters that you can play with um so you know you might find it to be better to even search even smaller chunks uh and then blow up uh a little bit less or blow up even more it's all about uh what's easiest for you and I'll I'll link this again uh for people who are just joining the idea is we can use laying chain though to simplify the actual creation of this so we create our parent splitter we create our child splitter we create our Vector store with the collection name split parents and our open AI embedding function and then we create an inmemory store now one thing you might have noticed if we're only searching for the child documents using d retrieval then we shouldn't actually need to uh embed the parent documents right we we never search directly for the parent documents so indeed all we need to do is store them in memory and that's why we use the inmemory store for our parent documents as you can see here we have our doc store which is our inmemory store our Vector store which is our chroma DB and our child and parent splitter our child documents will wind up in the vector store and our parent documents will wind up in the doc store they will be Associated behind the scenes so each child document is associated with a uh parent document all we have to do now is feed in our docks and it's going to take care of the rest and now we can create a new chain again we're going to use the create QA chain uh helper function that we built above and we're going to pass in this specific Retriever and that's all we have to do now we can ask a question what is Rag and we get rag stands for augmented generation it is and it goes on and on it gives the answer that's good so now let's use this new chain with the parent document Retriever and let's evaluate it using ragus and we'll find that our context relevancy score is perhaps uh you know uh leav something to be desired our faithfulness is quite High our answer relevancy is relatively untouched and our context recall has improved some the next uh retrieval method we're going to try right because we want to see which is the best of these methods uh is The Ensemble retriever now we've talked a lot today about dense Vector search which is this idea of comparing dense embedding vectors of passages of text and determining their semantic relatedness using some distance measure like cosine similarity well instead uh of that we're going to combine that dense Vector search with with a sparse uh search so we're going to use the bm25 search method which is much closer to a bag of words style uh search so it's more caring about what words are in the uh in the query versus what words are in our documents the idea here is that we collect a number of documents from both of the search methods and then we rerank them using the uh reciprocal rank Fusion algorithm uh which is you can read about in in the paper it's pretty cool the idea is it's going to fuse these two disparate uh relevancy searches into one ordered uh uh entity so that we have some you know top K results if we retrieved say in this case four but we're only actually going to keep three it would order the results by relevancy such that we dro the least relevant between the two um it's just a cool algorithm to combine two different uh collection methods you'll notice that there's this weights hyperparameter so we can actually prefer one or the other so if we know that we need some level of keyword es search or sparse search but we don't want to weit it too highly we can shift this to 0.25 and 0.75 as long as these numbers sum to one you're H you're chilling the idea here is that we're going to do these two search methods and then combine their resulting uh documents in into one ordered list and then choose from the top of that and again we ask the same question and we get another great answer feels good we're going to set up the same ragus data set that we did before but this time using the Ensemble retriever we're going to uh essentially mark this as it goes through and then we're going to go and look at our total results in order and we can kind of see that generally our basic QA result while it was quite decent on uh some of the metrics you know our the faithfulness kind of fell behind uh the answer relevancy kind of fell behind and with our more advanced methods we were able to uh boost our faithfulness and our answer relevancy we didn't take too much of a hit on context recall but we did take a hit on context relevancy and so what this means is while we are retrieving more and more uh more documentation that makes our answer uh makes it so that we're answering closer to the source documentation we are in fact retrieving uh more information than we necessarily need to answer the question and so that's a Improvement we could seek to make in the next iteration right perhaps we could tweak the hyperparameters relating to how many documents we're retrieving with each pipeline but overall we're happy to see that our system in some capacity improved and gave us better results than just the basic retrieval method which is fantastic and this is the idea of ragus it helps us to understand how changes we're making to our retrieval uh and generation pipelines actually impact our uh our end results uh now again this is all this is all by AI right so AI is answering the questions and asking the questions obviously you could substitute humans and any part of this uh if you have domain experts you could have them generate uh these questions and answers based on the contexts uh but this is a way we can do it without having to spend those resources on very expensive humans so can we use llama index for the same capabilities is not yes yeah yeah yeah definitely llama index has analoges to all of these that you can leverage if you want to stay in llama index's ecosystem yeah all right awesome we just saw rag on rag with ragus and we're gonna go ahead and Rec recap exactly what Chris was talking about here and get this dialed in here the way to take rag to the next level is to get it set up right generally if you're doing this in production like cash is King you don't want to be redoing things like recreating all of your embeddings in your vector database all the time or rerunning and inferencing prompts that you've already gotten in the past but outside of that it's very easy as we saw to go ahead and pick up an ensemble retriever that helps us do a quick reranking to do really Next Level builds go ahead and set it up instrument it with evaluation start looking at things like chunk sizes chunk overlap get into that black art of chunking and really look at doing some interesting hierarchical metadata data hybrid retrieval stuff you know maybe even add some quantitative data as we've seen earlier Chris showed the parent document Retriever with the big idea right small docks are good but big docks are good too right so why not just use both we saw that children split parents in the child document receiver retriever as sometimes they do in this world and the Ensemble retriever was essentially this way to you know take a bunch of references and to then down select to the only top references and so we're kind of using this interesting idea of a sparse retrieval plus a dense retrieval when we're doing this kind of Ensemble and a lot of the vector databases today have this built in but here we're going ahead and we're just leveraging this tool directly in langing chain rather than using like a pine cone that is going to do this out of the box for instance so that's meta rag with ragas and advanced retrieval what did we see at the end of the day we saw that when we measured context recall from the base model to the parent document retriever to The Ensemble model and remember context recall is ground truth sentences that can be attributed to the context divided by the total number of sentences in the ground truth so it's about not just getting the right stuff but getting the right amount of the right stuff we saw an improvement with each of these additional retrieval capabilities not really surprising if you think about the way these are operating so with that ragas is uh pretty dope it's leading the way retriever metrics include context Precision context recall generation metrics include answer relevancy see faithfulness it's not really cost effective in general unless you compare it to like humans doing it so beware of setting these things up and just like smashing your open AI API key way too much and the eval is definitely evolving like you're going to see in the next session there's actually built-in tools now in llama index that you can do eval out of the box so you know to the question of do it in Lang chain or llama index you know it's like well you can instrument either one with ragus but they're all coming out with their own metrics to be able to measure these things in just one line of code within that infrastructure tool so lots of interesting stuff happening in the eal space very very cool to be able to build at least with this tool even since we started teaching this the ragas team at exploding gradients has made a ton of updates to the tool and to the documentation watch these guys it's a great open source project to potentially contribute to and um and that's it for rag eval state-of-the-art chat anversary year one 2023 next up we've got fine-tuning of embeddings where we're actually going to do some measuring using some evaluation tools of the real Improvement we get from that fine tuning of the embeddings so check that out coming up next in session eight that's a wrap