Semantic Chunking for RAG: A Technical Deep Dive

[00:00:01]hey whiz we've talked quite a lot about the black art of chunking in our courses over the past six months or so haven't we yeah we sure have yeah yeah people are always asking questions and they're always saying you know hey so how should I chunk my documents what's the answer is there an answer there's not really a one siiz fits-all answer to the question even uh even 2024 Greg well there's this new method I heard about and everybody's been talking about recently that apparently asks the question but what's the meaning of each chunk it's called semantic chunking have you heard about this I have yeah yeah it looks pretty promising doesn't it it looks like uh it looks pretty good yeah yeah yeah yeah okay well today we're going to dive in and we're going to see exactly how it Stacks up quality atively and quantitatively versus the gold standard sound good sounds like a great plan all right let's do it man let's jump in today welcome everybody my name's Dr Greg that's the whiz AKA The llm Wizard we are co-founders of AI maker space thanks for taking the time to join us today today we're talking semantic chunking for retrieval augmented generation it's one of the newest methods a lot of people are talking about it a lot of the new Frameworks are implementing the method and we're asking the question today is it something you should put in your toolbox as you start building shipping and sharing production llm applications today we're going to get to the bottom of it and you'll learn if it is something you should definitely keep in your pocket to pull out as you try to take your rag systems to the next level we'll have whiz back couple of times today to dig into some details about specific chunking we're doing on the document we chose today and of course we'll be back at the end for demo if you have questions along the way drop them in the slido link for us that way they can be upvoted and we can make sure that we get to your question all right let's get into it everybody today we're talking semantic chunking and overall today what we want to do as we align ourselves towards the session is we want to understand first of all the process of semantic chunking we want to contextualize why we would care about semantic chunking for Rag and then we want to take a really close look at how the kind of best practice chunking method is different but also kind of the same in spirit as the semantic chunking method it's all about meaning after all that we're trying to extract when we deal with words and when we deal with generating answers to our questions based on what's actually contained in our private documents and then we're going to leverage a tool that you've maybe seen us use before called rag assessment to actually look at how performance differs quantitatively between the two methods for building a simple rag system so first off we're going to sort of contextualize this thing with a little bit about chunking we're going to talk about two methods for text splitting AKA chunking one is going to be the recursive character text splitter the one that you probably are grabbing off the shelf if you're building llm applications today and then we'll talk about semantic Chun we're going to assess the method and do a complete walk through with the rag assessment and finally we answer any questions you guys have as we get to Q&A so when we talk about chunking it's really a simple concept we're going to break our text into smaller pieces we can't just feed all of the text at once in despite kind of the you know proliferation of people asking us about long context Windows today rag is always going to be something where we're chunking we're always going to be splitting things up into the right amount and what is the right amount for any given context is a very hard question in general but it's important to ask ourselves this for our application because all of this fits into the process of retrieval we need to go find the relevant information that we have in our documents based on any given question or query we might put to an application we build with an llm and retrieval of course is the first part of rag it puts the r in rag it's where we find our reference material it's where we get our facts that we're going to leverage to improve our Generations after we augment our prompt and of course putting the r in rag is just this idea of retrieving often dense vectors and if we take rag as sort of two component pieces we can look at the fact that it's really just this dense Vector retrieval piece doing a lot of the heavy lifting and then we're leveraging the concept of in context learning by putting this information into the prompt by augmenting The Prompt so it's important to really focus on this retrieval step when we ask a question that question is then going through getting chunked and getting turned into a series of numbers those numbers are then we're looking for that representation in embedding space of our question and we're looking for stuff that's similar to it we're looking for stuff that in embedding space is close to our question and that's where we go to our Vector store which is full of embeddings that represent our data that are also in the same embedding space and we're looking for a distance between our question and the stuff we want to retrieve this is the key point of the day because if we chunk differently whether we're chunking our question or whether we're chunking all of the data in our Vector store it's going to change the results that we get and so this is why we sort of refer to this as the black art of chunking it's the thing people often don't talk about because there is no sort of right answer although there are best practice methods this allows us to then after we set up our prompt template return those chunks in natural language after they've been compared in embedding space as reference material so this dense Vector retrieval is the point really it's the thing that drives the improved Generations when you finally Feed The Prompt into the chat model and get your answer this second piece is the in context learning piece it's really not the point of today's lesson but overall this retrieval augmented generation starts with retrieval and retrieval starts with chunking in other words what gets chunked gets retrieved this is the reality of the situation and so when we look at retrieval we need to take a really close look at this Vector store this is where we take our data we put it in in vector format to a place that we can search for stuff and the thing about the vector store is that the only thing in there is Chunks that's it so chunking really is fundamental here and so we want to make sure that when we're chunking we're doing it in a way that we believe is useful we believe it's going to produce the best possible results and we believe aligns with what we're aiming to ultimately use our LM application for improving retrieval can be done through different retriever methods but it can also be done by improving your chunking now leis Carroll famously said in Allison Wonderland the best way to explain is to do it we're going to leverage leis car writing today and we're going to Chunk Up Alice and Wonderland and put it into a vector store we're going to do this a couple different ways two different ways but to make sure that we're clear on why those two different ways matter let's talk about the chunking methods that everybody's mentioning right now number one is the fixed size method like give me your chunk size and number of characters give me the overlap between chunks that's it okay very basic very simple the recursive method is the one we're going to dive into a little bit more detail in just a minute but if this meme is any indication what we're doing is we're combining some ideas of fixed size now it's worth noting that you can use a chunk overlap and chunk size fix exercise approach you could also do something simple like say why don't you just chunk by sentence or chunk by paragraph there sort of a natural indication that that's going to be useful in some ways to us as people who are reading and writing in sentence and paragraph form there's also document specific type of chunking this will leave aside for the day but the great use case here is to think about programming code oftentimes there are natural breaks within programming code that indicate well this is kind of a chunk of code right here and so you can do this with different document types and we won't get into this as we get into more and more structured type of language data there's going to be a lot of different ways to potentially do chunking in the future for more complex documents today we're focused on sort of unstructured text more like novel that we'll use and then of course there's semantic chunking and we're going to dive into this in a minute this is the purpose of today's event it's also worth noting that agentic chunking is a thing that's starting to be discussed now the idea here is that after initial chunks are created then you use an llm to reason about whether or not you should chunk differently this is sort of the meta pattern here the idea of a gentic chunking we want to make sure we're clear on is a pattern and if you missed our event last week on agents for rag systems give that one a watch to get a little more insight into the many many different ways you might use an agentic chunking approach roach this would basically be putting some reasoning into the retrieval step and there is a specific sort of propositional approach people are talking about today with agentic chunking but we expect that this will be more and more sophisticated um the methods will grow as the popularity with agents and multi-agent systems grows but for today we're going to focus on recursive and semantic chunking in order to understand semantic and recursive chunking we have to start with this most basic fixed siiz chunk number of characters overlap by sentence by paragraph do these things make sense well how about if we kind of combine these ideas into more of an ensemble approach if you remember classic machine learning this always worked well and indeed it works pretty well when we talk about chunking introducing the Swiss army knife of chunking shout out to Greg comrat for coming up with that analogy this is really the deao standard and what recursive text splitting does is it kind of leverages the fixed size what's good about the fixed size and overlap chunking while also leveraging the natural flow of language in sentence or paragraph form so this is sort of the recommended generic text splitter it essentially has a fixed chunk size and a fixed overlap in mind but then it goes through and it looks at different separators for instance the double new line separator is indicative of a new paragraph a new line separator is [Music] often indicative of potentially a break in the meaning of what's going on uh this could be a new sentence but there's a lot of different ways that authors can use new lines in their writing if you think about the many different typ typ of authors you've read and how they can get pretty creative and then as we sort of move through each of these types of separators we sort of return and we ask oursel about how close we are to our desired chunk size and we go down through to the Space level and to the Character level and I want to look at an example right here from Allison Wonderland this is from the very beginning of the book page 2 to three if we use a recursive character text splitter out of the box this is one chunk that we might get and I want to bring the whiz back up on stage now to discuss this a little bit with us because so whiz we've got this double new line new line spacing and character sort of level of Separation we're looking at but yet there's this recursion there's this return to this fixed size and overlap can you talk a little bit about how you understand this chunk being quote unquote decided upon by this algorithm here why would it choose this particular chunk yeah you bet so the basic idea here is that uh we we lack the double new line in this chunk right so uh you can see that there's a double new line before there which is above this uh this specific chunk um and uh you know we don't have one in the highlighted text so we can't split on that right we're we're kind of already out uh we're then going to see okay well can we split on a new line character uh yeah well we have a couple uh but we don't have any that are well-sized let's say right we have a very small one we have room left over so you'll see that we split here on our uh space character because of that right so uh we split at the end on a double new line uh since we don't want to go into the rest of that paragraph based on the chunk size that we have so far right um and then we are we don't have a new line that gets Us close enough so we're going to split on a on a space character you can see like because these are generated in order so from start to finish we left off the previous chunk at o so we had to start at Deer basically and then the next closest to our desired size which is 200 was the double new line that's at the end of this chunk so after again period there's going to be two new line characters that's basically it um so like this when you start a document obviously you're just going to start at the start um and then you go until you find a double new line or your chunk size and if you find your chunk size but you don't find a double new line you're going to look for a new line that's close to your your chunk size and if you can't find a new line that's close to your chunk size you're going to look for a a space that's close to your chunk size and we just keep repeating that recursively even um until we we get our desired chunk and so this kind of in this specific instance right so we we we we were talking about how we're not going to talk about document specific chunking but because our text is a book and because it has paragraphs and chapters and sentences It lines up such that the default recursive text splitter is actually uh a great document uh text splitter in this case because it is following the format that we'd expect paragraphs contain single ideas sentences smaller ideas and we want to not break on middle of sentence possible that's the idea I see I see so really it's not that different than the fixed size it it's sort of Shoring up the edges of the fixed size in in a way it's it's kind of not doing something fundamentally different it's still sort of aiming at this fixed size in the end yeah absolutely right the only small Advantage is that we're more likely to break or split on a place where you would expect there to be different information on the double new line right if you think about a Wikipedia article right the double new line is going to be a new section the single new line is going to be interp paragraph and then uh a uh you know a space is inter sentence and then character is inter word and we want to avoid inter word where possible uh because those words are definitely a piece of information and so we'd really rather not split on the word we'd really rather split on uh you know these bigger pieces of information where we're we're likely to see a difference in meaning between the two so that's yeah ex right yeah okay yeah yeah definitely not trying to split words up that makes a lot of sense okay so but yet this still leaves much to be desired potentially I mean intuitively I'm reading this and I'm looking at you know there was nothing so very remarkable in that and I I want to include this in the chunk I mean something about the nature of the paragraph like this feels like it's appropriate to the meaning of the chunk doesn't it it sure does I mean in fact we we split the direct quote and in this case right it's it's maybe not so important because we're just loing off oh dear but imagine we lopped off a negation at the beginning of a quote right like this is a this is something that's potentially problematic and with the fact that this naive chunking strategy really has no real Dynamic size right it's just kind of like it can get as big as it can get and uh it's proa it's going to greedily try to get as big as it can get um I think this is we wind up in a situation where we're you know we're we're definitely losing something from this quote like and again in this specific example it's not super impactful but it's very easy to imagine as situation where that first word was a very important piece of the of the quote right so I think it's uh yeah you're exactly right to say we would love to have that previous stuff because we know it's related to this later stuff right and so uh we you know we can think to do this in different naive ways maybe we only ever Chunk on new lines and we never do inter sentence so we let our chunk sizes be quite big or we don't Define a chunk size right we just know our documentation but there are many instances where we can't perfectly know our documentation or that's going to take too long or too much processing or etc etc so that's right that's right yeah I mean let's say I asked you know hey who is late and I return this chunk it you could probably maybe a a middle schooler could kind of maybe eek this out but it'd be tough without saying to hear the rabbit say to itself you know it's like boom there we are so okay very cool then um I really like that we're sort of seeing there's the super naive fixed size and then there's the slightly less naive recursive character text splitter so that's the big takeaway from here all right thanks whiz we'll have you back in just a little bit so as we take it to the next level here let's talk about semantically splitting text is there a better way can we do it in a way that's more meaningful let's say well consider a classic retrieval algorithm Improvement choice that people building with Lang chain will often make today this is something that we've used extensively in our courses and that we encourage folks to check out as a entrylevel advanced retrieval method it's called the parent document Retriever and there's two big ideas in the parent document retriever one is that small documents are good and the other one is that big documents are also good let's focus in on why small documents are good right now they're good because the embeddings accurately reflect meaning and relevancy small is good because the embeddings accurately reflect meaning this is key because if we can accurately reflect meaning in our embeddings without having to make them small then we solve the same problem that small docs within the parent document retriever are solving this is exactly what semantic rag is trying to do in other words as we look at big documents are good because they retain context Within each chunk and small documents are good because embeddings accurately reflect meaning semantic rag is going to accurately reflect meaning and retain context within each chunk at the same time semantic rag one way to think of it a is as the best case of a parent document retriever now we've got a event in a couple weeks on Advanced retrieval if you want to see us break down a ton more of the state-of-the-art methods definitely check it out and join us for that as we look at chunking semantically and Shout out again to Greg comrat he's kind of the guy who came up with this idea and he came up with this idea based on a tweet from lonus here Greg actually went did it very cool and he said the hypothesis is we can use embeddings of individual sentences to make more meaningful chunks so we're going to chunk by sentence we're going to look at embeddings of each sentence chunk when we create chunks what we'll do is we will number one split the document into sentences what we're going to do is we're going to group sentences as well so we're going to take sentence 1 2 and 3 we're going to compare it to sentence 4 five and six this is a this is the standard in the method you could imagine doing this with groups of any number of sentences of course we ask ourselves is the block of sentences 1 through three close in embedding space to the block of sentences four five and six how close how similar is that similarity metric if they are similar right if they are similar we want to combine those into one chunk because if they're similar in meaning we want to use why small docs are good and combine them right they're good because they capture meaning well so if they're similar we want to combine them if they're too different right if they seem to mean things that are too different from one another then we'll split this is the big idea now if you've got a good intuition on this you might think well we might get some pretty freaking large chunks if we do this and in fact you you'd be right because out of the box when we Implement semantic rag from Lang chain we get a massive chunk again we're not chunking here based on fixed size and overlap this is the beginning of the book and this is on page six well at least of the addition we grabbed offline it goes all the way from the beginning to page six using this embedding similarity chunking approach so I want to invite whiz back up here again to talk a little bit about how he's thinking about how this thing captured six pages in one chunk is that right is that what we got here whiz how did that happen I mean yeah it kind of makes sense right like the chapters are ultimately a large semantic chunk of information uh you know likely what we're g to think of as a single semantic related portion of a book right if you if you ask someone you know hey how would you best group chunks of meaning in a book they're probably going to say oh chapters is a good start right usually it's a coherent idea now there's there could be some difference and in this case there is some difference right so we see like a distinct shift between the two uh the two chapter or the two sections of this chapter uh one which is before Alice starts to fall down the rabbit hole and the second after which she starts to fall down the rabbit hole and and you know I think this is why we have this break you know to try to guess at the uh the reason but the idea is that the first part of the chapter is basically just about Alice wanding around bumping into the rabbit right and uh all of that information is relevant to to this chapter and so that's why we get this uh this split here and that's why we're capturing so much information right the idea especially when we talk about long context window and everything blah blah blah blah blah um when we when we talk about these things we are we are left in a place where we're thinking about well if this is all related to the first part of this chapter if all this is interconnected right if this whole first bit is about Alice meeting the rabbit and then falling down this this rabbit hole then we should we should group it together right we should provide that as context we're most likely to include uh relevant pieces of information if we do that as well you can think of this as an analog to parent document Retriever as you suggested except instead of just relying on locality right so we know that pieces of information that are near each other are probably related to each other instead of just relying on that uh that that concept we're also going to leverage well we should think about if they contain meaning to each other right it's like in a textbook if you're talking about one concept and then you pivot to the next concept yes the locality would would would would lead you to believe that these things are related but they're they're different in meaning right the first bit might be about uh a certain you know a certain situation and the second bit might be about another situation and so um you know this lets us build these larger chunks without compromising the meaning of that of that chunk or of those individual chunks the other way to think about it that that I like to think about it is whenever we embed something into this space we you can think of it as literally a point in in 3D space or end dimensional space right and what we're looking at is every time we add a new set of documents the point's kind of in the same place right and that's true until it until we shift it past a certain threshold right which is what we're going to talk about a little bit in the in the code and if we move it past that threshold we're saying okay there's a difference now but as long as we keep kind of pointing to the same place in that space I mean all that information's you know wonderfully relevant and it and it will only help us to understand the context and so that's I think about it yeah I I I really like this textbook idea or sort of the technical report idea it seems like in those cases there'll be much more natural shifts in context that you might not find in a novel I mean fundamentally right a novel is kind of all the way connected on some level and so you know to sort of do SUB chapter even is tough I mean if in many novels there's sub chapter headaches and that's the natural break point right and so you know we we could we could you know sit here and get very very particular about well if it was up the rabbit hole and down the rabbit hole as you say well why did we get down down down in the first you know and and there's all sorts of really nuanced things when we try to be data Centric here but this is a very useful way to to think about especially in specific types of documents I think love the textbook idea love the sort of technical reporting idea places where you sort of switch context more than a novel this might be incredibly useful and you know that that to me feels like a potentially big big win for this method although I don't know and we'll see what the numbers tell us here shortly won't we we sure will yeah yeah all right well let's go ahead and introduce this build we'll have you back in just a little bit whiz what we're going to do is we're going to now see if we can compare these quantitatively everybody likes numbers it's hard to compare these things qualitatively for any given type of document in general as I hope we've sort of demonstrated with this simple novel what we're going to do is we're going to load the document we're going to chunk using two different methods the ones that we've talked about the recursive character text splitting and the semantic chunking we're going to in the notebook see if we can look at an example or two from Alison Wonderland is it clear which one's better well we kind of talked about the prim primary example of the session I think it probably makes a little more sense that we kind of split the chapter in half maybe up the rabbit hole down the rabbit hole kind of thing versus being more rigidly attached to a 200 character chunk approximately but one that's looking for new lines or spaces between words and then by building out a simple langing chain rag system we're going to be able to quantitatively wrap that in the rag assessment framework and decide what the metrics we're getting are telling us of course we're using Lewis carrols Alison Wonderland to do our chunking we're using open AI models here and we are using the rag assessment framework which we've talked about in a recent event on the art of rag evaluation if you want to go into details on the latest from the guys over at ragas shout out to them and their recent y combinator appearance then check out this event to dig into all the details of the ragas framework but for now you'll understand kind of what's going going on at a high level as we see the numbers go up or down as we assess semantic chunking of rag quantitatively in the notebook whiz sending it over to you oh yeah okay so we have uh pretty straightforwardly you know the the The Way We Begin most of these builds we're going to get some dependencies uh we're going to grab laying chain experimental uh which is which is new that's because this is an experimental feature still being worked on uh not everything's ironed out some of the strategies are still a little bit naive and this needs to be you know built upon or uh you know expanded on in order to to reach like the full quote unquote the full power right so uh we're going to keep that in mind as we go through the notebook we're also going to grab open AI since we'll be leveraging those models through both our rag LCL chain as well as our uh rag assessment we're going to just grab Lang chain core and ragus we're also going to grab phas CPU and Tech token uh phas CPU is just what we're going to use to build our local embeddings uh store sry Vector store uh we're just going to use face to power that next we're going to grab aliceon Wonderland uh we just grab it from Gutenberg this is a copyright free book so it works for the uh for the for the demo today we're going to read it into memory and that's it we're just going to leave it there uh you can use a document retriever from Li chain but there's there's not a significant uh difference here so we're going to first just chunk uh you know these naively to see what the chunks look like um and we're going to see that we get our recursive character text splitter we're going to have a chunk size of 200 no overlap we're going to just use the python Len function which means that we're not using token length here we're just using characters uh and then we're going to keep the is separator Rex equal to false this is just a parameter that you should set um we're then going to split our documents and look at some of the chunks we can see right this is exactly what Greg outlined to us we have a number of small chunks uh and these chunks are broken apart kind of just you know in the middle of sentences uh they're they're broken apart uh you know maybe inconveniently uh again these this chunk size was picked purely to illustrate this point U this is still true at a larger chunk size uh it's just going to be less frequent depending on how many documents you have but the idea is pretty straightforwardly right uh these chunks don't contain a lot of information in each chunk and it's disjointed from potentially relevant information nearby so how you know how do we Implement semantic chunking well first we're going to need your open AI API key you can use a local embeddings model for this uh like you know any of the hugging face models you use coheres embeding whatever embedding model you want we just do need something to embed uh our chunks so we're going to look at semantic chunker a little bit more technically here so we kind of got the intuition of how it works and you know why we care about why it work or how it works but uh we're going to talk a little bit more about the actual implementation so the implementation is very straightforward uh first of all we're gonna have a number of different thresholds uh these descriptions are straight from the Lang chain dots uh you can see here uh I just you know that nothing explains better than the actual documentation so there you go but the idea is you know when we think about when to actually split right when to make a a gap in chunks we want to do it because there's some kind of difference so the first thing that we can think about is a percentile difference right so once we get all of our differences in between our sentences we can see you know if there's an X or greater percentile split uh that's when we make a a break in the chunk right uh we could also use the standard deviation so if the chunks are greater than a certain standard deviation away from each other then we should not incorporate those together and lastly we can use interquartile distance between the chunks to determine when they should be split uh these actually have a significant impact on how your document is going to be chunked so I would I would experiment with all of them uh in this case we're just going to leave it as a a default which which is percentile so if we have uh you know these greater than a certain uh percentile difference between two chunks we will not combine them together um and the way that this actually works in the code behind the scenes is we first split our entire set of documents into sentences we're going to do this very naively uh just based on period question mark exclamation mark right so we're not we're not doing like Spacey sentence tokenization or anything like yes we certainly could and maybe that would lead to different results maybe that would be better certainly be more computer expensive though so we're just going to naively you know break on S sentences uh as we understand them uh based on these punctuation characters we're going to index each sentence based on its position the document so the first sentence is the zeroth sentences sentence and the uh last sentence is the number of sentences sentence right so uh pretty straightforward stuff we're just going to index them based on their position then we're going to do something that I think is integral to this strategy we're going to combine sentences without thinking no no embeddings required we're going to combine groupings of sentences in in by default we're going to use this buffer size equal to one which means that the sentence that's distance one on either side of our Target sentence is going to be included in a group by default so as Greg expressed right right sentences 1 2 3 are considered a single unit sentence 456 is considered a single unit in this in this specific example right um you know 0er one two and three four five if we want to get very compi about it but you know this buffer size is something you can play with right the the basic idea here is that we expect that the sentence by default will be related to the sentence before and after it uh this is not strictly true doesn't have to be true but it's a good thing to think about uh and it's a good place to start uh because we do want to make sure that we're capturing some some uh you know combined representation of these sentences right we don't want just each sentence because each sentence might be very different from each other but still related to the whole subject matter so we're going to seed seed the embedding uh with this uh this kind of three sentence idea uh then we're going to calculate distances between groups of sentences we just look at their embeddings and then we calculate the cosine distance between them and that's it uh you know now we have a bunch of groups of sentences and we have their uh the cosine distance between each of those groups and then we're going to merge the groups based on the similarity uh based on the chosen threshold right so this is where we're making the decision uh if if sentence group you know a is uh X percentile different than sentence Group B we will not merge those together and so on and so forth and so that's the idea again this is experimental it's still being worked on we still have a lot of space to explore here there's not a ton of research that's that's concrete on this so uh you know expect that this will will change and and become U more more and more performant over time uh the actual implementation is is uh is quite straightforward right we call a semantic chunker we provide an embedding model and we choose the break point if you don't choose the break point it will default to percentile uh but you can choose it so we we show that here you'll notice that we're going to use Tex embedding three large I this is intuition based only but my assumption is the better our embedding model is the more accurately we're going to be able to chunk Things based on that uh embedding so we I went with a large embedding model here though you could choose a smaller one so you can choose a tech embedding three small uh it's totally up to you I just went with the intuition that a better embedding model would lead to better chunks um and uh I have no concrete you know there's no research that says it's true it seems kind of likely so that's why we did that uh and then you can see that we get this chunk exactly as we saw from Greg you know we it's quite long it's it's in fact 4,000 characters right so that uh you know 4,000 characters is quite a lot each of our other chunks only has 200 characters and so if we have if if we do the math right uh you know we're going to see that we have the equivalent of like 20ish chunks um and that's uh you know that's something that we'll keep in mind going forward so now we can create a rag pipeline uh we're going to use text embedding three large for our uh Vector stores embedding model the there's no research that indicates you know that we should use that but the again the intuition would suggest we should use the same embedding model for uh for comparing our query Vector to what we have in our Vector store that we did to create the chunks right that's going to uh that will communicate the best semantic consistency between the two different uh processes so that's why we went with that but again you could use anything I mean it will work if you use anything uh but we we chose this with the intuition that you know because we created the chunks and the chunks are related to each other semantically according to this embedding model then we should use the same process to compare them um and then we are going to just use one chunk here right because they're big chunks and so makes sense we'll just use one uh semantic chunk and then we can ask questions you know like who has a pocket watch and we get some kind of response then we're going to create our uh uh augmentation prompt then we create our model and then we create our chain this is all LCL stuff we got ton of a ton of uh you know previous material on this if you're if you're interested in exactly how this is working but uh for now we just create the chain and then we can ask questions how does Alice find herself falling down the rabbit hole into Wonderland and we get this awesome response it's quite uh quite fully fleshed out pretty cool uh and then of course we get uh we ask about Dena and we get uh dena's Alice's cat and she's important Alice because she is a great Mouse catcher that's pretty cool uh then we can go to the other strategy which is naive we're going to use the same embeddings model just to make it fair and then we're gonna use 15 chunks uh of retrieved context the idea here is we want to we want to level the playing field a little bit right so where we have these one one big chunk we're going to use 15 smaller chunks uh in order to give this a chance right and the idea is that this should make it more fair comparison as a token count should be relatively consistent between the two methods then we're GNA ask the same question and we get a slightly less robust answer uh and we get an answer about Dena uh but it focuses more on just that she's great and it doesn't focus on why she's great and why she's great is because she's a good Mouser so uh you know we lose that information okay so we so like I mean just looking at it you could kind of say okay I you know maybe this is actually preferential uh the the these responses seem a little bit more relevant to the answers okay let's see if we can actually put that into numbers so we're going to go ahead and we're going to create our raggas assessment uh comparison data set we're going to chunk the uh Source document in a different chunk size just so our uh our our naive rag can't just cheat right it can just cheat if we use the sameeh size so let's not let it to cheat um so we're going to create it in chunks of 400 you can choose any number here it's just just as long as it's meaningfully different from your uh your other strategies right your other pipelines uh because we don't want it to be able to cheat then we're going to create questions which are synthetically generated by gbd 3.5 turbo this is for each Chunk in our uh data right uh we're going to use those contexts that we created above we're going to create ground truths which will be generated by gbd4 Turbo and then we're going to have our chain answer the question based on the question and context uh though it will retrieve its own context uh as that's the point so we have this here question prompt you're a teacher preparing for a test please create a question and using the following context and question to answer the question only the provided context there you go and we'll create these two chains and we'll run this through you can also use Rus as built-in synthetic data generator um you know it is it is a wonderful tool um we're just showing kind of what's Happening under the hood here and then we will create our data set and remember the question is generated by gbt 3.5 turbo the answer comes from our uh semantic rag pipeline the context come from the uh ground truth contexts and the ground truth answer comes from gbt 4 then we're going to evaluate based on answer relevancy faithfulness context recall and context precision uh just the defaults from from ras' uh documentation and we'll get some results I mean we see the result context Precision is high faithfulness we have like a hit or mess we get some zeros and some uh some nans here so not of numbers our answer relevancy is pretty decent at 0.73 and our context recall is quite High which makes uh exact sense so that's great now let's look at the uh naive strategy right so same thing but the naive strategy and we can see that they're a little bit different and I'll let you guys go through the numbers in detail in Notebook if you wish but we can see that the actual naive result is pretty much the same right for everything except for answer relevancy and her answer relevancy is notably worse with the naive strategy which kind of makes sense right like it has less of that relevant information and so it's able to answer the questions less fully and uh and there you go that's the that's the basic idea and we see the semantic chunking strategy uh seems to have better performance on answer relevancy without giving anything else up and so it's uh it seems like a good idea to pursue it um with perhaps some modification and perhaps more testing on which uh thresholds work best but the idea is you know if if if you're to ask me uh is semantic chunking better the naive Chucky I would say to you uh seems so yes that's that's what uh that's what the assessment seems to indicate so uh with that I will push you guys back to Greg who will take us to our Q&A all right thanks whiz so it does what it says on the 10 all right well excellent that was semantic chunking of rag and in case you didn't quite catch those metrics just briefly context precis ision is asking how relevant is the context to the question context recall is asking is the retriever able to retrieve all of the relevant context and we got similar performance on both of these faithfulness is asking is the answer fact checkable or is it a hallucination again similar performance here but the difference was on on answer relevancy which asks how relevant is the answer to the question interesting the semantic chunking process of splitting then indexing on position before grouping calculating distances between those groups and then deciding whether or not to merge based on similarity thresholds this is something that we'll continue to see evolve and and it does appear that it produces good results in some cases as we think about this particular key result of the day the answer relevancy Improvement on semantic chunking versus recursive text splitting this is a generation metric and again it asks how relevant is the answer to the question importantly this does not consider factuality but instead penalize it cases where the answer lacks completeness or contains redundant details specifically to calculate this score an llm is prompted to generate an appropriate question for the generated answer many times and then a mean cosine similarity between generated ansers an original question is measured so that over overall if we can generate an answer accurately addressing the initial question the llm should be able to generate questions from the answer little Uno reverse card metric here and we see the Improvement so that's pretty cool more relevant answers with semantic chunking and that's kind of the takeaway for the day remember what gets chunked gets retrieved so if you're doing rag you should think about chunking you can be naive and that's fine but maybe you can take it to the next level and maybe you should keep this in your toolbox and watch the emerging techniques especially the thresholds at which we do semantic chunking it was quantitatively better and perhaps it was even qualitatively better perhaps it could be even a little bit qualitatively better on different types of documents better suited for this and we've got a couple question I just want to ask welcome back up to the stage whiz um you kind of mentioned this before but I do believe that it's really kind of worth tripling down on does it matter which embedding model we use when we're going and we're thinking about chunking or should we always just kind of you mentioned bigger and better model likely going to be more useful in this case that's sort of your intuition should we be playing much with the embedding model as we're playing with semantic chunking yeah I mean my so I feel like the answer should be yes to this question uh we'll have to wait for more kind of results and research for for that to be upgraded to like a yes definitely that's true uh you know I'm not just saying it because it feels right um but it does feel right right so the better the better chunking we have uh in this case is related to how well we can capture semantic information about our uh about our sentence groups so the better we can represent those sentence groups the better chunking we should have right uh the smaller differences we should be able to capture and so I think that's uh I believe the abetting model should matter similar to how chunk Size Matters in in you know naive chunking right uh or the characters you split on matters I think it's the same for the embedding model and its performance here because we're actually using it to do the chunking that's correct that makes sense so it's literally directly dependent on it okay so um I want to go through a couple questions here briefly um one super common here we go uh are there better Alternatives than cosine distance what's up with this coine similarity can we can we get better than this man that is a uh that's a tough one I mean the answer is yes but they everything everything that might be better is much more compute intensive and and that's the the kind of the game we have to play uh with these distance measures is we have to do a lot of them right so if we have 500 you know sentence groups we we have to compare each of those right that's a lot of calculation that needs to happen and so for it it to be very efficient um and performant is very important and I don't know that uh we have something that's drastically better than cosine similarity that would work well in that slot without blowing up compute costs uh that that being said uh you know it does kind of depend on your data and you could use some of the other uh simple distance measures if they work better for data but realistically we build everything with this idea of of cosign similarity into it and so we we want to keep using it yeah and if you want to do a sensitivity analysis on all the different metrics have at it I think there's a lot of people working on their phds doing this kind of thing right now all right how compute intensive speaking of compute intensive stuff is this strategy in general um would it be appropriate in the scenario where we have to constantly update and reindex stuff or not really so okay so that's an interesting question right uh if you have changes in documents that you're reindexing then this is going to definitely be uh not the option for you if you're adding new documents I mean doesn't matter right we're only doing this in each document right we're not whenever we add new documents we have to do this process once but then it's done and so if you're working with Dynamic documents that are themselves Dynamic so they change uh this might not be a great strategy in terms of cost if you're talking about uh you know documents that that that you know are added so you have Dynamic documents but the D Dynamic part is that you add or remove documents from the pool that it's great strategy I mean it is always going to be more expensive than a naive strategy because we're having to use some embedding model even if we run it locally even if we run it on CPU it's going to take more clocks to get it done uh than than you would have if you were just checking uh based on which character was being split on so yeah okay all right cool cool so you know we talked similarity distance measure we talked embedding model um the last thing we get this question all the time we'll end on this uh what chunk overlap should I use whiz like what's the what's the answer what chunk overlap is the answer you should use the one that's best for your uh for your documents um and you should determine that uh through tools like ragus or any other evaluation framework um you know chunk chunk overlap is a hyperparameter we should do hyperparameter tuning of some kind um you know in this case maybe not literally hyperparameter tuning uh but an analog we should we should empirically determine the chunk chunk overlap as we should the uh chunk size by running some kind of search and choosing the one that makes the numbers the highest and there you have it everybody whiz I believe we have one more ask of everybody today don't we yes that's right I forgot to say it uh please it helps us a lot if you uh subscribe to us on YouTube as well as click the little bell notification uh to get notified of our uh future events we do on every Wednesday at this time you can always find us here uh so join us for your lunch break if you're EST uh otherwise you know uh thanks so much for for tuning in today thanks whiz awesome man and it's time to close it out everybody if you like this session definitely ring that Bell but also consider joining our Discord we've got great Vibes and we'd love to have you it's growing rapidly and there's always some interesting conversations going on every day at this point if you're looking to learn more for free and get started with production LM applications check out our open- source course the one we taught last August on llm Ops cohort 1 you can find that on GitHub and we've got a number of YouTube videos associated with it of course if you're ready to accelerate all the way out at the edge we are happy to announce that we've got our upcoming cohort 2 of our AI engineering boot camp that kicks off next Tuesday April 2nd and it's going to be the best yet we've been rapidly iterating in cohort one which wraps up this week check out LinkedIn for the demo Day event that you could join us live for on Thursday and check out what people have been building shipping and sharing and finally if you have any feedback for us we'd love to have it let us know what you'd like to see next let us know how you thought these events have gone recently and what you're paying attention to out there from chunking to retrieval methods to agents to multi-agents and Beyond and as always everybody until next time keep building shipping and sharing we'll continue to do exactly the same see you back next Wednesday or maybe tomorrow night for ae1 demo day cheers all have a great week