Chunking Strategies for Retrieval-Augmented Generation: A Comparative Analysis

[00:00:00]hello everybody Adam LK here and today we're going to dive into something that's commonly overlooked when setting up retrieval augmented generation applications and that is the chunking step chunking is of course what you do when you have a large knowledge base of text and want to split it up into the workable pieces that can be embedded into your vector database and then efficiently retrieved back from your vector database to do the augmented generation step there's many different ways of actually chunking text before embedding it and as someone who designs and develops these rag systems I was interested to know about all the different ones out there and maybe a little bit of what might be the best so when I came across this technical report from chroma DB the open source lightwe Vector database on evaluating chunking strategies for retrieval that introduces and puts to the test all sorts of different existing chunking strategies as well as some novel approaches I wanted to put a lot of them together into one place and demonstrate exactly how it all works so that's what we'll be taking a look at today in this notebook we're going to cover character and token based chunking recursive character and token based chunking semantic chunking and then the two novel strategies that chroma DB developed cluster semantic chunking and llm semantic chunking so to start one of the things that we'll actually be taking advantage of is along with this technical report chroma also released all of their code in this repo called chunking evaluation and so the chunking evaluation repo here has has all of the different text chunkers nicely organized into this repository as well as a whole evaluation framework we're mostly just going to be looking at the actual chunkers today but might dig into the evaluations at a later date that being said if you're interested in getting into the granularity of how each one of these algorithms specifically Works definitely check out their technical report and look into the code and the repo so with the repo installed and all of the different dependencies imported I wanted to put together more of a light-hearted knowledge based example so we'll be using just Pride and Prejudice by Jane Austin it's available on Project Gutenberg for free in a nice plain text format like this so that made it very easy to just export and use as our example I also have a quick helper function that we'll be using across the course of this video that'll just grab the 200 and 2001st chunk out of all of our chunks so that we can do a little bit of comparison and Analysis but before we get into those different chunking strategies I want you to ask yourself are you as confused as I used to be about all of the different ways that you can even start learning about artificial intelligence well it turns out these topics are actually very approachable with the right resources which is why I'm excited to be introducing the partner of this video brilliant brilliant is the platform where you really learn by doing with thousands of interactive lessons in math data analysis programming and artificial intelligence with brilliant you can bolster your critical thinking skills through actual literal problem solving not just reading textbooks and memorizing as a plus brilliant makes it super easy to start learning anywhere anytime right on your phone whether you're checking out a new topic or just brushing up on some existing skills You Can level up on the go in just minutes one topic definitely worth checking out is brilliant's comprehensive range of math courses their lessons get right to the point and focus on the essentials highlighting just the most useful applicable math concepts you can directly apply build and use formulas to solve real problems in both business and everyday life to help enhance your Visual and spatial problem solving skills to try everything brilliant has to offer for free for a full 30 days visit brilliant.org adamc or scan the QR code on screen or you can click on the link in the description you'll also get 20% off an annual premium subscription a special thanks to brilliant for sponsoring this video and so with that initial context out of the way the first actual text splitting approach that we'll look at is the simplest form character-based text splitting essentially this is just going to be counting some number of characters and then putting in a split at that count this isn't actually a chunker that's part of chroma's repo but it's super simple to just Implement like this as it's simply just counting the number of characters and then making a chunk and then moving on to the next number of characters so for our first example we have a chunk size of 400 characters without any overlap and we can see that it will literally just go through and then once it hits 400 character make a cut and then the next 400 characters will be made into another chunk and then cut again one of the slight modifications that we can do is actually add in an overlap size and what the overlap does is if we have a chunk size of 800 characters we will have the first 800 characters and create a chunk and then go back 400 characters and then start another 800 character chunk again and so with this second example of a chunk size of 800 and an overlap of 400 characters we can see that when looking at the 200th and 21st Chunk we have a bit of an overlap that we can see of of course 400 characters more specifically you can see that at the 200th Chunk we have this 800 character chunk here and then starting at the 21st chunk you can see that it starts right about halfway through the 200th which would make sense given that our overlap size is about half of the chunk size you might be wondering why include an overlap and the main idea about an overlap is actually more so about preserving context across chunk boundaries in our original approach when we're just splitting as you can see we split you know right in the middle of a sentence literally right in the middle of a word and that might cut off and not preserve some ideas that might span across multiple different chunk sizes so an overlap might be a better approach to trying to get all the context across individual chunks better preserved at a chunk size of 400 we saw that there was 1,871 chunks and then nicely with the chunk size of 800 and overlap of 400 we also see that there is about 1,871 chunks and of course these chunks are what would be then embedded back into your vector database and then each one of these would be a candidate for retrieval during the similarity search One Step Up For from the character-based chunking is actually token based chunking and what people very quickly realize is that language models don't actually process things in individual characters they process words and phrases as a series of these tokens so it might be better to instead of operate at a character level operate at a token level to better chunk the text so that it's more efficiently processed through these neural systems and so for a quick visualization on how tokens differ from characters we can actually look at open ai's platform and they have a very good example of how this tokenization actually works and so we can see that with each of these words they become split into sections or phrases that might not even be the full word and it might also include things like spaces or in their example here individual Unicode tokens of course this text isn't actually stored as text but then becomes the numbers that are passed through the language model to do its next word predictions and this conversion from text into the actual numbers that the language model can understand is done using what is called a tokenizer which is essentially a compression algorithm that can map different phrases words or parts of words to numbers that the language models can understand the most frequent tokenizer that's used for for these token based calculations especially for Tech splitting is the CL 100K base tokenizer from open AI the 100K refers to what's called its vocabulary size which means that it has about a 100,000 different representations of all of these chunks of words into tokens that it can do for a quick example if we run the word hamburger through the CL 100K based tokenizer we get back the tokens 71 and 4777 five and then if we do the reverse and instead of encoding the word hamburger we decode it with the tokenizer you can see that this first 71 corresponds to the H and then the second token corresponds to the hamburger part of it putting both of those through fully decodes into hamburger but all of this is to say that language models operate by first encoding all the text that they process with these tokenizers and so naturally it might be even better to do chunking at a token level so using pretty much the same approach as the character-based chunker but instead counting tokens we can run through and see how that looks so using a chunk size of 400 and the CL 100K base encoder we can create chunks of 400 tokens with no overlap we get about 440 chunks which makes complete sense given that our knowledge base of of Pride and Prejudice comes out to roughly 176,000 tokens and 400 * 440 is 176,000 our chunks then are a little bit larger since they are now counting tokens instead of characters with this 400 number and we can see that looking at the first chunk it comes out to exactly 400 tokens similarly you can also pass in a token overlap and get some of the same benefits that we discussed when talking about the character overlap and this time we see that we have 878 chunks and at a chunk size of 400 we're still getting 400 tokens but then we also see we get overlapping text of about 200 tokens so as a step up from character-based chunking token based chunking makes a lot of sense sense will be counting in a format that the language models tend to actually process in but one more step up from that is a recursive approach and so what these different recursive approaches understand is that when we're actually writing text we tend to actually naturally separate it into paragraphs sentences or other logical units in a way we're already naturally splitting up some of the main information into chunks ourselves when organizing and formatting different unstructured writing exercises and so this approach aims to take advantage of that fact by initially looking for different natur separators and creating initial splits at those to start the general process that it follows is it'll first take a look at the entire document and then initially break it up into paragraphs by these double new line paragraph breaks this creates the initial set of chunks then what it does is for each chunk that exceeds our specific size limit whether that be a character size limit or a token size limit it will recursively process them using Progressive smaller separators and it follows this order past the initial paragraph break if the chunk is still big it will then try to split the text by line breaks if that is then still big it will split by sentence boundaries like periods question marks or explanation points if those are too big then it'll split by words and then finally if no other separator works it will then split at the individual character level one of the little nuances here to actually call out is that most of the time people are accessing these different text Splitters via the Lang chain library and one of the differences that chroma DB's research actually implements that Lang chain does not is that in their recursive text Splitters they only use these following separators whereas chroma implements a few extras with the different sentence-based separators their claim here is that the original implementation would commonly result in very short chunks and performed poorly in comparison to the Token teex splitter which is why they implemented a few of these different separators so in the end what this does is the splitter actually tries to preserve as much of the natural structure as possible but it will drill down into the smaller separators when necessary to meet the different explicit size limits that we set so a chunk that's already small enough stays intact while larger chunks get kind of progressively broken down until they fit so looking at our first example of aun size of 800 characters with no overlap using these separators according to the research we can see that we get 1,270 chunks and all of our chunks are actually pretty much nicely going to fall usually on sentence or paragraph separators which makes sense given that looking back at the actual text itself of the book it is mostly formatted in these nice short paragraphs here but one of the things that we should call out and look at is actually counting the length of one of these chunks if we look at chunk 200 it's actually 635 characters which we said that we wanted a chunk size of 800 characters but this is where we're going to start to see a little bit of a discrepancy between our defined sizes and the actual lengths of our chunks themselves and this is because following this recursive approach this is more so going to find where paragraphs or sentences more naturally end rather than actually forcing us to follow Strictly To the Limit that we set which in this case is the 800 character limit this is because this chunker approach prioritizes maintaining the natural text boundaries over hitting the exact maximum sizes but it will make sure that we don't exceed the maximum sizes checking out the recursive character splitter with an overlap we we start to see something also interesting it looks a little bit of the same but even though we specified for an overlap once again we find that there is no character overlap and this is because as we just mentioned the chunker still prioritizes these natural text boundaries when making splits and so when it does all of these splits and has clear brakes to create the chunks under the size limits the chunker will actually use those brakes rather than forcing an overlap however for the times that it will need an overlap it will do that overlap and we can see that this actually happen because looking at the number of chunks with overlap we see that it's 1,536 versus are no overlap of 1,270 so as you likely expected after the recursive character-based Tech splitter we can also apply some of the same ideas but just using a token count instead of a character length count this will follow the exact same recursive process of splitting on kind of the natural separators from writing and then just looking at our length and tokens instead of characters so you can see very simply 800 tokens no overlap we start to see these chunks of a little bit longer because it's 800 tokens and then of course we can provide some overlap and do exactly the same thing just instead of characters we are looking at tokens nothing super new there just a different length function so moving from the regular just length-based or recursive and then length-based approaches if you've been around and built out some retrieval augmented generation flows it's likely that you've heard then of Greg cat's semantic chunker this is a super popular approach that Greg here introduced in his Legend AR five levels of text splitting notebook here which I will of course have Linked In the description below but in this he introduced the idea of semantic splitting or being able to actually use embedding models to find the similarity between chunks to better group them together the main idea is that through using embedding based models and comparing different chunk sizes together we might be able to better find the natural semantic bound in text while maintaining different consistent chunk sizes and so the way that this works is that the chunker initially splits all of the text into small fixed size pieces around 50 tokens each using just the standard recursive splitting that we've seen with separators and then for each of the pieces it's going to consider the surrounding context which in this case is going to be three of the little segments before and after the current segment to understand its local meaning mean what happens then is that we actually use the embedded representation of each of these segments to calculate their cosine similarity or the cosine distance between them in the vector embedding space what we can then glean from this is understanding that if we see a higher distance between a segment and the next segment that would suggest some sort of natural topic transition and might make a good splitting point so at some sort of threshold of the similarity that we can set we can actually have it automatically hit a split it then uses a bit of a sliding window approach so it'll go through all of the different chunks in the document from the first to the last and then at all of those thresholds when calculating the semantic similarity between each segment insert a split within chroma's Library they actually modify this very slightly by implementing a binary search feature essentially the issue that they're trying to approach is that the original implementation could produce very unpredictably large chunks as it slides through the entire document but what chroma aims to do with the binary search implementation is keep the actual chunk sizes to some sort of specified length similar to how we specify the amount of tokens or the amount of characters that we want we're still taking advantage of the semantic similarity calculation but also taking into consideration the chunk size that we want to maintain which the original implementation doesn't necessarily do to quickly show the difference between the original implementation and the modified implementation from chroma I loaded the original from Lang chains integration with the semantic junker ran it over the documents and we can see that we got about 305 chunks here are the 200 and 20 and1st chunk as we continue to look at but then looking at the actual number of tokens of each of those we see that it's 577 for the first one and 511 respectively for the second one we didn't actually pass in any sort of argument to specify the size or the length of the chunks that we want and from what I've seen and actually using which this is the one that I tended to default to before going on this deep dive is that it can create unpredictably small or unpredictably very large chunks using chroma's modified version then we can pass in the average chunk size that we want to Target and then running it through we can see that we ended up with 434 chunks looking at the token count of the 200th that comes out to 2011 tokens so this is very good to see because if you want to keep down your chunk size for either your embedding models maximum token input or even your language models maximum token input you then have the ability to do that targeting with chromas modified while still actually being able to take advantage of the idea behind the similarity and semantic similarity based approach for our next chunker the cluster semantic chunker one of the things if you're paying close attention that I said about the regular Greg comrat semantic chunker is that it actually just looks at the local decisions when doing its semantic splits because it has the sliding window approach and just goes straight left to right through all of your chunks you're only going to be looking at the immediate window that you're observing the researchers at chroma realized this and wanted to have some sort of approach that has all of the benefits of the semantic similarity based chunking but actually looks at it from a global optimization approach using some fun fancy d damic programming techniques essentially the cluster semantic chunker is going to try to consider the relationships between all pieces of text simultaneously to then find the most semantically coherent groupings while also maintaining those aformentioned size constraints the process that it follows then is it is going to do the traditional recursive splitting with token sizes into those 50 token segments and then instead of just analyzing the consecutive pieces what it does is it creates a huge similarity Matrix and embeds each one of those pieces and calculates the cosine similarity between all possible pairs of these tokens what this does is it gives the chunker a complete view of the semantic relationships throughout the entire document not just in the small sliding context window then what it does is using the massive similarity Matrix that it puts together it uses some fun dynamic programming that you can of course find in the code to check out the optimal way to actually group these pieces into chunks what it does then is that for each position in the text it'll try different possible chunk sizes and calculate a reward and this reward is based on the total semantic similarity between all of the pieces within that potential chunk then what it does is by building up from each one of these small pieces and saving each one of the intermediate results it efficiently explores the space of possible chunking to find a global Optimum of where these splits should be put chroma also of course considers the size constraints which are inforced by limiting the maximum number of pieces that can actually be combined into a chunk with the max cluster variable and so then within this limit the algorithm is of course free to create these chunks maximize the semantic coherence which should hopefully lead to more natural groupings than approaches that actually only look at the local context since what it can then do is recognize when pieces far apart that might have maybe a slight deviation in the text are still overall closely related what I've put here to expand on that a little bit is that with the local or original method we might miss different opportunities to actually group related content that's separated by potentially a brief topic shift the cluster can see all of these relationships in the similarity Matrix and thus understand when even if we do get a little bit of a dip in similarity from one segment to another segment if there is a following segment that's very similar to the original one it will still be able to group those together so this is of course a natural kind of evolution of our regular semantic approach but taking into consideration all of the text in your document running Pride and Prejudice through the cluster semantic chunker with a Max chunk size of 400 tokens we come out and see 992 chunks and a few of the examples of what that looks like here the final chunker that we'll take a look at is actually a bit bit of a funny approach because we're going to be directly leveraging large language models with an approach that literally is just you know why don't we just ask a language model what the best way to chunk text might be the way Chromo approaches this is that they use the traditional recursive token text splitter to create manageable chunks of roughly 50 token pieces for the language model to analyze they wrap those pieces with special tags like start chunk one and end chunk one at the beginning and end of these chunks to maintain the different identities of the actual split segments throughout this process collections of these segments that add up to approximately 800 tokens or so usually containing multiples of these small pieces are then shown to the language model and we do some instruction which we'll show The Prompt in a second here to actually identify the natur natural semantic break points and so the language model what chroma found is that having it directly repeat out the chunks actually tended to perform very poorly as the language model would not very accurately respit out the exact text so instead what they have it do is state where at each of these chunks it should be split using those numbers that the language model outputs we then just combine those split up chunks back together move the window and progressively move through the entire document splitting based on what the language model thinks is the best place to split the actual prompt that's being ran through the language model is this one says you're an assistant specialized in splitting text into thematically consistent sections the text has been divided into chunks marked with start and end where X is the chunk number your task is to identify the points where splits should occur such that consecutive chunks of similar themes stay together it then asks to list the chunk IDs where you believe the split should be made and gives a bit of an example and then it says what the form should be so so passing our document in and using the GPT 4 language model to do all this we can see that it runs through about 4,871 iterations of this prompt and once all of these splits are analyzed and once it's all put back together based on what the language model believes is the best we get back these 671 chunks and the first couple look a little bit like this I find this to be quite an amusing approach since of course if we're going to be using a language model at a final step to do some sort of generation why not just see what the language model might naturally expect to be the different chunks themselves this might be kind of a perfect approach but what we can actually do is literally determine what maybe the best potential splitting option is since chroma did some wonderful wonderful evaluations on this so after all of this what really is the best chunking approach for retrieval augmented generation systems chroma put all of these to the test at different sizes and overlaps that are available in this table along with much more if we scroll all the way down yeah we can see a lot more different specifics in the actual research report here but their overall summary here shows that for recall the actual language modelbased splitting approach tended to be the best and then for Precision Precision with perfect recall and intersection over Union metrics that they measured we actually saw that the cluster semantic chunker with a size of 200 tokens performed the best there a few details about what these metrics are actually measuring and what they tell us the first is that chroma notes that this is a token level evaluation approach which differs from the traditional information retrieval metrics which those tend to focus on different document level relevance and rankings but this token level approach makes it much more suitable for actual retrieval augmented generation systems and so what each one of these does as a very high level is recall is going to tell us what percentage of relevant tokens were actually successfully retrieved Precision is going to tell us what percentage of retrieved tokens were actually relevant Precision with perfect recall is the theoretical maximum Precision score that assumes all chunks containing these relevant tokens were successfully retrieved it essentially helps us understand the inherent token efficiency of a chunking strategy and then the intersection over Union metric is actually going to measure the overlap between relevant and retriev tokens well accounting for redundancy IOU is very similar to the jacard similarity coefficient so check this out if you want more of the specifics of why that's being measured digging a little bit deeper we can see for the best overall performance we saw that the cluster semantic chunker with 400 tokens achieved the second highest recall at 91.3% while maintaining decent efficiency the llm semantic chunker achieved the highest recall overall with average efficiency metrics and then the cluster semantic chunker with 200 tokens actually achieved the highest Precision Precision with perfect recall and the highest IOU a few of the findings that we can glean here is that actually reducing chunk overlap generally improves our IOU score by reducing the Redundant information retrieved one of the funny things here is that open ai's actual default settings for an 800 token splitter with a 400 overlap actually resulted in below average recall in the lowest scores across the other metrics so I used to default in my examples to using open AI um file assistant parameters with this but it probably won't be anymore and we also saw that smaller chunk sizes of 200 to 400 tokens generally performed better than the larger ones adding overlap also between chunks generally decreased efficiency metrics while only marginally improving the recall perhaps the most surprising finding here is that the recursive character Tech splitter with a chunk size of 200 characters and no overlap actually performed consistently well across all of the the given metrics this is pretty surprising because it is one of the more simple basic straightforward approaches to actually splitting text that we have given all of the different approaches and things that have come out that we've just gone over here so if you're looking for and need a straightforward lightweight and easy first approach to splitting your text that one might be actually the best approach as such we can say that from chroma's research and from looking at these that if you're building some sort of drag system you might want to use the recursive character text splitter with a 200 to 400 token chunk size and no overlap for a simple yet very effective and manageable solution but if you do need the maximum performance and can handle the additional complexity you might want to check out their novel cluster semantic chunker with the 200 to 400 explicit token size one of the things however that was noted about the cluster semantic chunker is that you do need to actually re-embed the entire document or knowledge base every single time if implementing new text into your knowledge base so maybe just the simplest solution was actually the most effective solution all along that being said a lot of great wonderful insights from chroma DB per usual always love their technical reports and the content that they're putting out and putting it to the test here I sure learned a lot and will be taking a lot of these techniques back into my day-to-day development and hope you might be too with all of the different techniques that chroma released here they also provided a whole evaluation framework so if you want to see a dive into that let me know in the comments and if you like the video leave a like on the video if you want to see more and support the channel consider subscribing thank you and have a great day