RAG for Complex PDFs with LlamaIndex: A Hands-On Guide

[00:00:01]hey Chris did we find anything curious about doing stuff with complex PDFs I I would say so yes we did find a few curious things uh it turns out unsurprisingly that it's difficult to do things with complex PDFs and the tools that L index has today are pretty good uh for a lot of different file types but still fall a bit short for PDFs yes I believe industry has run into this issue the question is does the embedding table unstructured retriever pack do what it says on the tin uh for the most part yes especially as as long as you stick to the uh the the kind of suggested file types and it does a great job uh absolutely oh man definitely looking forward to that so the exclusive silky smooth perfect oneclick Auto magical solution for PDFs it still doesn't exist and we knew that coming into this PDFs can get crazy complex it seems like this is still a work in progress for the industry is that right Chris that's ultimately correct yes yeah okay well I'm pumped to see how far we can go with what data types and how we can build pipelines with our own PDFs in a more streamlined way than ever before that's what you've got in store for us today right you know it well I'm really looking forward to this one I'm GNA outline for everyone all the core constructs we need to start building with these tools and with these ideas we'll see you back in just a little bit to lead the data demo all right everybody thank you for joining us myself founder and CEO of AI maker space and Chris alexic the llm wizard co-founder and CTO at AI maker space for another YouTube live event if this is your first time welcome we'd love to hear where you're joining in in the YouTube live chat what you'll learn today is you'll learn how to build rag systems with really pretty awesome indexes for your data that contains both tables and text special thank you to llama index and unstructured do. for partnering with us on today's event we're really excited to highlight their tools for all of you today if you have a question throughout today's presentation please drop it in the slido link in the chat description or go ahead and click the link in the YouTube live chat right now all right let's get into it so let's talk rag for complex PDFs what we like to do at AI maker space is we always like to align our aimim for the session if you stick with us what you'll get out of today is you'll understand how the core constructs of llama index stack up to Advanced retrieval methods and how we can sort of build these ideas like Lego blocks in our our mind so that we can deal with more and more complex llm applications you'll learn how to deal with unstructured text natural language data and structured tabular or table data at exactly the same time using this very very cool unstructured doio functionality that llama index has wrapped and made easy for all of us to use so we're going to talk first about rag as an overview I know we've got a lot of new folks in the room today so we're going to do a quick overview of rag we're going to look at some llama index core constructs but we're going to get into the PDF problem as quickly as we can today we've got a lot of great stuff to show you so rag it's just important to sort of put this into perspective as we get started it's trying to solve the problem of hallucinating that llms have they're confidently telling us things that are false this makes people very uncomfortable especially in Enterprise and the way to solve this is the same way we solve it in real life we want to make what we're doing fact checkable we want to make sure that we're providing references for everything that we say we want our llms to do the same thing references matter and so retrieval augmented generation is all about those references that matter in improving Our Generations by using them retrieval is about finding references augmentation is about adding those references to your prompts and then generation is about getting better answers because we added those references to the prompts that's all there is to it retrieval augmented generation now this is useful as a first cut very very useful in specialized domains where you have lots of jargon and lots of words that maybe have slightly different meanings than they do in other contexts and our example today is going to be in the financial domain domain so you'll notice that uh this is another good candidate for things that you want to try to start teaching your applications ways to understand that specialized language of course you can fall down the rabbit hole of specialized domain language and start fine-tuning your embedding models we did that in a pre prev llama index event we won't be doing that today but we will be hopefully able to engage with some specialized language that's in the PDFs we take a look at rag from a system architecture perspective can be looked at a few different ways I like both the linear visualization and the sort of highle viz from llama index the idea of you have to build your indexes first you're going to load all your data you're going to figure out how to build the indexes you're going to get everything ready to rock and roll before you do querying and before you start engaging with rag I think this is a very useful mental model for rag systems of course once you start getting queries and responses and you're doing these Generations you want to be able to evaluate are these high quality Generations how can we improve them are there things should be measuring Etc so I like this also just because it sort of shows a lot of different data types that all go into this idea of index or indices or complex indexes that are managed in very very unique ways this is really where llama index shines as far as we can tell uh brighter than anybody else in the industry is when you need to do complex things and really really get your data into the right order for the best possible results so we ask a question we look amongst our index or our indices we add that relevant information to our prompt before we go ahead and put it into the llm and get hopefully a better answer one of the ways we like to visualize this is we ask a question we provide a query that query is to an embedding model that embedding model then turns our query into a vector format that vector format is then fed into a vector database and we're looking for similar things this is just a vector similarity that we're looking at something as simple as cosine similarity to get nearest Neighbors from there and it's important to note here that even when we have quantitative and qualitative information like we'll have today ultimately everything is being converted to vector format in the examples we'll see today and so this overarching idea of looking for similarity within a vector database is still holding you can imagine um as you get much more complex in the way that you're storing different data types perhaps it's not always vector format but it is for today's examples and then we're going to go ahead and set up our prompt template our prompt template we might make sure says something like use the following context to answer the user's query you may not answer the users query unless there is specific context in the following text if you do not know the answer or cannot answer please respond with I don't know some good oldfashioned prompt engineering there to T us up for when we do find those similar reference materials we shove them right in to the prompt and hopefully improve our Generations so the simple retrieval process the the simplest retrieval process big idea is you ask a question you search a database for stuff that's similar you return that similar stuff and of course since we're dealing with Vector databases we're going to talk about taking our documents splitting them into chunks creating embeddings for each chunk and then storing those embeddings in our Vector store so now that our documents or sort of PDF level documents let's say are being chunked we're just sort of doing something a little bit more complex here where we're taking the data that's language data we're taking the data that's tabular data and we're chunking and indexing in very powerful ways and we're providing very powerful patterns that we can leverage in our applications that we are building today with llama index llama index again it's really shining on the data side as far as we can tell and that's represented straight top level in their documentation a data framework for llm applic ations it's all about the data the data Centric AI Paradigm does not go anywhere just because of llms so llama index is all about understanding that llm applications Shine the most brightly with private and domain specific data as you flavor them that's how you get your competitive advantage of course data is often all over the place in Oran ganizations the bigger the or the more all over the place the data is generally speaking and getting it all to the right place for the right tasks and for the right applications that's what llama index helps us do so to understand this new capability we have to understand the core constructs of llama index the first of which perhaps the most important for us to understand and get familiar with the language of are nodes now nodes are quote First Class citizens in llama index they're just chunks of source documents okay so we talked about chunks a node is a chunk but node is a specific llama index term node is going to inherit the metadata from this the parent document in this case document means sort of PDF level document and in this case we're doing just a simple query where we're looking for the top K nodes that once we find those similar nodes to the query they'll be fed into a response synthesis module and we'll get an output of course we're going to do something a little bit more complex today but this idea of node doesn't go anywhere as we start to stack more and more interesting pieces together and the way we create nodes is we parse our data and we use node parsers we go ahead and we take our documents we chunk them we eventually have node objects lastly it's important to understand in llama index that the Retriever and the query engine are really the things that this is where the magic happens in llama index so what you want to think about is you want to think about sort of that retriever that helps F fetch the most relevant nodes and the query engine that is allowing us to sort of interface these two pieces you'll see a lot of retrievers and query engines in the code today these are the two pieces that really are are sort of the most interesting pieces to pay attention to as you start working with llama index there's so much power in these Retriever and query engines um we're consistently amazed as they come out with with better and better ones all the time the important thing about rag is that it really relies on retrieval to be done well and done well could mean a lot of different things in terms of evaluation but what we want to think about is we want to think about if we improve retrieval then we improve generation okay and so getting retrieval right again it's that data Centric piece it's so important we talked in the last event that we did high performance rag with llama index about how to do a couple of things to go from simple to Advan we talked about fine-tuning embeddings and we talked about using small to big retrieval we talked about a sentence window retriever to do that and this is one of the Llama packs that have been recently put out what we're going to talk about today is we're going to talk about how to go ahead and deal with embedded tables and this is the embedded table sort of llama pack this is a notoriously hard problem to date everybody asks about this everybody wants to see this capability rolled out in every tool out there on the market um why though why is this so important you know it's kind of like the VP in your pocket problem this is sort of The Meta middle manager problem the CEO is going to go to the VP of Any Given function or organization in the company or department and they're going to say hey how's it going in your department or function and in order to figure this out there's a ton of words that need to be read but there's also a lot of data that needs to be looked at and maybe there's some summary documents that put together some of this context that we know is done on the analysis side some context that was done just on the qualitative thinking through things and how they connect a strategy side but a lot of this answering of these types of questions how are things going in area X area y area Z these could really be streamlined if we could look at both qualitative and quantitative data at the same time very easily and the simplest way to think about doing this is within one single document so this is sort of the ground floor of this bigger overarching problem consider the annual report this is a document people work very hard on each year for every public company out there this is called a 10K filing and it's submitted to the Securities and Exchange Commission the SEC this is is a super important document and it's a great one for us to think about today because it's got text it's got tables and it's got tables with data that really matters if we get it right it doesn't have a lot of other crazy other things like super complex diagrams or you know electrical wiring diagrams or simply schematics of how to let's say put a piece of furniture together or to fix Ace pie of equipment it doesn't have a lot of other pieces like actual images that people have taken and added to the document that you might see in technical reports or service manuals it just has tables and text and some of those tables are pretty complex in their own right so this is a great use case for us to start with because if you're going to ask questions about uh an annual report as a stakeholder or as an investor you want to make sure that you're getting the right answers back so this is the use case we're going to look at today how are we looking at this how does it actually work well the magic here where llama index has taken a tool from unstructured doio comes in the form of this embedded table llama pack and what it allows us to do is this higher or iCal parsing this idea of text splitting of tables it's historically hard but can we take that text Data that's unstructured and the table data that's structured yes we can in just five lines of code so what we do is we start with a summarization of the table it we check out the table we create a summary of it we're cre creating a schema for the table at the same time this sort of structured table summarization allows us to put that together then we use hierarchical node references to sort of take that summary that was created and a link to the table and then plug that into a recursive retrieval process that is a lower level process in llama index and one uh that's been out there for quite some time so we're sort of again as we solve more and more complicated problems we're sort of stepping up layers of abstraction and layers of you might think metadata within our indices to allow us to solve these more complicated problems with constructs that we already have the way we can visualize this is we can look at this unstructured element node parser that's going to be collecting nodes that are just text and also collecting nodes that are tables you see this summary and schema here this is what's being collected finally this is fed as I mentioned into this endtoend recursive retrieval and what this recursive retrieval is doing is it's allowing us to explore the most relevant nodes and explore the relationships that these nodes have to additional retrievers and query engines so at query time if the index node is fetched now we're sort of combining the idea of indexes and nodes into one construct called an index node then the underlying query engine or retriever will be queried so now we're sort of again stacking one example of the way we might think about this and and look at this is in this sort of contrived example but one that allows us to sort of see how this recursive retrieval is working tell me about the sports teams in Boston and the positive aspects of Seattle the subquestion query engine is now breaking down sports teams in Boston into question one and positive aspects of Seattle into question two the recursive retriever is then figuring out okay I'm going to go and send this question to this index node which underlying it has a query engine and retriever that can be used and the positive aspects of Seattle of course go to the Seattle index node so this this idea of sort of going to the right place for the right thing is all we're really talking about in this recursive retrieval process so for today's build what we want to do is we want to go ahead and grab some data we're g to get 10K filings from who better than to look at but Nvidia and as we get this thing ready to rock and roll the first and most important piece is to process our data using all the constructs that we've seen so far we're going to bring up the llm wizard himself to show us how to get this done and get ready to build out the rest of our rag system whiz over to you man hey yeah okay so hopefully my mic is a little bit uh better for you guys uh but I'll just try to speak up if it's not so the idea here is that we are going to be doing a conversion step now uh there are some methods we can use that rely on OCR but the actual performance uh of those tools was a bit lackluster compared to this so this is the solution we're going to use which is basically we're going to convert our PDFs into HTML files in order to be able to parse out the uh the table information uh within and uh definitely still not the best but uh hopefully if I'm speaking a little bit closer to the mic here I'm louder if not I'll just yell a bit more so the uh the idea is we are going to uh yeah we can definitely zoom in here we're going to go ahead and we're going to add some additional information first we need our open AI key and then we're going to grab this llama pack now the Llama pack that we're using is the embedded tables unstructured retrieval pack so this idea is that you know when Greg's talking about those five lines of code this is one of the ones so uh the idea is that we're going to use this pack in order to power our uh our actual information so you can see we just grabed the pack from the uh Hub and then we are going to store it in this directory now we have to do some data pre-processing so we're going to use a very specific Tool uh which is going to be this PDF to HTML ex if we look at the actual uh link here it's actually a very good tool for this exact uh purpose very good at converting the uh you know the actual information we want into the desired format so how do we use it well we just have to grab the package and then we install the package and then we can run the package as a subprocess in our jupyter notebook so we're going to convert this quarterly Nvidia PDF which you can see here and we're going to convert that into an HTML file and that's going to be what we use with our actual uh index and query engine and all we need to do to set that up is actually call the embedded tables on structure retriever pack on that HTML and provided a path to save our nodes that we've processed and that is it that's the whole process to begin querying these uh structured uh you know documents so uh let's uh at this point go back to Greg who will talk us through the next uh section Yeah Yeah Chris so so cool to see exactly how easy that was and you know just in case you're not tracking here completely this is the embedded table functionality and this is the process that as soon as we get that PDF into the right form we're going to go ahead and be able to take that unstructured and structured data do that structured table summarization and connect that to hierarchical node references to do recursive retrieval that connection to recursive retrieval is what we have to build out next and that rest of our rag process is what we want to focus on building out next but the data piece once we get it ready to rock and roll you saw how easy it was and so we really hope to see a lot of people start building shipping and sharing some amazing things with this new data processing because the rest of this is going to be pretty straightforward if you've built rag systems with Alma index before the models we're using today are models that you're going to be comfortable with if you're building these tools at all today open AI gbt 4 open as Ada embeddings of course you could plug in any open- Source embedding models or chat models here and we're also going to be leveraging in terms of our tooling we're going to be sticking with straight up llama index here so we're not going to get fancy with the vector database we're going to go ahead and create Vector store using llama index and we're going to go ahead and do the infrastructure and um if we wanted to do evaluation we could easily do that in llama index we're going to leave that out today but what we're going to do is we're going to essentially glue all the pieces of our rag system together with llama index we're going to put open AI models in and we're going to now leverage the data that we've chunked appropriately we've created those nodes we need to do that hierarchical indexing that then we can retrieve in a more sophisticated way so with that I'm going to pass it back to Chris to show us how to do rag for documents including PDFs with tables the rest of the way Chris yes you bet okay so I will continue to uh speak at a high value uh of our volume sorry and uh first of all I just want to show kind of the output or the understanding of this uh specific uh tool so with the Llama pack which is uh we're going to walk through kind of step by step what that looks like underneath the hood uh we have the following uh you know uh output so we can ask questions that are simple like revenue and get responses that reflect the revenue uh we can ask questions about revenue from sales outside of the US which we're going to get a good answer for because it's represented well in text we can talk about any policy changes you know AI is going to uh change the actual policy so that's fine uh you know they are worried about it that makes sense uh when it comes to uh asking questions that are a bit more uh generic or uh hypers specific we might get a lot of noise especially with the default retrieval pattern which is going to be quite low so we'll want to modify our llama pack now the Llama pack is just a loose abstraction of the uh of the actual code that we'll see in the next notebook but the idea here is straightforward enough we're going to load the path to our HTML file we're going to parse that in an unstructed element node parser and then we're going to build our index based off of this mapping that we can create and then we're going to use our recursive retriever now what we want to do in addition to this is perhaps modify the amount of retrieved documents or modify the llm we're going to do that by actually modifying the code that you see in uh if we go to our uh files in our base. py we can actually modify this code and then re uh repport it and that's going to let us get uh a better understanding of you know this text perhaps so the idea is we're going to increase the amount of retrieved documents as well as we're going to uh move through the uh a better quote unquote model so that we should have better performance so we'll just repport the tool from our copy here and then we will run it again we can ask the same questions about policy changes we get a better response because it's gb4 uh turbo we can also ask questions that are uh quite specific to only table information I mean one of the things that we want to be able to confirm is that we can ask questions about only tabular data and receive an appropriate response and indeed we can ask the question about the uh research and development expenses and percentages of net revenue and the only place that you can find this information is in the actual table uh and it gives us the correct answer for both periods of time so this is showcasing the ability to uh operate in that cular only format uh and that's the the big idea so how did this actually work uh well it's pretty straightforward and we mostly saw it uh in the other notebook so uh we're going to first just again convert our PDF to HTML and then we're going to load it in using the flat reader from llama index we're going to parse the nodes with our unstructured node parser this is uh using local inference so it's not going to the API we are using the uh raw nodes to get nodes from documents and then we're creating a map in this next step then we are going to go ahead and we are going to uh grab the actual Vector index store which is going to be constructed of those base nodes our retriever which we're going to again uh increase the top K retrieved documents uh past the default one so we can collect more information which can be used to provide a better response and then uh the actual you know query engine that we're going to use is this Vector query engine uh which again we're going to use that similarity top k equals to 3 we're going to use the recursive retriever from llama index in order to do this a little bit better uh and then we are going to uh provide the mapping to our nodes as well as our Vector retriever uh and that's going to essentially be the whole process from here we can move on to asking questions about the PDF in the expected format but the big idea here and if we'll go back to the other uh notebook the big idea here is a lot of this comes down to the efficacy or quality of this uh processing process right so this conversion process now again we we tested with the OCR uh implementation and found them to be a little bit underwhelming and this was the best format that we could come up with uh in terms of actually being able to answer questions about that data that's represented in our uh note or in our PDFs that's not just uh you know raw text but that actually has some kind of structure or structured representation if you wanted to move to some kind of process that uh examined Graphics or charts you would want to move to a process that involved some amount of uh you know multimodal language model uh but for the actual e e efficacy uh this is the strongest uh solution that we found through the Llama index tooling and it does for the most part what it says I think with a few adjustments at least uh you know actually modifying the model we're using as well as increasing the amount of retrieved documents we're able to get really good responses about the text representations of information as well as the tabular only represent representations of information where we're actually leveraging that structure in order to understand more about our data and that is how you do rag with unstructured and llama index uh on a complex PDF that's it very cool Chris so you know it looks like that's that's a wrap for today it looks like these HTM and HTML files they work out of the box PDFs take a little bit of massaging right now but llama index and unstructured do make it easier than ever to do rag on complex PDFs and PDFs with tables so you know get your hands on these tools start building with them today start making an impact at your company or in your startup or whatever it is you're trying to build next these are certain to impress and many more data types of course exist images diagrams all of this stuff is going to continue to roll out on the market you'll be able to see us sort of testing each step of the way but there's no reason to not try to start building with these tools today if this is a problem for your customers for your stakeholders in your organization and so with that we're going to go ahead and move to questions and I'd like to invite Chris back up on the stage to uh to join me for the Q&A period yeah Chris what's up so man it it did it did what it said and for the most part yeah I mean we have to take this kind of loopd loose step uh of conversion and I see a lot of questions I just want to address a few of them in the chat before we move to the slido um the so why are we not using the unstructured partition PDF is the big question right so because of the way that that's set up it's actually not compatible currently uh with the the the tool stack that uh is existing in llama index even in the Llama index demos that reference PDFs they're actually just using HTM files I think it's uh we're waiting on unstructured to get a bit more of a robust tool before moving on to actually using their PDF uh partition um technology uh and as for all the questions about privacy or the API uh the uh API to unstructured is not being used it's all being uh done locally but the actual you know llm is open Ai and the embeddings are also open AI so you would need to use uh locally hosted solutions to keep everything private you could do so by again modifying that that base. py file we saw to run your local uh pipeline but outside of that uh you know there's nothing else other than the open AI components that are going to be leveraged through an API or an external API sorry in this tool all right all right so it looks like the first question that we have in the slido so everybody go ahead and please upvote your favorite questions in slido is is there a way to do hierarchical chunking on PDFs such as document section headings subsection headings paragraph chunks this kind of thing yeah definitely I mean it's a it's a more uh engineer intensive uh thing to do so it's not going to be like an out of the-box tool which is what we were examining today uh you will have to build that uh and build those kind of like node graphs in terms of their hierarchy but LL index does give you the tools to be able to do this uh it it just does take some work so creating that metadata creating those sections as your parsing through your PDF files and there you go all right now how important is the recursive retrieval uh it seems a bit unclear Islam asks can you sort of give your take on this Chris uh it it's it's not super useful but it is useful um recursive retrieval is good because it lets us peer into uh deeper components of our of our uh of our node graph uh so it would not work as well without it especially the way the unstructured is creating these entities right so we are we're leveraging that unstructured node parser to build a a fairly not like super complex but fairly complex representation of our data and so we're going to to be uh the the recursive retrieval is a good way to let us kind of dig deep into specific sections of our of our document as they have been uh parsed by our uh our unstructured tool Islam also asks how much of an improvement is there using the HTML compared to directly parsing the nodes from PDF uh yeah like if you could use directly from PDF in a in a in a in an acceptable fashion right now that really preserved that structured information from your tables uh I would say uh you know there would be not so much improvement but right now uh that doesn't work so good so there's a huge Improvement in going to the HTML representation first and then uh you know building your your index yeah and that was one of the key things that we found as we investigated this tool this time around is you know that that's an opportunity to reduce some friction potentially for you open source contributors out there you know if you want to jump in and help out with this I think that that's a really really nice piece to try to maybe bite off and start contributing to one of these projects on all right then what are the differences Chris between llama index and Lang chain like the retrieval QA chain how do you look at these tools and um when should we pick up one or the other yeah I mean uh they're great tools they're both great tools uh you know they're going to be able to do a lot of the same things uh especially because they're both constantly implementing new and awesome things uh and I mean what it comes down to is just your preference and your comfortability with the apis uh they are all going to go ahead and eventually leverage uh you know uh something that's similar on the back end right we're constructing these these representations out of the box though I would say like llama index does take a lot of care with their idea the concept of nodes and building these node graphs I would say like llama index for these more complex representation tasks is probably going to give you a a smoother outof the-box experience with less engineering hours than Lang chain might require nice nice nice yeah and um and you know this is a question that everybody wants to discuss all the time we love sort of discussing these things within our community if if this is something you're interested in keeping up to date on as the space rapidly changes there's lots of things that one week one tool might be slightly better at than another tool and the next week we see it equalized again so really really interesting time to be in this space and to be asking questions like this so Martin asks do you believe that combining knowledge graphs with Vector database retrieval provides substantial improvements especially for Big Picture questions y yes uh knowledge graphs are a very rich way to store information and with tools to navigate or uh Traverse those graphs uh is yeah uh they are very good uh at those things you know if we can represent our information in a more uh information Rich way then you know uh those those kgs are going to be hugely hugely advantageous for us yeah definitely an interesting space to continue to look at so tactical question can this be used with outputs generated from AWS text track text track table extraction uh yeah I mean if if you can put it into a standard file format uh or a any kind of uh you know usual representation then yes uh it can be used with this uh if you have some kind of uh you know alternative method of EX of converting the PDF to an HTML file or an HTM file that will also work fantastically I saw a lot lot of uh a lot of people were were talking about using adobe's API great solution uh we didn't want to use a uh external API today but uh there's a lot of really cool uh external uh you know managed PDF 2x tools that exist out there next up how do we get amazing PDF parsing without having to call unstructured doio Api their API I think you already covered this um and it is ideal to do all the work locally can you be super clear about exactly what you did today yeah I mean it's just running locally uh for for the example that we went through today that's that's the solution um you can run it or through through your API but uh you can also do local inference uh there's like a local inference package uh that you can import and leverage to do that without hitting an API now this is run locally and the next question question is would this solution actually work in a scalable Cloud infrastructure yeah I mean it depends on what you mean I yes of course it will work uh it it will it will scale like every other uh rag application uh it really comes down to what the different components you're using are how you're leveraging them but uh I mean this doesn't have any inherent scaling issues that are unique to it uh so in the same way that it's difficult to scale complex rag applications generally uh you know there's no new or not new challenges for this task yep yep yep and so another couple of sort of maybe rapid fire questions here about how we can augment these systems one can this work with llama 2 other open source models or just open AI GPT uh it can work with everything the the model uh is agnostic to the rest of the stack the model receives text information though please do be sure if you're using a open AI functions uh query engine or anything like that it will not be compatible and then uh on top of that the performance is not going to be guaranteed uh to be at the level of say GPT 4 Turbo if you're using like a 7B model uh you know with fine-tuning maybe you can get them up to par but otherwise it's going to be a bit tough yep yep absolutely and you know it reminds me if you're building these things build with GPT build with synthetic data if your data is behind somewhere and you can't get it out in public and then go ahead and check that Baseline by changing to an open source model before you put your private data into that we've got a couple of questions sort of jumping up in the slido here Anonymous do you dump part parts of the table for the llm as context or the whole table any special format uh we dump Parts the table as context with the unstructured tool yeah nice okay so it looks like we've got some other movement here have you tried to implement rag conversational memory and not just question answering uh yes so you can basically just uh keep track of all of the text that your llm has sent and your queries as a kind of conversational history uh object uh both all the popular Frameworks have like a specific memory object for this that you can use to maintain kind of a thread throughout the conversation nice okay so if you were going to work with csvs here and Excel files instead of table within other files uh would you use this tool or would you would you use a different approach how how would you go about that with csvs I would definitely use another approach csvs are already structured they're structured in a understood and known way uh so we would not need to use a tool like unstructured uh we could use CSV loaders and CSV node parsers uh to achieve the same result or we could even take it a step further and do some kind of uh you know CSV uh you know tool that's going to let us actually calculate you know aggregate statistics about our csvs or more so it's it you know when we have these known file stru structured file types we we want to stick with using tools that are very performant with them uh and uh we we don't need to use generic Solutions like what unstructured is doing okay okay uh Ali asks and and maybe this is a point of clarity that we could add for everybody would it be better to convert the tables into a natural language description like table to text explanation and create embeddings for them yeah I it depends on what you mean by better but I mean that's it's a great that's a great thing to do I mean small part we're doing parts of that by parsing through a table developing some kind of metadata about it right or describing what the table is about what information it contains and then using that to help us with our retrieval uh but I mean for the most part you know if we care about the structure of the data then we really want to keep the structure of the data and in order to describe all that structure you're talking about a lot of tokens right so uh I would say like uh in terms of like out of the box performance it's going to work quicker but it's not going to work as well uh if you just say parse a table into some uh unit of text yeah my my sort of intuition was we're kind of doing that uh already in this tool and then you know I was thinking of the example we had uh recently where we're looking at service manuals and trying to look up part numbers and and I'm just imagining sort of a part description part number table converted to text and I'm thinking is that really the best way to solve this problem so I think again it comes down to sort of it depends and and yeah conserving that structure of the data might be very important LAX asks any tips on scaling let's say 10,000 documents how about a million documents what do you think Chris many many many many documents yeah I mean the more documents you have the more expensive it gets the harder it is to retrieve the specifically desired information and so the more complex your system needs to be in order to ensure you are actually retrieving what you need versus uh just the first thing that you find um you know things like graphs can help us to do this things like really uh appropriate metadata application filtering and so on are also going to be key in helping us navigate these huge document spaces but ultimately it's just very expensive uh it is uh yeah yeah it's just very expensive and uh that's what it comes down to yeah and you know in order to sort of try to decrease the cost as much as possible in order to decide how to do that metadata filtering Etc you really can start to dial down okay what are the questions I want to be able to ask and then say do I really need the 10,000 documents for that if so which parts of them do I need exactly and you can start to if you chunk it down into specific tasks you need to be able to do which AKA specific questions you need to be able to answer things you need to be able to get context for um that's going to allow you to do it in as efficient a manner as possible uh can you explain the node mapping process that you did uh basically we're just constructing a kind of hierarchical node structure that we can navigate based on the kind of thing this is in a document uh and that's that's about it okay okay mapping based on the kind of thing this is in the document um what what do you mean by that exactly the the you mean the data type yeah or what unstructured has classified this as being part of so like uh you know spans versus paragraphs versus X versus y uh we want to understand the kind of text this is the kind of uh you know uh document component that it is uh so that we can better understand how it fits in with the rest of the data what things are relevant to it or not relevant to it is is it relevant at all to us right so like like a JavaScript how we don't really need that stuff right so uh all of this information is parsed out for us uh in a way that makes uh that makes sense uh but we need to map those nodes in order to understand what unstructured has done for us okay okay right so as you start to get a little bit more into it you may need to drill down into what some of these tools are doing exactly to decide on how things are being parsed but yeah I mean if you're just applying it let's see go ahead can we get some out of the box uh results that are really good so we we got a question about Pi PDF here what do you think about Pi PDF Chris uh it can it handle the PDF the same way as unstructured does all P PDF uh you know pimu PDF all of the PDF uh extraction tools are dope and they do what they say they do which is extract text from PDFs the thing that they're falling behind a bit on is struct uh you know getting that structure out of the PDF uh and that's why we've used the the process we did today but yeah I mean Pi PDF is great if you just want to rip text from uh from pretty clear PDFs you're I mean that's going to be uh something that you can do well uh to be clear though this is PDF file format not scanned PDFs or images of PDF FS uh that's not going to fly yeah okay so um Charles is asking us to transition to talk a little bit about evaluating the quality slash you know how good this thing's doing is it hallucinating is it returning the right stuff am I getting good Generations um how do you think about evaluating the quality of of this particular system that we built today Chris yeah I mean llama index has a lot of great out of the box tools that you can use to evaluate uh not not trying to just be like the Llama index uh you know event today but they do they they really do things like ragus uh are great tools to understand how your rag pipelines are performing uh you know it is it's very uh it's a very difficult problem because there's not like a perfect answer especially for a complex task like this where we want to be able to say like okay so what's our retrieval from the text of the document versus the tables of the document do we understand the table structure well enough uh these kinds of metrics are harder to uh are harder to understand or harder to get information about uh and there's always going to be this loss right for if you do a conversion process from PDF there's going to be a loss if you use OCR there's going to be some loss right like it's not going to be perfect and so we're always going to lose information when we're going from the PDF file format to a non PDF file format or even just extracting information directly from the PDF itself uh and so uh we want to be building systems that are sensitive to that kind of loss you know things like do we lose structural information I think is key and using tools like ragus llama index LM as a judge Frameworks is going to be a great way to get that information uh and and understand you know how good or how bad that system is and uh more importantly will give you directional signal when you make changes to see if you're getting better or worse all right I think the uh the last question that we'll end on today and feel free to Riff on this for a little while is uh are retrieval augmented generation and document question answering the same exactly Chris what do you is this is this a one to one that we should be thinking about here no kind of so they I mean it's kind of like one is a subset of the other one right so document question answering is a subset of retrieval augmented generation or at least we can think of it that way right we have documents uh we are able to retrieve them and then make Generations based on them so this kind of idea of document answering is is almost inherent almost inherent in that rag pipeline uh it doesn't have to be though rag can be used in ways that are exclusive to uh question answering so I I would say like they're they're oftentimes used synonymously but I would I would more think of the the retrieval augmented question answering or Rocka as we like to call it uh is is a subset of what actual rag is and is doing yeah I like how you brought that up we we teach retrieval augmented question answering as sort of the rag 101 in our llm operations course and the reason is is because this sort of subset idea if you're going to get started with something and you're getting into this field it's very very useful for you to think about rag as simply a question answering machine although doing retrieval and augmenting generations is a more General process and one that we can use to dial in other things if you go back to sort of classic NLP question answering is just one task that we can do with a generative model so you kind of you kind of get back to basics when you start to get into is it just question answering well no but it's it's a useful lens through which to begin learning it well Chris I think we're going to go ahead and wrap up that was an awesome Q&A thank you Chris appreciate so the demo today and all of your insights for Q&A thank you everybody for joining us today and for your participation thank you to llama index and to unstructured for supporting the event and for letting us take a good hard look at those latest and greatest tools this brings us to the end of today's event and if it resonated with you and you'd love to build more cool stuff with us definitely check out our Discord our community calendar and our upcoming courses on llm engineering and llm operations from deep within large language models to actually deploying them in Frameworks like Lang chain llama index and cloud computing service providers at scale we sort of cover as much as we can of the space to try to keep you up to dat all the time join us on YouTube live every week where we're investigating a latest and greatest tool or we're diving deep into a classic principle that everybody should understand if you have feedback on today we'd love for you to share it with us please fill out the feedback form in the YouTube live chat but until next time keep building shipping and sharing and we will most certainly be doing the same thank you so much everybody we'll see you all soon