Build a Search Engine from Scratch: Text & Vector Search

[00:00:00]this Workshop that we are going to have today is an optional workshop for the course the course has not started yet and one of the reasons we actually have this Workshop is for you to learn about the course that if you didn't know that uh we have this course and this course is free and it starts in one month roughly even less I think so yeah we don't have a lot of time to actually um we we are still working on preparing the content but I will also share the link here if you haven't signed up and what I want to ask you to do is give this video a like so YouTube starts recommending it and more people will learn about the course and about this content so please take a few seconds to hit this like button and in the meantime I'll talk a bit about the course so in the course we will Implement a rock system rck stands for retrieval AO my augmented generation right which is about creating a system that allows you to speak sort of with your own data you probably have watch I will not go into many details about that you should watch a pre-course workshop that I did a couple of um weeks ago so in the course repo there's this video and code that you can check where I talk more about L let and track and so on um so there in this in that Workshop Workshop what we did we used elastic search to index the data I'll talk about the data a bit later and then we used open AI to actually send this data that we retrieved with elastic search to open a eye and get back the the response so the answer to user query and I thought like since for ra search is such an important thing maybe it's worth spending some time explaining the basics of search so this is what we are going to do in this Workshop so we will just Implement a toy search engine so it will not be a production R search engine this is not something you should use in production this is something just to understand the principles of how search works and this will be an in memory thing like in memory index so this is not something you can like let's say if you decide to run this this and your machine goes down then of course the this will not uh remember the documents so with that in mind so let us start I have already sent you the link this build your own search engine repo this what we're going to use today and the reason I have a different laptop a different tablet with me is because I can actually draw on this laptop if you took any of the other courses we have on in data doop maybe you remember this so I have a a pen and I use Microsoft Surface for drawing and many of you asked what's the name of this tool that I use it's called drawboard so I'm going to explain the basics of rck so as I said rck is retrieval augmented generation and this R stands for retrieval right and this is basically search R is search the way Rock works is let's say we have a user user and user has a question okay um and actually I forgot to talk about one thing very important thing the data set we use so in this is all here so in our courses we have a bunch of free courses and these courses have uh FAQ documents frequently asked questions documents and this is one of the documents for our data engineering course free dat engineering course and it takes quite a while to actually load this document because the amount of frequently asked questions there is huge like 300 Pages or something like that right and we ask students to do so before they ask questions about the course so let's say they have problems with running postgress and Docker right so what we ask them to do is before they ask questions in slack they go to this document and check uh like if there's already a solution if there's this question is already there because the problems students have are quite typical quite common um yeah so they will 99% probability with 99% probability they will find answers here but this document is simply pretty long like yeah so it say it's 200 almost 300 Pages it used to be I think more but it doesn't matter like you can see that it's um not super easy to find um a solution there but of course like this is what the students need to do so what we're going to do today is build a system or this is what we are going to do in the course today we will just implement the search functionality is we will use search to to find things from this document and in the course we will build a system for answering questions about the course based on the information from this documents so this is what we're going to do in the course so let's say we have a student and a student has a question I just join the course is it late I just found about the course is it too late to join right this is very typical question and then instead of somebody replying to that or student going through the FAQ document we want to build a system that answers these questions based on the FAQ document that yes it's okay you can still join and the way this system works is we have a database with uh these questions uh in the workshop before we used uh uh elastic search for that and the way it works we send this query to elastic search and remember the queries is like I just discovered the course is it too late to join right we send it to the database and then we get back let's say uh five most relevant documents five right then well we actually don't um return to the user we we send it let's say here and then what we do with this is we built a prompt based on that and the prompt is something to sent later to our llm LM so in our case it was uh open AI which we use chbd 3.5 um model right and then the model replies with answer and the prompt looks like that can look like that question then we insert the query from the user then context so this is our documents these five documents that we received from our index from our database and then answer answer 1 s right so let me just make it cleaner okay whatever and then uh what llm is doing is trying to complete the promt so it says okay so it stopped at answer let me answer this question right so this is how rack works in simple terms and as you see here we have this database this search engine so this is such a crucial thing in an system that's why I thought that we should spend some time talking about that and so here search search can be like there are two types of search at least for the purposes of this Workshop this is what we're going to cover so one is text search and another is semantic or vector search semantic search so text is like let's say again if we talk about the query uh I don't want Tope want to draw it here it's too long um actually I think I need to like I just is covered is typ of the course can I still join so this is thech query with uh full Tech search we can find all the documents that have this words discover course join right so usually teex search in teex search at least without any extra modification we're looking for exact matches so what is in the query what is in the document and in mat we will see how to implement that so this is a text search right sometimes users might form questions that don't exactly look like documents in our database so for example let's say we have a question that looks like that I just discovered the course can I still join so this is a document we have in our database but the user says I just found out about the program can I still enroll right so all the important keywords find out program enroll they're kind of the same right it's synonymic semantically so they're synonyms so they have the same meaning but they're right text search will not work in this case at least without any heuristics uh plain text search will not work but we still want to be able to discover these documents this is the case when we use um semantic Vector search so in this case we encode the query and the documents as vectors and we use Vector search for that so this is something I will talk about in the second part of the workshop uh so this is uh second part and in the first part we'll talk about this we'll talk about text search and now it's part zero so I explained what we're going to cover right now we're going to prepare the environment and you actually don't have to do it the way I do this because I'm going to use GitHub Co spaces you can use something else like let's say if you have an environment already with Jupiter pandas and secondy Le that's all you need to do you can just start the environment or start the Jupiter and follow it from there what I will do and what you can do too is use cod spaces for that so I'll call it search show but you can call it anyway you want uh it should be public let's set a read me file and we at K ignore for python I'm creating M repository and there there in this repository I will create a code space so maybe I was too fast so here I clicked on code and then you can either create click on the Plus or create Cod space in Main and make sure that Cod spaces tab is selected local not Cod spaces and right now what is happening is uh um um in the background uh a instance of Visual Studio code is created with some so basically there is a virtual machine and this virtual machine has a lot of stuff there for example it has uh Docker so let's say we do Docker on hello world it works um what I'm going to do is I have Visual Studio code desktop so I want to use it and because we're going to forward we're going to use Jupiter and for Jupiter we will need to forward a port um yeah okay I have like a special tablet that's why it's I hope that that I'll be able to continue using Visual Studio code okay so what is happening right now is I'm opening I'm connecting to that code spaces to that remote machine through what is happening I just tested it today so okay let me think what can I can do so I was going to actually do some drawing maybe I need to use a different laptop trust me it worked just a few hours ago when I was testing and let me try one more time before I you up yeah weird okay let's improvise use it [Music] different yeah we just for that actually let me see if there are questions I don't see any questions yeah if you just joined don't forget to like the video and I will what I'm going to do is I'm going to connect to the same Zoom session from my other laptop and then I'll just share my screen from there it's the first time I'm doing this nervous can you still hear me I hope you can okay so where did we stop so I'm I'm I will share my screen soon I will just open GitHub first and then share my [Music] screen okay you can see it right so this is where we stopped I will so this is the um Cod space we just launched and what I'm going to do now is click on open in Visual Studio code I have already installed um code spaces extension if you do not have it before you can do what I show is uh you need to go to extensions and then type code spaces here and then yeah just installed uh probably it will also prompt you [Music] to um to install the extension okay too bad I will not be able to draw so that's what I was hoping to do with my tablet but yeah maybe next time right now I'll just switch it off if you don't have Visual Studio code then you can install it I hope you can do that and then I just open Terminal by clicking control Tilda and then uh here we have python python uh 3.10 and now let's install the libraries we need P install you can create a virtual environment if you want but since we're in COD spaces it's kind of like we're kind of in Virtual envir environment already I mean in vual machine so what I'll install now is um Jupiter then I install pyit learn and then I install andas for the second part of the workshop we'll also need um a few other libraries um but uh for now let's just install these ones and now to run Jupiter I just do Jupiter notebook and yeah it will just run the notebook and we copy go away we copy this thing from here and notice that we have an open port so if we go to the port stab we see that the port the local Port uh 8888 is forwarded but to the remote Port 8888 is lo forwarded to the local Port 8888 and let us just uh I pasted the link I just just copy it and of course if for you the forwarded Port is different you will need to adjust the link so let's just create um notebook select me make it a bit larger and yeah I'm going to to go where it is so I want to I don't see it here appearance like maybe it's a new version so I wanted to hide this thing I hope it doesn't bother you import P does SPD just let me check that it works yeah so should be fine this actually annoys me so what I want to do is yeah it's a new version I guess so I wanted to hide this bar bottom at the top anyways um let's see so where is this um the rep one that we wanted to use so this is thereo build your own search engine and I already talked about fq documents uh we installed that we open tributor notebook and now I already showed you the fq documents and we have passed these documents and converted them to Json and this line of code goes to fetes that document uh with the request Library which I don't think I installed um yeah it's good that uh Cod space has already had the request Library um so I should have also done if you run this uh not on COD spaces let me open a new terminal so I should also have done people install requests right um so this library for fetching for for making HTTP requests and now this uh variable documents contains documents from this speciic Q documents so this is actually the question that we talked about can I still join the course after the start date and the answer is yes if if you don't register blah blah blah so this is one of the entries in that in that FAQ so what we want to do is we want to build a search engine that searches in this field in this field and in this field right and if there's a match it shows us the document and also there will be multiple matches we also want to sort these documents by relevance so the most relevant documents are first and so on and this way we retrieve let's say five most relevant documents so this is how Lucine works this is how elastic search works this is how roughly Google or work too okay and for that let me check my notes for that we need I will first converted to uh pandas so we can do this by doing PD data frame and then putting these dictionaries there documents right and then we get this uh data frame um so can see the document and here I don't particularly like the order of columns so it's first Tex section so first I want to have column then section then question and then text text is the answer that's why where it is yeah that's why I pass this extra argument columns which enforces the order right I do if I specify the order of columns then this will be the order of course section question text and let's say we want to build a bot for answering questions about data engineer C camp but we also have if I look at the tail we also have other courses like ml Zoom camp and mlop zoom Camp which might have different answers for the same questions so we first want to limit the our search space to only a particular course and with pandas it's pretty easy to do so if you know pandas uh yeah for some reason a complete doesn't work for me too live demos Zoom C so typically for pandas the way we do it is we do filtering right so if we have a data frame and we want to have a sub set of this data frame right so we use this syntax so we filter by this field course and we select only uh entities from here right only only questions about the course right so this is how we can easily limit uh the answers to the course only and now we will start with the interesting part is actually implementing the text search and for that we will use um we will use uh second learn and right now it would have been very nice to have my tablet because I wanted to illustrate something let me improvise right so in order to do that we use a thing called ve Vector spaces so let me just I'll write it here Vector spaces so Vector space uh and this is true for like search from 70s to search nowadays is we take a document that we have document is can I still join the course after the start date and we turn this document into a vector turn the doc into vectors the way we turn them could be different but the easiest so for text search the way we can encode it is let's say we have um a matrix with many many many different columns and for each column uh for each word we have in our dictionary so we have uh a column and each document is at arow so this is called term document or document term Matrix let's call term document Matrix we have um uh rows documents and columns are words or tokens uh from this so in our case um let's say if we consider only words like join course start and date we will have a column join where for this document we will have one uh for the column course we will have one and for the column start we will have one and for date we will have one right and in second learn we have um plus a Transformer that is doing exactly that it's called cizer and it's in I it learn think processing let me just copy it because I don't remember the exact Library so this is the place where it lives right so second learn future extraction text count vectorizer so count vectorizer is a vectorizer and vectorizer is a term that we use for turning texts into vectors right so this is um we have this Vector spaces so we are interested in turning our documents into vectors and a vectorizer this particular count vectorizer is doing exactly that so let's initialize it so we create a cizer and then we in terms of um Psy learn we get it we train it so what we are doing here is we look at what words in this uh in our data data set exists exist right so I'm going to fit and let's say first we want to to do it for text right so it will be a vectorizer for the text field right so we have conf Riser and then we will oh actually I have this get features names out we can see all the different um terms or tokens or words we have in our document and there's quite a lot of noise actually like every single token in this uh document is turned into this Matrix uh into this column of a matrix and if we look at yeah so there are 6,711 distinct tokens in this data set which is pretty like quite a lot and most of these uh tokens terms they just use in one document and not in others so for that we can say that uh let's say we we only care about the documents uh the ter that appear in at least let's say five documents right and then um the column names the these terms uh set is much more meaningful and we can see that it's much smaller right so we focus only on important words that appear in at least a few documents we don't want to look at all the all the documents and and here I have an example which we should probably use start using so here it will be easier for us to to see how exactly this count vectorizer turns text into vectors so let me just do it one more time I will remove this pin DF DF stands for document frequency so I will fit it on this documents I think yeah it's kind of conflict documents documents document let me call it DOC examples and I will execute this line again so now we shouldn't overwrite it and then if I look at get features out so these are all the different terms we have in our documents and now we can uh transform these documents into a matrix so do examples that's the name right and let us call this Matrix X so this is a matrix it's a sparse Matrix um meaning that for all the zeros we only care about non-zero centries so we will just we will not go into like details of what dance or sparse Matrix is but we can see what exactly is inside uh this matrix by in walking to dance so we can see that this is the first document so we so the first document contains this tokens 15th 20 24 it doesn't have the word on after it doesn't have the word and it has this word and so on so we can uh what we can do is we can create a data frame for that uh just to to for us to be able to see what's actually inside so create a data frame like that and then of course we want to know like what kind of words are here we can say columns is this so now we can see what exactly is in each document uh by doing something like that and of course we can transpose it so then for us it's easier so this is document number zero one and so on and these are the terms right so we see that um so the um so usually our matrices are not transposed this is just visually uh is more like is easier to see um so we can see that for example document number with A4 contains the word and and um yeah python is also is only in the last one um yeah right and then we see things like and for um not uh on so they are not super relevant so these words are called stop words and we can remove them we can disregard them by saying stop words English English English right so then uh count vectorizes like these vectorizes have stop words for many languages or maybe just English I don't remember but at least for English it has so when we do that we see that it removed some of the some of the words and then uh yeah so now it looks differently so we we removed some of these um words we are not interested in right and yeah so this is how it works so so this representation is called back of words so let me add a note here uh back of words so what is uh interesting in this presentation is we uh don't care about word order so we just care about the presence of the word so we know that uh this document and this document have the uh the the word course but we don't know exactly in which order this appears it's at beginning at the end in the middle uh so in this representation we lose this information so um board order is lost and typically we use uh sparse matrices for that for the representation because we have most of the records are zero right if we think about uh remember in the previous example we had 6,000 or 7,000 words of course most documents don't contain that many words so we will just have one like maybe in our case 10 ones right and the rest will be zeros that's why we use the sparse matrices and yeah so this is how we use contact risers so then we talked about term document matrices back of wordss um and then one other thing I wanted to talk about is T ADF uh so let me just do this real quick so I I will check what I implemented here so this is uh so I will do this for documents text and remember we also added here uh minimal document frequency 5 right so we have this uh like maybe we shouldn't transpose it so we have uh 900 documents and these are the words that are there right um so I will remove this this is not interesting for us but this is something we will start with so we talked about what back of fors is and now yeah maybe I should so mention that so let's say we have the word yes right so maybe yes contains in many many document and it's less important than the word yaml because the word yaml contains only in a few documents right so here the logic is the more often the more frequent the term is the less important the term is right so if I look in my query I type uh yaml then the words the documents that contain the word yaml are more important because yaml is a very infrequent word compared to some other right so for example to word yes or you tube or I know Zoom Camp maybe Zoom Camp is a very frequent term in our data set right so this way it it gives us so we kind of can understand which terms are more important which are less important and for that we use TF IDF vectorizer so TF IDF is a way to give more importance to less frequent terms and more importance to less frequent terms right and the only thing we need to do is replace count vectorizer with um tfidf vectorizer the rest stays the same and we will now see if I uh round it to two that instead of ones and zeros we have uh like you see for example 05 uh or 021 cuz uh 21 so this six appears more often than right uh so that's why it's considered more important right or the word do we have like normal names no whatever yeah um so like words like haml will have more um higher score than words like yes ah actually these are the documents right and these are the words I should remove this transposition and then so the word yes is this one yeah F I think in this data set maybe the the word yes is less frequently used than yaml because it has more importance or maybe it just appears multiple times um but you get the idea so this is what we're going to use for scoring so we'll use this tfid vectorizer so we turn award we turn a document in vectors and now each term in uh this set um of documents uh has different weights right so the more important the more important the word is the higher the weight for this uh term okay so this is an example let me actually try to copy it and see uh yeah I think I'm applying it to this uh to this one right um yeah I think this is what I wanted to show you and now we need to do the same thing so X this x is our Matrix document our Matrix and now we have a question query from the user uh let me just instead of typing uh copy it so do I need to know python to sign up for the January course right so what we do is we apply our vectorizer to turn this query into a document and this is how this document looks like um yeah it's actually like most of the the entries are zero but some of them are non zero and maybe we can actually see which of them are non zero so we can do it like this so what we are going to do is uh so names is remember is the list of uh columns a list of terms so here we will now get the these arrays with names uh we will kind of connect them let me just show it to you um so first we'll see the query I think this is not something I intended to show but let's just go with this um yeah go away so we see all the terms and I hope there will be one somewhere H yeah so okay so this is the course you see so most of them are zero but sometimes they are non zero and these are the terms from our query and the same is for the documents like for example if you take this second document remember the one about um if you can still join for the course so then I think for the word course if I will be able to find it we will probably have something like non zer2 but I guess you'll have to believe me I am not able to find it easily right and then in this case we can see that okay for the word course in both cases we have non zeros and then I see recommendation apply filter yeah I can do that but like I'll spend more time figuring out uh how to do this now than just talking about this you if you follow this tutorial along you can also um actually I was supposed to show this with this two example uh but yeah I hope you're still not lost um okay so now the idea is that if a document and a term contain the same words they're relevant so the the document is relevant to the query and because each of these uh term is weighted CU we have this TF ADF weight we now know which documents are more important for this query and then what we can do is we can just rank our documents according to this uh criteria so what we can do is we can multiply the weight of the course in this in the query with the wait for the course in Deco document and some across uh all the the matching terms right this will give us a measure of similarity how similar the query is for the document and if we do this for all the documents uh then we can score all the documents and then we can simply rank the documents by the relevant score and let us just Implement that right so this what I just told you this multiplying and summing is called dot product so if you have ever taken a linear algebra course uh this is exactly that so in linear algebra we have Matrix and we multiply a matrix with a vector and this is how we can compute this easily so we are going to implement that I will just do x dot so this will compute the similarity score and I think I need to transpose it right so right now you might feel lost because like I started talking about dot product and matrix multiplication and like all that um don't worry so here we this is a way for computing the similarity between a document and a vector and we can do this for every document in our data set very quickly because this dot product this function is super fast like this is very very fast right and then if we see if I do to dens we can see that this is the result for the first document this is the similarity for like force document and so on right so now we can compute the similarity for all the for all the documents right and this with um so let me think how to form to formulate it so I see a question what kind of similarity is this so This is actually cosign similarity so the way we compute cign similarity is we use this multiplication and then do some normalization but like for the purposes of this Workshop we can just say that in this particular case case this is cosine similarity and I don't want again to go into details of how exactly cosign similarity is implemented the idea is that it gives us a number that is doing what I described previously that says how relevant this document is to the query and I will just now import cign similarity um yeah where it is oh okay think it's from py learn where yeah where's aut complete sign similarities so it leaves in uh yeah metrix PSE similarity so this is what we need to import from Psych learn metrix pway in paral science ity and we do it this way cosign similarity between our Matrix X and query q right and then it gives us the same result right so it gives us um this measure of similarity between each document in the query now we can turn it it into a vector like that so it was a two dimensional Matrix with one column now it's just a number array with one dimmension right and this for each document it gives us the relevant score so let's call it score and now what we need to do is we can sort this and uh when we sort we pick the top the documents with the highest scores right and I'll call I'll use numpy ARC sort so so here when we sort yeah of course I need to import as P so what it gives us is instead of giving us the actual values it gives us the indices of the document so the document with this index has the highest score the document with this index has the second highest score and so on um the thing is uh it sorts um so in our case the lowest record is zero the highest record is I don't know something so this actually has zero score and this is what we are interested in so this top five so what we can do is just take from minus 5 to so these documents yeah I think I did it correctly let us check uh I look maybe I should use a a different query so let me use the query from the example I gave previously I just disc covered the course is it too late to join right and then let's see so the document number 22 text is the highest uh Maring document well I guess um it might just work course let's see the other one here's how you join slack or yeah so this one is one of the relevant ones right yes you can you won't be able to submit some of the homeworks but you can still take part of the course right so this way we find relevant documents and yeah course they not all of them are super relevant so some of them just match the words but not necessarily give us the answer but one of them actually is relevant okay and we can do this for all the fields so we have Field section we have field question and we have field text so let's just uh let's Implement a search across all the fields so I think I'm looking at the time now and it feels like it take more time than I expected but it's fine like if you don't have time now you can leave now and then come back later cuz the recording will stay here and yeah you can just come back and later P it so now what I want to do is to go across all the fields and heels and account vectorizer to that field um and this is something we're going to use for search so let's call so we will need matrices and let's let it be a dictionary and then we also need Vector risers right so matrices are this acccess and vectorizes are this CVS and then yeah so first we we fit we create a vectorizer we fit uh we Feit it so then we have the X which goes to matrices X and then to vectorizes CV goes to vectorizer CV okay so now it was rather fast suspiciously fast let's see what we have in matrices yeah we have all of them right of course text is the answer it has most of the like information it has more toking than others then question has second uh largest amount of tokens and then section is usually a small um like it's just do have it here so section is just um to which section this question belongs right so yeah now we have matrices and uh theorizers for all the fields and now we can look across all Fields right so let's say uh our query is let me copy it one more time I just discovered the course is too late to join so we can score it across all the all the fields so for example I I'll first we will create an array with all zeros and then we will Loop over all the fields and for each field we will compute the similarity and then we will sum we sum similarities across all the fields so the size of the Matrix will be the size of our data frame so we have 948 documents so we'll create a number array with just zeros of that size and now for each document we'll compute the similarity so it's again for field and Fields we do it for all the fields we get um uh first we need to get a vectorizer uh then we do fit transform of this key query we get back this Vector Q so this representation of this query for this particular field then uh our X is matrices is field we compute cosine similarity between x and q right and then we flatten it to turn it into um a vector so to onedimensional array instead of two dimensional Matrix so let's call it f score right so similarity for this particular field and then uh the total score will be score plus this uh score for this particular field right and then I have a type of it transform right and then another typo yeah of course I just need to call transform apologies we already fitted previously now we just need to do transform and I just need to execute it one more time I guess yeah so now the score is there we just do it one more time so I'll call this variable idx short for index and then data frame iock idx will give us all the um vectors or all the documents with answers right um yeah and then we already see a problem right um so the problem is let's say we want to build a system that answers questions about data engineering Zoom camp but here we have answers from other courses we need to add filtering and let us do that so we can say that our filter filters so we can have multiple filters let's say in our particular case uh we'll just have one filter for course but let's say if we have a different data set could be course and like for even for this one could be course and section right or if we have a different data set where we want to filter across two Fields this is how we can do it so add data engineering Zoom Camp so basically right now is what we do is we do post processing so we only keep the results that are interesting for us and the way we do it is for um um for field value in um filters uh values is uh what we do is something we did at the very beginning so we did do you still have it yeah remember I showed you how to filter documents only for particular uh course so this is what we are going to do now um so let's call it mask and for mask will be uh that this field equals to Value right and then uh this is a Boolean mask so if this is true then the value is true and if this is false false so this is not data engineering Zoom camp but ml Ops Zoom Camp it will be through so we can turn this into S Type int so now let me just show you how it looks like what items yeah um yeah so we see that this is the mask and we can turn turn it into so this is a pandas Series right where one is um when it's true zero overwise and we when we write values we get an ire right and then what we can do is we can multiply the score by the mask and if it's one the score stays at this if it's zero it becomes zero right so this our score at end now we have a lot more Zero Records at end right so when it does not much and now this is something we can again sort and show so here in this case we have only answers from the data engineer zom camp and I think one of them is uh yeah can I all the course of trade finishes when the course starts what can I do before the course starts yeah so actually I don't think it contains the answer to this query but let's say if it's 10 uh yeah probably it's somewhere here um right one more thing that we can do is the question field is at least it feels more important than the text field right so when I ask a question and if a term in the question field matches one of the terms in the query it's probably it makes this document more relevant than if um the answer matches right so we can give more boost more score to question rather than text and this is um this is how we do it in um elastic search in elastic search there is a concept of boosting so we give more importance to more important Fields right so let's call boosts we can can call it boosts and for example the question field will have more importance so let's say it will be three times more important than other fields and then what we can do is uh get the Boost from um from here from this dictionary and if it's not there the default one will be zero and then we can just multiply our score by this so this effectively will give more score to the score from the question field rather than to the other others and then let's say if we want to give text less importance we can also do that so then text will be will will have um only 0.5 boost which is like less than one then section will have one and question will have three I will just try with question now and then let's see what we have so course starts so now yeah actually it still doesn't look like what I wanted to see so what I had what I did before is maybe I don't do the Sorting right so what I did before is i instead of uh doing it this way I sorted the negative of the score so meaning the most important come first and then the rest and then take top high from here let's see if this improv the situation uh yeah that's actually the same results right it doesn't change anything wondering why but um yeah I guess this is just I formulated how I formulated the query um so I think these both ways of getting the most relevant questions are the same so this is how we perform search across multiple fields and we also add filtering and this is exactly the functionality we use in the workshop we already did from elastic search so in elastic search we filtered by the course field and then also gave more boost to the question field and then used section and text with no boting so now we can actually put this together in one um in one class I will not um type it right now what I'm going to do is I will just copy this uh this document so this is everything we implemented but in one class so what we do is here fit part is we go across all the fields and fit Transformers and save matrices and vectorizes and in search we do basically what we did here so first we compute the cign similarity we compute the Boost score and then we do filtering and then we do uh keep the top results and then return results so now let's just use this class I will again skip typing I will just show how to use it so first we create an instance of this class text search we say that these are the text fields that we want to uh we can call we can say index what it actually means is we can we hit count vectorizer on them right on then perform search and then we this is how we specify boosting this is how we specify filter and this is how we specify query and the number of results we have and we get back answers so and it's already uh in the same way as elastic search returns so it returns you the do the the dictionaries not like lines of data frame um so this is something that we might actually use in the course and we already put it I created a small Library called M search which is doing exactly that so this is everything we talked about today right now is put together in a simple in a single python file right um yeah so this is something you can use uh uh this I believe is a good illustration of how text search works and remember that this is an inmemory implementation so all these matrices and vectorizes are kept in memory um yeah and this is not for production is just for illustrating how text search works and I also told you about um two types of searches the text search and vector search most of the time we spend talking about text search we still have a bit of time I probably it will take longer than I expected now we'll take about Vector search and Vector search is useful when the words don't match exactly and I will show you a few things um they're pretty old so one of the things the first thing I will show you is applying singular value decomposition this is scary looking word um scary sounding word um but this is just a way of reducing the dimensionality of a matrix so let's say we have this Matrix and the dimensionality of this Matrix is it has 1,333 columns right we can reduce this dimensionality we can compress The Matrix to let's say only 16 values right and singular singular Val composition is doing doing this and trying to preserve as much information as possible from the original Matrix and I have actually a good um illustration here so this illustration is for uh images not for text but in text it's very similar right so here in this example is um so they took this picture and applied singular value decomposition and only retained uh the first in this case five vectors that come from SVD right and you see it already kind of resembles the um the original image right and then the more columns you use the more dimensionality you have the more similar it is it's kind of similar maybe to well it's not exactly J compression but you can think about that so it's a lossy compression so the the picture is not exactly the same as the original one uh but it tries to retain as much information as possible while compressing the dimensionality of vectors right and it works with images and it also works with text and I think there is a very uh old paper called latent semantic analysis from '90s which talks about doing that yeah so have it's in German for some reason but yeah so you see that if you understand German oh yeah it was first published in 19 1990 so it's a pretty old so um these things have been around for some time and I will now show you what exactly is happening when we do this so I'll just copy this as we don't have a lot of time so we'll save some time typing so we will now apply SVD so this singular value decomposition to the text Matrix right and then um so let me just do it step by step so now our X did I call it uh I don't remember vectorizer yeah so our CV is the text vectorizer the vectorizer for text and text is the matx for text so now we will do fit transform on X so which means that we will reduce the dimensionality of the X Matrix and we will if we look at the shape now shape it will have only N 16 columns so the same number of rows but only 16 columns and this is how the first document looks like so you see now instead of like a ton of zeros and bunch of nonzero records we have a dense representation so this representation of a document is called impeding uh so it's a dense representation of the document which so when we reduce the dimensionality SVD tries to capture as much of the original information from the original documents as possible and what happens then is doing that that synonyms such as like enroll and sign up and uh whatever they're kind of reduced into the same concept let's say right and similar words uh kind of grouped so this way of turning documents in dense vectors captures the semantic uh similarities between different words right and then at the end we get this representation where we can think of this as some sort of of core to what extent this document belongs to certain topic right let's say uh the topic of enrolling to the course or the topic of Docker or whatever right and uh we can do the same thing with the query so I'll just copy again the the document like the code so with do the same with query right so I take the query from the user I apply the same transformation as we did previously so this creates a sparse Matrix from this tfid vectorizer and then we use SVD singular value decomposition to turn this uh representation into a dense vector and this is what we have right so we can represent documents from our database this way and we can represent the query in the same way so now what we do is again Computing the cosine similarity so here the principles are the same except the representation of a document is different so previously we had the sparse matrices now we have dense small matrices but the principle is still the same now we simply compute the cosign similarity the similarity between the entire database that we have uh and the query right it gives us a score a relevant score and we can use this score for ranking the documents and this is uh yeah let's just see it for illustration so we compute the similarity between this vector and this vector and this is it right and we can do it across all the documents like that so we take the entire X we take the uh this representation for the query then we turn it into a onedimensional array then we sort it by relevance and then we see the answers right so the answers what we can find I think it's more or less similar so and actually like uh as you see uh if I show you maybe this one so now ah okay okay so this is we don't do any filtering here u i I was going to say you see like the the relevant answer is the first one but yeah this is for a wrong course for a different course well at least if the this system we use it for machine learning zom camp like the most relevant result would be the first one uh can I fold the course can I yeah so the answers are still relevant and also this search is actually to some extent faster uh well not always but like there are some some benefits for this right um first of all we take care of synonyms uh then uh yeah this is probably the the major one right and then we also have the spects that we can use for other things like for example let's say we want to train a machine learning model and we just can put this into our XG boost or whatever right with the sparse Vector the original one it would be much much more difficult to do this um so this is how we use SVD singular value de composition for tuning documents in vectors uh there is a bit of an issue it's maybe not an issue but more like um something to consider like if we think about if we try to interpret these terms as extent the extent to which um extent um it belongs to a certain topic let's say the topic is Docker and the score indicates how relevant it is for the topic Docker then it's kind of very hard to interpret negative values like what does it mean it belongs to like a certain topic with minus 0.08 uh uh score whatever right now we of course don't know the topics so the topics are unknown to us we can of course look at the documents with high score in this particular Dimension and see that but this is some internal representation we don't really know that but see like in in the X Matrix we have only positive values but somehow here in the SPD we have negative values and they might not be easiest to interpret right so there is a different way of um compressing this uh Matrix and it it's called non- negative Matrix factorization and the idea there is is that if the input is non- negative the output will also be non- negative and I will just copy this so it's the same thing it's just instead of using SD we use non- negative uh Matrix factorization and I forgot to add uh the Imports non non negative Matrix factorization yeah it's uh decomposition NF second learn de composition import NF and right now we see that uh we can do the same thing uh and this is how documents look like and also as a bonus we have this a lot of Zero Records which is a good thing so we can see that there are two topics to which this document belongs to to this document it belongs with this core and to this document it belongs with this core so probably this document is about this particular topic so you we can think about this as topic modeling so we have 16 different topics or concepts and this document is about this topic like 1 2 3 4 5 6 7eventh topic and Ninth topic and these are the scores for these topics right I I'll check the questions you have um soon let me just finish that uh so we do the same thing now with the with the question all right so we see that this particular query we have belongs to this to topic this topic and this topic but not really to this topic so yeah they have some intersection between this this and this right so now we again compute this cosine similarity let me just copy this thing we compute the cosine similarity and again we get that right so and this uh is actually quite good right so this is what I wanted to see right yes even if you don't register you're still yeah so like even if we don't use here question at all we just use text we can find the answer which is pretty good um so this is how like this is the simplest possible way of creating EMB bendings and these methods of creating ending thingss have been around for very long like this uh Lattin semantic analysis which uses SVD is like as almost old as me so it's been around for quite some time the issue with these things is that the input The Source it uses is back of wordss and in back of wordss uh we lose the information about the word order right and sometimes the word order is quite important right there are applications where it is important for rock I think it might not be that important especially for this particular data set um but for some cases it might be important especially for maybe more advanced Rock uses uh where queries are quite large maybe I don't know but sometimes you might actually care about the world order and uh yeah before I start the next topic so I wanted to talk about bird and creating embeddings with bird let me quickly take a look at the questions um compress Bing from CD help with that uh with memory usage um so I think I answered that so they capture the semantic similarity do we need to transform X Matrix and query Vector with the same Transformer yes exactly so like if we use um so remember how we did it here uh for this this class right so we for each query or for each field we had its own contactor Riser it's own Matrix and if we wanted to add uh let's say uh singular value decomposition or non- negative Matrix factorization then of course each of these fields should have its own uh SBT or nmf instance right so we can uh maybe have uh self ED there I don't know if this is the proper word but this is something that turns like high dimensionality vectors and low dimensionality vectors let's call them embeds and then we would need to um also say s SVD blah blah blah uh fit transform and then actually um matrices here if we only if we don't want to if we want to ditch text search completely then matrices would be like it's embedding after we embed so here we do fit transform and then X embedding would go to matrices um here here uh right and uh embedders um would have SVD right or nmf or whatever um yeah let me check if there are other questions we can try to choose number of topics yeah like this is a different topic maybe not for today but of course there are so many parameters now like you can choose different ways of compressing your Matrix different dimensionality so you need to have a way to numerically quantify the quality of the results and typically in practice we use uh methods like uh we use things like mean average Precision or Precision at five or things like this so these are uh the metrics we use for measuring the performance of informational retrieval systems and there is a very good book it's called introduction to informational retrieval it's free you can download it here PDF and they should talk about uh this relance feedback uh but like in general most of the topics we talked about today like for example this Matrix de composition L semantic indexing they are covered there right uh I don't see though uh I think they talk about things like Precision recall and different other um metrics um yeah but metrix you can Google metc for uh ranking and then you will find like in dcg mean average Precision uh yeah Precision at K recall K like maybe yeah this is something that you can check this n dcg um is quite commonly used and then you can use that to as a guide to selecting like which embed you should use how many dimensions you need to choose and so on okay um maybe I'll quickly show you how to get embeddings with bird we will not spend much time here I'll just show you how to get started and then this is something you can do after the after the workshop we will not spend a lot of time here um so what do we want to install for birt so bir is a neural network that turns uh maybe you have seen this it's called B uh by directional cond representation from Transformers yeah so it's um mouthful but basically this is a neural network that turns uh like a document into an emetic right so this is we need and these embeddings they capture not only semantic similarity but also word order and this is especially useful for things like find find out more or like this uh longer phrasal verbs or whatever um or when we have negations like don't do this and things like that um yeah so what we do do we actually need to install yeah just transformers so Transformer Transformers is a library from huging face Ken face is so this is um how to say like a repository with models so there are different models so they have uh this reposit database with models and everyone can just access these models and use them and we will use haen face a lot in the course too cuz they host a lot of uh llms there but also they host like neural um a lot of neural networks for example birt and yeah so we install that Transformers and tqdm tqdm is just um like a progress bar and uh while it's installing I will copy this piece of code so what this piece of code is doing is um so it these Transformers um usually have libraries that models that are implemented in pytorch and you can take the model for example bird based on case uh it's already trained all you need to do is just download it from haging face and then use it right then let let's execute that and torch we will because they use torch we will use we will need to use it in one case too so now we download the model it's a rather small model compared to what we are going to deal with in the llm course but uh let's see how we can use it so there are two things we unload the model and the tokenizer so tokenizer is something that turns text texts into some representation um let me just show you I think it's easier so first we have two text yes we keep the materials after the course finish so what it it's doing is uh it turns the text into this right so this is probably some indexes for example the word yes um I actually don't know what exactly but this is some representation that the neural node Network expects so first we turn our document our text in this representation and then we feed it to the neural network so this is how we actually do this so that's why I need to import torch here because supposedly it helps to make it faster so we Feit this input this this stuff into the model and this variable will contain embeddings so let's see what we have and I will just look at the shape first so now um yeah so there are two documents and then it creates two matrices uh for these documents right so this is kind of oops multi-dimensional so for each document we have a matrix this is not super convenient so what in practice we do is we just um take like let's say uh what's the shape of this so 15 and 7 100 so what we do is we sum across or take the average value across the rows right and then this is something we use as anetic this is the code for doing that so we simply take average or sum across the rows and now we have uh this right so now this is an embedding for the document number one this is an embedding for the document number two and and yeah we have negative values but like in birt um yeah if we don't think about topic modeling like we don't really care whether the values are positive or negative and supposedly it works better but again the dimensionality is quite high so for data sets like we have here this is actually an Overkill right because uh we look at our X like even the 10 the sparse Matrix is just a little bit uh yeah bigger than that so bird for this particular example would be an Overkill but it's much much more powerful like for more complex complex documents for more complex queries it will perform better and of course now we need to compute it for all the documents for all the fields so we can compute it for the field um section the field question for the field text and then we follow the same approach as previously right so we just uh um take our question take our query toize the query and then extract the embedding from this this is how we do it you can go through this code I I don't think it makes sense to go through this now so if you're interested in that just go through the rest of the um tutorial and there was more one more thing I wanted to to discuss we don't have time today a lot for that is practical implementation aspects and tools because here right now what we discussed what we did is a to example this is not something you use in production this is just illustrates the main principles of how search works right so how relevance search works so that the main idea that we we represent all the documents and the query in a vector space it could be back of for space it could be a bird space it can be embeddings from llm whatever some sort of vector right so Vector is nothing else but a one-dimensional uh num array right and then we compute the similarity between all the documents uh that we have in our index and the query right and this is the main principle of relevance search and in this particular example we had less than 1,000 documents so we can we can of course afford Computing the similarity between the query and all the documents it's simple it's very fast but let's say you're Google and you index the entire internet like you cannot like when I type latent semantic analysis you cannot go over all the documents on the entire internet and compute the similarity between that of course you need to do something faster and there are two key Concepts two key ideas for that for text search we use a posting list in information Ral or inverted index it's called inverted index and then in this inverted index there is this posting list this is something that we' seen uh uses in order to quickly retrieve documents so if you're interested in how this is implemented in practice how text search Fast text search that you can index like millions and millions and millions of documents and it still works fast um so this are the key ideas inverted index and uh posting posting List information did I say yeah so are POS list yeah there is even video like how exactly it's implemented and uh then another idea is for Vector search is lsh lsh is Lattin semantic hashing locality sensitive hashing sorry and a good very good implementation that is pretty fast and works quite well is a random projection right and actually uh random projection I think there is um random projection in um py lar so the idea here is that we take this long edding and turn it into a bit Vector with ones and zeros only right and then when it's in this form we can still compute the cosine similarity between the the bit vectors and it is roughly the same as the cosine similarity between the the original vectors right so it gives us a very powerful and fast way of computing similarity and also there is a way to uh make it faster but yeah I I don't have time to explain it so maybe you can like there is I think called lsh binning uh that this is exactly how it is usually implemented in different um Vector databases so they use lsh they use binning uh for quick retrieval for example Minash or yeah so there are different uh if you check locality sensitive hashing you will find a lot of interesting things about that so this is how it's implemented in practice in uh Vector databases and there is a good course about that I think the course is pretty old like 10 years old it's called mining uh large data sets or something like this or massive data sets massive data sets mining of massive data sets yeah so this book has a chapter about um you can guess that it's rather old so they have a chapter about lsh yeah this finding similar items and lsh so they talk how this is actually implemented in practice Yeah so I managed to cover most of it I guess not with the speed I wanted um I also wanted to talk a little bit more about how to implement uh like the main ideas be behind lsh and R projection but this will be like home reading for you if you want to read learn more about that uh let me go through the questions and then call today uh could a specific hen face and betting do better than SVD yes or maybe not like the only way to find out is to set to come up with some sort of evalation metric that is important for you we talked about them like n dcg or Precision at five or record five whatever and then you evaluate different embeddings uh and look at this course look at the metrics and the one with the highest metric the metric you care about is best right you should also look at the speed of course like if we talk about using bird then of course it's a heavy model then we need to spend more Computing resources on yeah Computing this embeddings uh and if simple SVD gives almost the same metric almost the same Precision but just little bit worse then maybe this small small small incremental change in the metric is not worth spending a lot of money on on these bird vectors vectors okay do this me do this methods struggle when the user query contains for that don't appear anywhere in the FAQ document uh what I showed you yes uh cuz we remember we fit the count vectorizer uh and if we don't see the the board anywhere in the document then the count vectorizer does not know about this for at all like it doesn't know that it exists and when we try to do the match then yeah we will not be able to match with these documents bir on the other hand is trained on very very very large um databases of text so it knows probably about all the words right so then in case of bir uh which could be like one of the advantages or other Transformer models not necessarily bir uh with them we will still be able to match documents even the if the term in the query is not does not exist in the in our database is there a way we can update the TF ADF Matrix so I don't have to append the new text Doc to the original data set um yeah I mean you can just um do uh instead of doing F transform you can just do transform and pend it to the Matrix right one way of doing that of course this is something we did not talk about in this Workshop is how do we index more documents right so we assumed here that the database we have does not change which in reality is not necessarily true right and again this uh back to the point that this is just a example to show how relevance search works but there are many many aspects we didn't talk about one of them is indexing new documents for example and for that you should use something like elastic search they can just handle this easily um do we still have the curse of dimensionality working with that product not in this particular case no yeah too many new Concepts I agree yeah I think I was lost myself a bit during the presentation and if I was lost maybe you also felt like okay what's happening uh maybe if you go over this again follow the tutorial also implement it and also try to follow this cuz I like I deviated a little bit from the document um if you just follow it maybe it will be easier for you um yeah and then also like there is a notebook that you can just use the notebook so this is The Notebook I used for preparing for the workshop and you can just follow through this uh through this notebook I think I forgot to update the r me I think if you just follow this through this notebook as I speak then um like if you have the workshop and this notebook hopefully some things will have more sense will make more sense but yeah it takes time right so I so my master thesis was about information retrieval about search so I've been around these things for quite some time uh and yeah I still feel lost like especially with all these new things like um embeddings uh like embeddings I mean like these Bird embeddings llms yeah it's a lot ofation so if you want to understand this go over uh the tutorial one more time check this notebook uh also the resources I shared will have yeah well I don't see other questions I guess most of you already left um and those who did not enjoy the rest of your day and if you're in Singapore and still watching have a good night um yeah I guess that's all for today I hope you enjoyed it and if you want to follow this tutorial and find some discrepancies cuz like I so a bit struggled today cuz like the an import was missing or something like that I'll appreciate blue requests so that's all for today and see you soon on the course and remember it starts in less than a month so sign up and this is something we will use in the course directly or indirectly and the class I implemented I showed you we actually probably will use in module 2 okay don't forget to like the video sign up for the course um send