Vector Search Explained: AI's Role in RAG Systems

[00:00:00]I thought we'd take a look at vector search, which is one of the techniques used in modern chat systems to find data and use that data when answering questions. Right? So, we know we talked about retrieval augmented generation in a previous video. In that video, to try and get the language model to answer a better question about I can't remember what it was I was asking about. I think it was a football stadium. Let me tell you about the amazing Olympia stadium in Berlin, Germany. I just jammed the whole Wikipedia article for the football stadium into the prompt. Now, that's okay, but what happens if your you don't have a Wikipedia article or what happens if your company records are thousands or tens of thousands of documents? How do we find a good document to help answer a question? If I say, where in our back catalog of documents is the the thing about how me Sean promised to pay me for doing this? Um, how can I find that?

[00:00:55]Right.

[00:00:58]The process we use is something called vector search. And it's actually quite cool because it's it's similar in in in sort of in principle to the video we did on Face ID. Right.

[00:01:05]>> Uh okay. Is that where you had to unlock your face with your phone?

[00:01:08]>> Exactly. Right. And it's painful. If you want to unlock a face with your phone, you won't let me forget that one.

[00:01:14]>> Sorry. It's just >> um I in that what happens is you train a neural network to look at a picture of someone's face or a depth map or a liar image of someone's face and put their face into some embedded space where they are near or far away from other people's faces and you can log them in if they're near to what you think they're meant to look like. Right? This is exactly the same but it's now for sentences and paragraphs or words rather than for faces. Right? So let's imagine that I ask a question. So I ask a question like why is the sky blue? Question mark. Now everyone knows of course it's because of rally scattering right and this is also quite a common example we used in AI. Um now let's put aside the fact that most big models could probably answer this because there's sufficient information in the training data. And let's pretend this is a question where we want to seek the answer of this from some data set or paragraphs of text that we've got. Let's say we've downloaded a huge encyclopedia. We've got all these texts.

[00:02:12]How do we find the bit of text that answers that question so that we can paraphrase it and respond? So, what we're going to do is we're going to embed this sentence, the tokens in the sentence using a language model into an embedded representation. So, let's just draw our representation in two dimensions because that's easier. So, this is dimension one and this is dimension two. And they're completely meaningless, right? And the sentence, why is the sky blue? Is over here. And maybe you ask a different question like what is an atom? Might have to ask one of the people from uh 60 symbols because I'm not actually sure. Um the what is an atom? And this will come out somewhere else, right? And the idea is that the network is it's been trained on enough text that it's contrastively learning to say look that sentence is not like that sentence right in this in this space.

[00:02:57]Now the problem with two dimensions is that you quickly run out of space, right? So you need to have many more.

[00:03:02]But the key to this is any paragraph that said something like because of rally scattering. I think I spelled it right. Apologies if not. The hope is that these two sentences are semantically similar, right? And these two are not. And so when we embed this bit of text, we come out somewhere over here, right? Which says, okay, we don't know that these are the same sentence or that they say exactly the same thing, but semantically the content of this sentence is similar. That should have been purple. How embarrassing. I didn't fix it. This purple embedding is close to this green one. So now let's imagine you were building a system where you've got hundreds and hundreds of paragraphs or thousands or millions of paragraphs of text. Some of them might be useful, some of them might not be useful. What do we do? Well, what we do is we store all of these embeddings and we can retrieve similar looking documents. So you ask a question, why is the sky blue? and embeds that into a numerical space and finds let's say the 10 nearest points to why is the sky blue. Right? Now, some of them might answer the questions, some of them might not, but we then use those to look up the original text and we put that into the prompt of our language model as well as well. So, instead of just dumping the Wikipedia page in like you don't know what you're doing, uh you smartly find relevant pieces of text. And this is an unbelievably powerful tool. It's it's it's also very very common, right? If you're inserting things into a large language model, vector search is a huge hugely useful part of that process. It's not practical to stick millions of documents into a large language model.

[00:04:39]And if you do that, you might not find the right bit of text anyway. Let's let's make the problem a little bit easier. So, how does this process work?

[00:04:46]Well, we train this model using something like contrastive learning, right? where we basically give it lots of examples of text that is similar and lots of examples of text that's different and teach it to put them in different places. Right? This is the same way that clip embeddings work when we did our video on clip embeddings. But that's for images and text. This is just for text. You can't you can't control what you're going to find on the internet. Right? So absolutely this is called vector search. Right? So you might have a database of many many different numerical representations and the text text one text two dot dot dot dot right it's a huge lookup table. So you you find your position, you find the these embeddings that are close to it, and you get the text, right? Um and then you just got to create this database.

[00:05:33]And to do that, you just find a load of text, you embed it once, and you store those embeddings, right? So it takes up quite a lot of memory, but these embeddings are not that big. 128, 256, 500 dimensions, right? Just 500 numbers in a row. Now, this is quite a nice way of doing it. I know I've been grumpy about large language models and AI hype and things in some of my videos, right?

[00:05:52]But actually this is one of the things that I think is really impressive about large language models is you never have to get it exactly right. Right? Which is a useful thing in this case. So for example, if I ask why is the sky colorful but the answer is probably similar. Right? And so but it might move this slightly but you might still get interesting bits of text or why is the sky blue but I spell blue incorrectly.

[00:06:15]It will be close enough right and the large language model the embedding will be in the right place. So you can make typographical errors or grammatical or or use poor grammar when you talk to a large language model because the way that it interprets these tokens that kind of is ignored, right? Or at least overcome. And that's I think quite a useful feature. It's one of the things that people like about these um these AI systems is that they offer this kind of natural language way of interfacing with something like this database. If if I ask you a question but I foul up my sentence, you'll probably get the gist of what I'm saying, right? Maybe I try and ask ask something in French. My French is not that great. Someone who's French will probably get what I'm saying because I get enough of the words correct even if it's in the wrong order.

[00:06:57]>> So looking at that on two dimensions, that's pretty straightforward. We can see where those X's are. But even if it was just three dimensions, one of those X's might be 100 miles away in the other direction.

[00:07:06]>> Yeah.

[00:07:07]>> How do you cope with that?

[00:07:08]>> Yeah. Um Okay, good. So there are a few ways you can measure similarity in an embedded space, right? The the common one is is something we call cosine distance, right? And it's the angle between two vectors. So if we imagine this is the this is the origin, right? N this here is a vector. It has a d1 and a d2 or an x and a y. This is another vector like that. And I nearly didn't have to bend it. Right? This is the angle between those vectors. Whereas this vector here, you can see the angle is much smaller. So in two dimensions, the cosine distance would just be what is the angle between these two vectors?

[00:07:43]What is the angle between these two vectors? Right? And we normalize it typically between let's say n and one or minus one and one. Now in three dimensions it's actually exactly the same. Do you think I can draw that? I mean we can give it a go. So in three dimensions you'd have something like I'm I'm regretting doing this already and I haven't even started yet. So you got you got this and then you've got this coming out here. Um, and then you've got a vector in 3D space like this one. And you've got another vector in 3D space like this one, which for the sake of this diagram is more angled that way. It's still the angle between them, right? And which is going to be between n and 180. In four dimensions, I'm afraid I can't draw on the page. In five dimensions, I can't draw on the page, but the calculation is exactly the same.

[00:08:29]So why would we do that and not just measure this distance here, right?

[00:08:33]because we've got essentially 500 or or or three numbers. They are a point in 3D space. They don't have to be a vector.

[00:08:41]Why would we measure this angle? Well, the answer is that it then basically degenerates everything. Everything is of length one, right? They're all on a a big circle like this, right? In 500 dimensions. If you can visualize a 500 dimensional circle.

[00:08:54]>> I try sometimes, right? It might be that for just for reasons of network output, this vector is actually really really long, right? It's the same angle as it was before. But what we're saying is the length of the vector is unimportant. The magnitude of this vector is unimportant.

[00:09:09]We only care about how where of the direction they're facing. And this essentially it reduces a bit of sort of error and noise in your process. Right?

[00:09:17]You could do it this way, but you'll find that maybe a sentence is just like a bigger version of what it should be and it ends up putting it somewhere else which is similar to other sentences. Is is it possible though that that you've got two things that are nowhere near each other but are on along the same vector line or more than two?

[00:09:33]>> That's what cosine measures, right? So cosine is measuring whether two things are essentially along the same line regardless of how long they are. So if you had a bunch of different lines on here, they would all have a cosine similarity of one or you know no angle.

[00:09:47]>> So they are similar anyway.

[00:09:48]>> Yeah, they are similar, right? And that's kind of the hope. um these would all be measured as quite different if we were doing it based on just distance.

[00:09:56]>> Yeah.

[00:09:56]>> In in in you know this dimensional space. All right. So I've coded this up.

[00:10:00]And when I say coded this up, I've used libraries to code up. Just a very simple example to show you how this works.

[00:10:05]Okay. So I've imported some libraries.

[00:10:07]Um I'm going to create my embedding network. Right? So the embedding network is not a network that predicts the next word in a sentence or predicts what image is interesting. It just turns your sentence, however long it is, or your paragraph into a numerical representation. Uh, so I'm going to import that. I'm using hugging face to do this. And the model I'm using is mpnet base v2 apparently, right? It's it's standard one, right? So it's downloading all the model and sort of sorting that all out.

[00:10:34]>> So there's obviously there's a there's kind of like a a process of going through and turning it into vector.

[00:10:40]>> Yeah, in actual fact, the process is not dissimilar to what a normal large language model does, right? is still a transformer network. It represents the sentences tokens. It goes through all these attention layers. It's just been trained to output lots of numbers instead of what the next word will be.

[00:10:53]>> Um, so I've got three embeddings here that I'm going to create. E1 is why is the sky blue? E2 is the sky is blue due to a process called rally scattering, which I would say is similar in theme to the previous sentence. Um, and E3, which is my wild card, is bicycles typically have two wheels, otherwise they're called a tricycle or a car. And I'm sure the people in the comments will point out all the other non- two- wheeled vehicles that no um I decided that wasn't the most important part of this video. So I'm going to embed these. So each of these will go through the network and produce E1, E2, E3, which are embedded variables. So these just think of these as positions on our in our dimensional space that we can measure the distance between. If we add another um code block to this, I can say for example E1.

[00:11:41]And you can see here it is. These are the embeddings for E1. And they're quite long. They're just meaningless numbers.

[00:11:47]They aren't useful to humans. They're useful because the distances between or the angles between this embedding and others are useful, not the numbers are useful in other ways. So if we scroll down, I'm going to calculate the cosine distance between E1 and E2, E1 and E3, E2 and E3. And as far as I know, this returns a number between 0 and one where zero is they are the same vector and one is they are very very angled very very far apart. Right? So let's run this. Um so I'm doing E1 to E2, E1 to E3 and E2 to E3. And remember E1 and E2 are our rally scattering related sentences.

[00:12:25]And there we go. So the distance between E1 and E2 is 0.2. And the distance between E1 and E3 or E3 and E2 is 0.94.

[00:12:35]Right?

[00:12:36]>> It's almost about as you can get really.

[00:12:38]>> So imagine you had a bunch of stat bunch of interesting facts, right? Like your bicycle wheel fact, right? Which is very interesting. In your database with these embeddings and you looked, you said we need to answer a question on rally scattering or why the sky blue. What text do we use to do that? This will find the nearest text which is the thing about rally scattering, not the thing about bicycles, right? And that's how you try and get the correct answer.

[00:13:02]>> There's one thing I'll point out is that we live in Nottingham. Very famous [snorts] bicycle factory called the Rally Bicycle Factory was based here.

[00:13:10]>> Maybe that's why this is 0.94 and not and not one. I don't know. Um but my my spelling of both could be wrong. So let's improve [snorts] my retrieval augmented generation process. A slightly more complicated example here. We're going to download and load up the NIST recommendations for key management document. Now, the NIST recommendations for key management is a great document if you're interested in key management versus cryptographic keys, right? How long should your key be? What algorithm should you use? Where should you store it? How long should you store it for?

[00:13:39]How many times should you copy it?

[00:13:40]Should you encrypt your key with another key? Right? These are all questions answered in this document and it's 170 pages long. So, let's suppose you had a cryptographic key management question and you didn't want to read the whole document. Right now, you could use the index. But but let's suppose you can't use the index for for reasons of it makes my example better. Um what we're going to do is we're going to load up this PDF and then we're going to look and here's an example page from my PDF.

[00:14:07]This is actually just I think the boiler plate at the beginning of the document.

[00:14:11]We're going to use something called a text splitter to chunk the whole PDF up into groups of about 800 words or 800 tokens. And we're going to have a bit of an overlap between them. you might you don't want your really important fact to be split halfway. So you just you just do a bit of overlap. You could fiddle about with this and change it. Right? So I'm going to chunk up my deck. So I can see for example that split 171 is something about exchange of information between internet identities, right? I don't you know who knows. The next thing for me to do is to embed every single one of these chunks into the embedded space and stick them in a a database where I can retrieve them. I'm using a database called Chroma for this. There are various that you could use. So all I'm doing here is I'm passing the embedding function by our network that we used earlier into Chroma DB. And then I add all the documents. And this takes a little while on the GPU. It's now actually processing all of those chunks.

[00:15:04]There's only about 800 to a,000 of them.

[00:15:06]It's not too many. And so now I have a query. What? Now I'm not going to ask about why is the sky blue because I don't think it's in the document. So I'm going to ask what is the current recommended crypto period for a symmetric key wrapping key. Right now, I don't know if there is such a recommendation, right? But it seems like an interesting question one could ask of this document. So, I'm going to query it. I use the vector DB. I'm going to do a similarity search that will be using cosine almost certainly with my query.

[00:15:31]And I'm going to return 10 potential par the closest 10 points, right? And the text for those. So, I run that. Doesn't take very long. The lookup these these comparisons of points is very very fast.

[00:15:42]And [snorts] then we look at the results for the rim. So the ah here we go. So the first text says the crypto period recommended original use of period of symmetric key wrapping key is that is used to wrap very large numbers of keys over a short period of time is on the order of a day or week.

[00:15:57]So I don't know whether that's I mean it's probably right. Right. It's in this document but it does seem like a relevant paragraph that it's dug out.

[00:16:03]Right. It hasn't returned anything on bite manufacturer. Um so you could actually build this into a kind of handsoff query and then respond thing. So I'm going to create a a mist 7B instruct model. All right. So I'm I'm basically just going to have a little little large language model which usually it's quite a small model. It's not great at answering questions compared to some of the bigger ones. [snorts] And I'm going to build a small pipeline where I automatically query, retrieve the data, and then answer based on that data. Right? So I've created a prompt template that says you're answering a question using only the information provided in the context. Do not use outside knowledge. And then this context is going to be populated with my query.

[00:16:45]It's loading up the the model. It's good thing I chose a 7 billion one and not the 170 billion one.

[00:16:50]>> Where's that running in the cloud?

[00:16:51]>> This is running on the on the cloud.

[00:16:53]Yeah, you could run this. It's quite a small model. You could run it on your own graphics card at home. Okay, this seems to have done something. So create my pro. I'm going to create my pipeline.

[00:17:02]Create my prompt template. Um, this is mostly using hugging face and lang chain. All right, and then I've got my rag query. So there's also a rag query.

[00:17:11]It just says why. [laughter] So I'm gonna delete that.

[00:17:15]>> Was it asking you why you're doing this?

[00:17:17]>> Yeah. Um, so the rag query says it does it first of all in it does a vector similarity search, a vector search with the query, returns 10 answers and then sticks them all into the context and then asks the language model to answer the question based on that context.

[00:17:32]Right? So that's fairly straightforward.

[00:17:33]So what is the current recommended crypto period for a symmetric key wrapping key? Run that.

[00:17:39]language model responds hopefully in a minute. The contest suggests that the recommended crypto kid for a key depends on the number of keys being wrapped in the duration of use. For wrapping large numbers of keys, order of a day or week, relatively small number of keys, it could be up to two years, right? So, we'll let the cryptographers in the chat decide whether that's actually reasonable. This doesn't sound too bad, but the point is I haven't had to look through the document. It's all happened automatically based on my query, which is quite cool. Now what happens if you talk you ask it nonsense right? Well in the prompt I've said don't answer the question if it's not in the context. So if why is the sky blue blue question mark and I run that my hope is it will say I don't know or it will say ah the provided context does not contain information about the color of the sky specifically what is blue the contract focuses on cryptography network protocols and key management right so that you know you have to think about your prompt otherwise it will start having a guess >> that I was going to say that's possibly my favorite thing about that the fact that it knows when it doesn't know >> that's an interesting point right a lot of people ask me when we talk about AI why don't they just say when they don't know a lot of the time you're not using retrieval augmented generation or vector search you haven't got this context of truth if every time you asked a question it's sourced a data source and answered based on that it could easily say the data sources conflict with what you're asking right or I don't know because it's not in the data but what they usually do is rely on their own network weights and training data to answer the question and they have a go um it's easier to get a language model to say no when you're saying compare this with this as opposed to just answer this with no comparison you see so it's about how do you develop these systems what do you want them to do when someone says something right and and it isn't an easy problem to solve if you train your network to say I don't know to a lot of stuff it will start saying it to loads of stuff even if it did know and then no one wants to use your system either right so it's not an absolutely obvious system but for building a system let's suppose you want to build a system that's more robust so maybe you're actually looking to put all company's data records in here and you're looking to find the record that that is is is you know linked with this theme. This is more likely to give you a response that's correct, but it's also more likely to say there is no such data, right? As opposed to yeah, here's the data and then literally make something up, right? Which of course is never a good thing.

[00:20:06]Add up the bites and check the sum.

[00:20:07]That's great, right? It's great except that it's quite common to get two bit errors.