Elasticsearch from the Bottom Up: Core Architecture Explained

[00:00:15]thank you so who here is using elastic search already awesome so elastic search is uh becoming quite popular these days whether it's for backing your apps search or your web search or having your applications and servers logs all in one central place elastic search is gaining lots of Mind share however it's as a search engine it's quite different from more traditional data stores so this talk is about um some how a search engine works and how a distributed one like elastic search in particular uh my name is Alex I work for found we do hosted elastic search as a service uh my background from the university is with it within search that's mostly what I've been doing ever since and through found I've been in contact with hundreds of developers and have an impression of what kind of challenges they face when they go from the basic usage of elastic search so uh this is about the uh sort of background Theory I have great experience from sharing with other developers so the kinds of questions you'll hopefully be better able to to deduce the answer to are things like uh why isn't my search returning what I expect even if I search for exact the same text as in my document or how can it make sense that deleting documents doesn't immediately shrink the index but adding documents can cause it to be smaller and why does elastic search use so much memory so before I get into the good stuff I just want to set some context around what we're going to talk about this is sort of like an agenda in reverse I'm going to first go in and then uh back out later on so when you work with elastic search you have a cluster of nodes and within the cluster you have lots of elastic search indexes that can span multiple nodes through shards and A Shard is essentially a Lucine index Lucine is the full text search Library elastic search is built on elastic search makes lucin's awesomeness available in a distributed setting so this talk is also a lot about how Lucine works and lots of elastic search documentation sort of assumes some familiarity with luine as well so within a Lucine index you have segments which is sort of like mean indexes and within the segments we have certain data structures like an inverted index uh stored Fields document values and so on and this is where we'll start so the inverted index is the key data structure to understand when you work with search it consists of two um Parts the sorted dictionary which contains the index terms and for every term you have a posting list which is the documents containing the term so when you do a search you first operate on the uh sorted dictionary and then process the uh postings so if you have this quite simple document uh you can turn you can index it by first lowercasing the text removing some punctuation and splitting or tokenizing on wh space so when you want to search for the fury for example you first find terms in the dictionary and then in intersect or Union the the postings depending on what kind of search you want so this is quite a basic example but the principle is the same for all kinds of searches first you operate on the dictionary to find candidate terms and then operate on the postings so that terms you generate that end up in your index structure decide how you can search therefore how you analyze and process the text is key when you work with search you really need to um understand the text processing that's happening so for example if you want to do a prefix search uh like in this case find everything with C starting with a c in more realistic case things like autoc completion you can easily do so by doing a binary search in the uh dictionary but if you want to for example find every term containing the substring hour you have to essentially go through every term in the index and this is quite expensive and doesn't scale but it's what happens if you for example wrap wild cards around your search so the right approach in this case would be to generate the proper terms and there's lots of different uh things you can you can do when what you have is the inverted index you want to transform the search problem until it looks like a problem where we have to find some prefix so if you want to search for suffixes you can index the reverse text and search for the reverse when there's things like G G locations Lucine will convert the data into a geohash which as your prefix is longer means more precision and something similar is done for numerical data because just indexing the string 1 through three doesn't really allow for good uh numerical range searches so even things that doesn't appear to be about uh string prefix lookups get converted to it so this ranges from the uh rather simple to the Mind bogglingly complex which we won't really get get into but it's an interesting story about how some really bright people uh came up with we can use what's called Lenin automatons to sort of go through and find uh misspellings in a really efficient way and they found a python library that they used to generate some Java code and they didn't know exactly what was going on but the tests proved it worked and The Benchmark said it was like a 100 times faster uh by by now it's cleaned up um but it's just an example of the really hardcore things Lucine will do to make things insanely fast so um when you work with search text processing is really important the inverted index is not very useful however when you want to look up the value given a document like what's the title for document number two so to do that there's other data structures like stored fields which is essentially a simple key value store where we have some a data blob that you want to retrieve when you want to render the search results by default elastic search will store the entire Json Source uh using this but even this kind of structure isn't very helpful when you need to read millions of values for a field such as when you sort or facet or aggregate because you would be reading lots of data that you don't really need so there's another structure called document values which is sort of like a column store it's highly optimized for storing uh values of the same type so this is quite useful when you want to aggregate or sort on millions of values if you don't uh specify that you want these document values elastic search will use What's called the field cache which means that it'll load all the values for uh the field in the entire index into memory it'll be quite fast to use but it'll use tons of memory so these data structures the inverted index stored Fields uh document values and certain caches uh are chunked up into what's called segments so when Lucine searches AC cross an index it searches all the segments and merg the results there's a few properties with segments that's quite important first they are immutable so they never change so this means for example when you delete a document uh the there's a bit map that marks the document as deleted and Lucine will filter it out for every subsequent search but the segment itself doesn't change so an update for example is essentially a delete followed by a reindex so keep that in mind for example if you store things like rapidly updated counters in your uh index on the upside however Lucine can use all the tricks in the book to compress things Lucine is really great at compressing data and as it turns out segments are a great scope for caches and we'll get back to why so these segments get created in one of two ways first as you index new documents elastic search will buffer these documents ments and then every refresh interval which defaults to every second it will write a new segment and the documents will become available for search this of course means that over time you'll get lots of segments so every now and then elastic search will merge them together and during this process deleted documents are finally completely removed so that's why um adding documents can cause the index to be smaller it can trigger a merge which causes more compaction so say you have uh these two segments that get merged they'll then be completely replaced by the new segment and we'll get back to it a bit later but uh this new segment will of course have cold caches but the majority of the data is in the older untouched segments at this point which has warm caches and this is key for elastic search Real Time capabilities as new data comes in uh the amount of cash invalidation it has to do is quite Limited so all this happens within a single Lucine index which is a shard in the elastic search index which is allocated across nodes in your cluster so when you search these shards it's pretty much the same as searching segments you search the all and then merge things together but at this point uh the searching can happen across different nodes and as you uh merge data here uh you need to transfer things across the network one key thing to to notice is that an elastic search index with two shards searching one elastic search index with two shards is essentially the same as searching two elastic search indexes with one chart each in both cases you are searching across two shards that is two Lucine indexes so sharding and partitioning into different indexes are two different yet similar approaches to uh slicing up your data to prepare for handling massive amounts of data you can easily feel a talk about different approaches to this but one approach is so common it's worth mentioning when you have log like data uh with the timestamp it's often a good idea to partition it into um one index per day for example this will massively reduce the search space when you only need to search today's data for example or last week's and when you need need to delete older data you can simply delete the entire index you don't have to uh Delete have documents marked as deleted and then eventually removed later on and also the indexing performance on today's data isn't affected by the fact that you have all data in other indexes so we have multiple elastics search indexes with two shards each in this case so shards are used to evenly distribute data across one index in this case because you don't uh you have too much data for one single node to cope with so when you plan how you're going to scale it's important to remember that you cannot split A Shard you can easily add more nodes and mo move data move shards around but you cannot uh turn one Shard into two while this might be possible in the future the the reason is that if by the time you realize you need more shards you probably have a high enough load that adding the the extra load of redistributing everything would be problematic so it's important to plan ahead so lots of people try to avoid the problem by okay I'm just going to make a thousand shards and forget about the problem but then you have lots of duplicated internal data structures like the the dictionary and there is also overhead to searching multiple shards so you want to have a balance between having enough and having too few so these shards get allocated to nodes in your cluster you can associate any attribute with the nodes like this node is running in data center a in a certain Rack or is quite powerful machine so you can do things like make sure there's a replica in every zone or make sure this popular index is hosted on the more powerful machines the cluster also has what's called the cluster state which is replicated to all the nodes it has things like mappings which is sort of like the schema that tells how a certain field has its text processed for example it has the entire Shard routing table so any node in a cluster knows how to route any search request so at this point we're essentially back on top abstraction wise so we'll try to piece things together by looking at how a real search request is processed so say you have this search with a query the query is of type filtered it has a simple term filter and a match query across multiple fields we also have an aggregation on authors we want the top 10 authors as well as the top 10 hits and I also specify Shard size which is something I'll get back to so this search request can be sent to any clust any node in your cluster that node becomes the coordinator for that search request it'll decide which shards to Route the request to based on what indexes you have specified to search across and which replicas are available and so on so it sends the request to the relevant charts but before the search can actually be be executed on The Shard there's a certain amount of rewriting that needs to happen elastic search query DSL is sometimes criticized for being quite verbose and deeply nested I actually think it's quite awesome for precisely the same reasons uh when it's deeply or it's nested structure makes it a lot easier to work with in code you don't have to compile this huge search string and there's also quite a close uh match between how elastic search defines it filters and queres and how the Lucine operators it ends up being converted to works so your knowledge of elastic search or Lucine will sort of go both ways one exception to this rule however is the match family of queries and the match query is something you're going to become quite familiar with because it's it's the kind of query that we look up in the mapping and see how the text is processed and as we remember how text gets processed is really important when you deal with search and quite a common source for uh pulling out hairs when you work with with elastic search is having incompatible uh text processing when you index and when you search so when you do not get the results you expect uh the um text processing should be your first suspect but the match match query does not exist in Lucine so it's a elastic search abstraction to make different things quite a lot nicer than having to do it yourself what it would actually look like uh when converted to Lucine is something like this um the match is actually converted to a bull query that uh puts together the different fields and the text holy gra in this case has been processed it has been lowercased and so on if you were to configure your match Curry differently Say by specifying fussiness this would be Rewritten to something with FY query in the in the bottom so at this point you have a Lucine query that can be run it'll be run on all the segments and at this point it matters what has happened before often you need to use the same filter or the same Fields you aggregate or sort on ac across multiple requests and elastic search will cach these as we remember per segment so assuming these two red segments here are newly created because of new documents or emerge it'll have Co caches and the filters and Fields will need to be reprocessed but the majority of the data really is in the segments with warm caches and this is sort of the source for elastic search mindboggling performance when the filter and the fields are already in the cache uh using them is really fast so uh filters are pretty much the same per um search they can be cached as a really compact bit map whereas queries are scored it's not just whether the document matches the document matches to a certain degree so queries are not cached if you need to do the same query over and over again you should probably cach it in your application layer so knowing this you should prefer to use filters when you can and use queries only when you need scoring so this is run on all the segments within the Lucine index which is a shard in the elastic search index and the results get sent back to interesting to the uh search cordin and the amount of data transferred here uh can matter a lot by default elastic search will just ask for the IDS of the documents uh for the top hit because it doesn't really need all the documents sources it just needs it for the top 10 results but this is quite different when you do aggregations uh it's quite possible that an author that should be in the top 10 the global top 10 is in the 11th position of one of the shards that's why we specified A Shard size of 100 to make it less likely uh that that happens of course it's still possible so we always need to weigh and balance the amount of data you transfer to the uh Precision you needs and this is inherent in any distributed uh aggregation so the uh coordinator has all the data it merges it together asks the shards again for hey can you please give me the source for these documents and send it back to you as the user so at this point we have been through um we've looked at the inverted index and seen how the index terms you generate largely dictate how you can search and that the text processing that generates these terms are quite uh important we have looked at how a search happens by segment and how a segment has several data structures some used when you search some used when you Aggregate and so on we've discussed the consequences of these segments being immutable and that this can affect indexing performance for example when you need real time or when you need great indexing throughput you may want to for example adjust the refresh interval so you don't constantly emerge new segments we've seen how A Shard is essentially the same as a separate Lucine index and that the elastic search index is is generally just an abstraction on top of Lucine indexes and you can combine them either in as shards in one elastic search index or across multiple indexes and at this point of course across nodes in your cluster it's a distributed search engine you can easily add nodes but you need to also be aware of the kind of data being transferred between the nodes as you search so this was intended to be an introduction to different things I hope you want to learn more about the the talk is based on an article of the same name and you can find it in Foundation which is our uh article collection about elastic search we try to keep them as helpful as possible for anyone using elastic search is just it's not just for found customers there's also a elastic search Meetup later today it's here around 6 I think so if you want to learn more about elastic search uh I hope to see you there and if you have some questions now now's your time thank [Applause] [Music] you no questions hello I have a question about replication so for example if I have some important document that I would like to search even if one of the notes or several notes go down uh what's the recommended way to do it in elastic search uh you want to you have a documents index in replicas and a node goes down uh well so I'm adding a new document to the index what's the recommended way to Ed it in such a way that one node failure doesn't take down the document yeah okay so um this talk wasn't that much about elastic search in production I used to do another talk about it there's lots of different things to keep in mind when you run a cluster of a any distributed uh system uh you want to have a majority of nodes available for example to avoid things like split brains you want to have replicas available in different for example if you're on Amazon in different availability zones to make sure that you always have a replica available when errors happen and in a distributed system um failure is guaranteed to happen so in any production configuration you should have uh multiple nodes in in running on infrastructure that's not uh have any common failure points you should uh make sure you have in bigger production clusters you should have dedicated Master nodes for example um and you need to have at least three to to have a majority in the event of failure a quite common setup is to have two nodes in your cluster one with one replica each but when when there's a network partition between these uh you cannot have a majority when you have just a single node and your cluster is composed of two so there's lots of different aspects and I'm happy to talk more about elastic search in production um after so just confine me all right thanks uh thanks a lot for the fascinating talk um would you say what would you say about the code base of elastic search is it worth reading through is it how is the quality would one actually learn something by looking through it yeah it's uh it's quite a complicated system it's I think the code quality is generally quite High uh compared to to other search systems I read Lucine has really really high quality code uh it's it's a bit higher than elastic search I'd say but elastic search is still quite quite good uh it's um you can see the fact that you now have tons of new developers which is good but it's a codebase i' uh I'd recommend looking at it's pure Java right uh yeah okay thanks a lot one over there thanks so um if I recall it right Lucine has this um proceed formula to rank documents and um how is this working between charts how are how is the ranking working uh between charts if you like having uh documents and uh term frequencies and inverted uh document uh frequencies just like you um yeah show I'll try to find the relevant slide here we still have a few minutes so um when Lucine is scoring documents it takes into account uh things like the frequency of the term for example the uh words like the and in are don't add much value but more uh rare words are considered to be more relevant and so it uses sort of like um it tries to find rare words in your query and prioritize them while sort of not caring too much about the uh really common words but of course these frequencies can uh be different across the different charts so that's it's possible to tell elastic search to as an before the search itself happens have all the shars report the true frequencies so you can get more accurate scoring but when it comes to to actually ranking and scoring I would look uh pay just as much attention to things like function score where you can boost based on for example filters you can say prefer new documents or prefer documents within a certain section of your content and so on so do not just judge relevancy out of the uh default relevancy that Lucine gives you but also look at all the tools elastic search has to tweak uh your scoring do we have any more questions is it measurable to um to compare it if you like just have a single luine index and you put all the documents uh in the um single loine index and then you have the same index across charge do you get the same results or is it different because you have statistics um between it so when uh if your data can fit in a Single Shard and you don't need to scale it for example you should prefer to have a Single Shard storing it in two shards will be more than twice as expensive so usually you want to prefer having few fewer shards um when you search multiple shards uh these frequencies can differ between the shards so you can get different results so indexing everything into uh Just A Single Shard can yield different results from having it in two uh usually it shouldn't be huge differences and again you probably want to also look a lot into uh function scoring for example thank than you any more questions people are hungry yeah thank you very much Alex please give a round of applause [Music] [Applause]