Building a Distributed Transactional Key-Value Store with Rust

[00:00:04]hey that's to me is to see your hair today I will talk about how we use rah to build a distributed transaction a key value database before we start let me introduce myself my name is Tong Neil a Chinese guy as you can see an underachiever and a pink hair and I have been dirty by the distributed database pidb and at high kv and we also open source an hour and have developed some open source a black right with TB Comalcalco massacre in exarch and the reform issues et cetera oh that's what this is agenda at first I will give you a brief introduction about why we use a distributed a transactional criminal database the problem for us yeah I will show you the hierarchy we sought for the database and an alas I will show you how we combine them war into our database ok that's so when we want to build a distributed transactional key value database we found that we need a to will meet many many problem and we needed to conquer many many challenges that face oh why is that like when we must guarantee our database is data consistency and our database is scalable and the stable and has a high problem is and has a higher level PDT of course we must read our database follow the acid compliance and many other and a prominent we needed conquer so that night day this hour and I may have for us so what can we do in a traditional Chinese saying a high building or low foundation so there are many medicines we needed to conquer and but we can beauty from bottom to the top with different hierarchies so let's begin but at the first but as first we needed to choose our language to be late it's a rust here okay yeah you guys are familiar with this so I won't talk about these animals and us to give me so let's build that you build our database from scratch the first thing is that because we are database therefore the first problem is how we can save our data in the local machine and that we call the Achaemenid database so we are choose available story engine there are many squeal engines in a word neck rock could be like a lever DB where tiger and the internal row under you can use even beauty is by yourself but here which shows the rock to be why rock to me use a ASM tweak as you can see Alice and we had a high performance for the ride is traumatic for the random ride and had a good performance for the read and the rock became itself that loss of optimization choose to beat other rocks DB and the provided a lot of features about Mac in jetski the fire deliver and cure later the user build their service so easily on top of a rock 2 DB and a lot of reason is that of the knee is very stable and I you take it user in the my rocks and rock rocks dB so we try to choose a rock to be higher up is a project where because rock TV is a say plural library and we wrap our own roster here you can have a try if you want to use a rock to be in your water project rock 3 is cool but it can only stay with your data in local one machine if the one more train crashed oak your data is lost so here we need that you find a way to replicate our data into multi machine to guarantee our data safety of course we must guarantee our data consistency how we can do here we view the raft anybody here know rock yeah oh yes that cool ruff-ruff is a common sense distributing the constants of altruism either user interface under pickle or when the client won't even write something to the service when we were in first we were use a raft to append the current to the rough log and the end user radical algorithm to replicated a raft command into the marketing machines after majority of the load except a rough log we can sync the rough log it committed and can apply the rock to the same machine then return the read our client using rather can't help but to keep the data consistency and doodle replication automatically roughly is good but it can't fade up in the country or your data in one motion we want your data growth huge and huge so how it how we can do yeah we can when you take it out to route large and large we can submit our data into market graphic group here you can see that we we secrete our award in her as a invading he ordered hash map sorry map under sleep and it's a bit our what the data into different experience and the each clear and crisp on get to a rough group here we use again we use a mark rock to suppose that data horizontal scalability so how can I do this for example if we'll all we have three little ABC and we add a new load T and we want to transfer mu our data and reading to the data from C to the team how we can do luckily we can use a raft interlock every algorithm membership change necklace we can use a raft of membership change completely change a load directly to add a new replica of written hook and load of tea yeah we can use a rather changeable load to remove the replica from C as you can see using rough that we can we easily transfer the data from Lola - Lola D and the to dock that and the do the data horizontals gap envy very easy that's cool and we build our own the rough rough implication here it's very stable and you can even you if you want to butyl your use a consensus algorithm in your service you can have a try and you use it and you can even using in your production environment now we have Marty graphed you suppose a teeter transaction that either horizontal scalability but a lot of problem is that therefore that distributed transaction let me consider the following case for example for the bank transfer if you want to transfer some celery from a condo to a condo P but I'm not keep both a and the beer are indifferent alone so here we are indifferent the market Marty rough groups under here here we needed to come so the problem that how we can keep the data consistency quality market low crossing Marty load crossing Marty rough groups that's easy we can use the traditional two-phase commit to PC but as you as you can see that Q PC has some problems so here we use optimizer two-phase commit inspired by a Google pack later and it has a market version concurrent control and the provider server celebrity or isolation and supporter only Missy transaction I don't I won't talk more about a distributed transaction here we call it a lot relative that in this topic so I was to give it Sookie B if you pay attention to it you can search a packet at paper and the more so no seem to go the better a lot of problem is that because we are distributed at ransac distribute the database and we have many machines and many services and each services need to commune with each other through later work so how we can do this here we choose giaggi C Java C is a PC framework developed by Google and had used in many pro team art projects like cube ladies like if Sadie and the charity has many language support EDA you can use Java you can use a prop house go long PHP Python GUI etc many language supported and jive say it's based on HTTP 2 so you can get a benefit at to directly an actor actually to beat us sorry HTTP so yeah we did I show you the Jia Jie see jobs Google has provided our state official state GBC library has full API and you Larry Klein streaming server streaming and the to practice dreaming they are very powerful but anarchy this idea this API are asynchronous as you can see you writing a sink roller cold is a nightmare for us and you can even get stuck into the core core back here but luckily in Rasta we can use futures here simple simple example when we pull the Euler API you know we return a future and then we can use the future wait and all up to the future later so here we use a lot of futures to provided a synchronous API for you to the use and it will build our own job is a wrapper here if you want to use your jab using your own service to be obviously from work please have a try so Mac that the database take shape take shift so a lot of thing that we can the tell you guys can work well but a lot of scenes is that how we can guarantee our database work we're here we can use some monitor two more hours to check our database can work well warlord currently from issues is is the famous monitor in the world loud so it Italian to use a pro me shoot some provided for at a counter gauge his Roman his program and a sampler here we're only with chemistry counter cage and his program here is a simple example and you can see you can use a rigid counter to reach together counter a premiership counter and increase min the Condor we also build our own premiership decline in the rust you can have our try and this ruff class is interested in dog promotion official talk or a high or you know already so you can try have it right now finally let's go here testy you might appealing pasty is the most important thing in to build a distributed database mate and we need a queue because we needed because we are building a database and the way neither the customer trust us and we must ensure that our treat they are data in our database cannot be lost cannot be Crockett so how we can do how can we do this I think the only way is to do more you test him and the way near that you Pro do many many tests him and for your uncle we needed you you need additional unit testing integration testing and the human we can do kills testing many many more but I don't cover them all here but here we are measuring about a fair point injection testing support that your injection is filled into the fair point inspired by a FreeBSD fear and you can see that example here here we you regret will reach a selfie or winter like in the function foo and the triggers of failure outside when the program ran and enter the function panic oh that's cool so this our wrath of your pointer implication if you want to ingest I'm a failure into a service and do some funny things and you can have a try so what that's what and now we can I have making the word hierarchy about and now let's convert comment them together this is a whole architecture a war from the bottom to the top rock TV to save data using raft Hugh replicator and a doodle data horizontal and had mu CD and the trans distribute transaction API you Java C it took for communication and user promise use for Molitor that's cool so this table or the whole thing pack heavy pack TV is my distributor transactional database I'll call you the reading in raft thank you many user have user hi kV many user Togepi is using many in the product environment and as I can see I can know that one tie cubic Lassiter had allowed deploy about 140 machines for one tightly cluster in production and from now on we hello one complains that their data is lost that's cool but this is law this in law our final goal here I will ask you a question which I don't mention which I don't ask you before at for us that's why we were we want to build a distributed a transactional key values database we're not only for the PI K V we aim higher first we want to build a distribute in a relational database this is a tidy be tidy B will build my circle layer on top of a high key B to provided a distributed online transaction processing solution you can use it Heidi be not using my circuit or actually with Jionni Winslow compatible e and we was a one computer distribute here analytical database this is ty Spock we can you can run the Apache schmuck on top of attack you directly and that you provide or any way provided or distributed unlike analytic database solution so what also hear what they are we're going to go we want you to go to build a hybrid transactional analytic processing database this is a big future of war as you can see that we have a tidy be to provide an OLTP and tile Spock you provide OAP and you can even you want to use a rock a value you can use it accurate directly this is a whole picture under this is Albert I let go oh and an ultimate goal is to use a one database for your data yes that's what thank you by the way we are hearing we're coming to Tyler you were going to pay increase signature you see you asked me that but how do how we handle the deadlock in a distributed transaction that look well okay because the our per second model is optimistic or mystic transaction so if you make so for transaction along to handle a and oke a and a beer and a second be a handle along to also want to handle a and a B and we were at first we are sort of a the world Keys Mac a and a B and then commit a yet so because the committed key is not sorted so you can't be done Kellogg yeah yeah order K before commit her knees yes Christine oh you asked me that we all the ladies hold up select short isolation so how we can support serious success sorry si si SSI yes now you call a you peak as you can see we some whole MVCC and everything only for si so if we want to use support SS I will move we must measure user to write a circle like track for update expecially to support either SSI that's a that's a mediator is the only singer the only way we can do here now oh you asked me that mmm we are building or C Webber but um many other people and sorry I don't know his name but I know he's the author of mio I had pure pure Raza inflammation but the white way you still choose that you have see well her because we have we build a CL we want to use geodesy about one years ago at that time there's no stable inflammation and and there are only over the only stable inflammation either say well it's a Seiko never raised we have to use it evil and but later if you the pure wrath inflammation is coming and we can really convert you try to use a this our player oh yes you asked me which secretary I would do we support I said I'm sorry Heidi be we built on my second layer on top tech area so you can ward us so-called die lack of my circle we supported but we don't stop hold sorry we don't support wait sorry I have a tooth we support joining it's burning lots of how you might circle some holding post critical we lost somehow Hawaii we lost about procedure yeah maybe warm the circle formation we supported I know you so you can transfer you can come right your you know change your pisa lady from them my circle to my career you're in the way from my cell go to the tiny bees it generally yeah yeah you you you asked me that that high kv supporter some API for the transactional if yeah of course I because as you can see there take away much probably in the transactional API so like so like use are you like a mega you three not sorry like this user can you do likely to pay you the begin to start a career and you get using courgette and I use course add and add and commit a transaction these are all opposed to somebody I mean Peggy oh you asked me that you thought spike a visa poppy no sorry like contact someone Oh max my checkup Enoch yeah oh we have a we have a Pinot information but not in time a B you see in tidy be because you can enter at work or the circle you throw tidy be so can't hurry we can recall the circuit or the circle and do recurring our you know get in here and polity spin no it's not different it's different from the magical okay and we can use Urbino under UN you can get a pin lock to synchro synchronized order your data from the tiny be to our lab to our the reservation back sink to the Kafka or you lack search or even another tidy B we can use the pin lock yes yes that's a cool occasion Mac I don't mention Pradhan here in tiny me because relative in this topic and butter you can I'm sorry I don't make sure I tidy taking me a little only provide the time transaction API user Coolio provide and copacetic yeah well you look copacetic oh sorry is the same the concept in HBase and you can push down your some logic to the taiga me directly and do some calculation and the returner Bazaar which we already do this but I don't mission is sir yeah [Applause]