Solana Architecture: An In-Depth Explanation of Proof of History

[00:00:00]hi everyone this is Josh as you know this channel is all about Salon development but in this video I'm going to take a step back and explore the fundamentals of Solano to help build conviction for the background holding or I mean investment so I'm sure during the Glory Days of 2020 we all did our due diligence before we made a large investment into solanum now let me ask you a question does anyone actually know how Solano works or even crypto in general well let's find out by talking to our longtime friend our local cardeno show and crypto expert Scott so Scott why did you invest in crypto and get me into this well Josh back in 2017 I was a young man with hope and pride of course I would like to invest in the technology and the technology would make the price goes up so I've seen clearancy we're all clueless and really have no idea what we invested in so I thought in this video we'll explore how Salon Works and really try to understand what was it that I invested in and maybe if I should dump my bags especially now after the recent dump from forty dollars to twenty six dollars at the time of this recording spoiler alert I'm still a fan so to understand the salon architecture I had to dig into the salon of white paper their documentation and some Googling around and while the deeper I dig the more questions I had I think overall in this video I have a pretty good understanding of Solano how Solano works but there's something I missed do leave it in the comments below the first thing is no shame I took the in this line of white paper I decided to take the their Network architecture diagram so you can see uh here's the basic diagram of how Solano works on what the red arrows are pointing to we have the transaction that users are sending to Our Salon cluster so every cluster has a leader node these are the nodes that our users talk to to do the proof of History more on that later and what these leader node does is that they generate these hashes but we'll just call them transactions for now and so what the leader node does is it generates these I guess blocks in this picture of hashes which are the transactions and these combine together form a block and then once you have these blocks the verifier nodes will actually check to ensure that whatever the leader node is generating is actually a block it's not some sort of malicious block and then once the verifier has verified that the block is correct they send the end state of that verification more or less as we continue and then once the verifier nodes are finished validating through the whole block or the transaction hash you see over here it will have a final end state which then it will send back to the leader which will decide to commit this block to The Ledger or not and that's in that roughly on the high level is how Solano works I'm going to go into a lot more details of these of this whole entire process those are the first question how is a leader selected so first thing a leader is a node chosen from a cluster of nodes you know these verifiers actually and they are used to process the transaction from the users now we know our leader is let's go look at the white paper so inside of the white paper itself it doesn't really clearly describe how a leader is picked but it says right here that a election for a new proof of History generator occurs when a proof of History generator failure is detected and the validator with the largest voting power or highest public Key address if there's a tie is picked as the new proof of History generator specifically what this means is that whichever validator has the most users staking they're still on it on it will become the new leader and I thought well that's not very efficient Whoever has the most Souls essentially always the leader and so I dug a little more into salon doc and here's what I found so inside the solar documentation itself there's actually a thing called a leader schedule which is a list of identities of all slot leaders how this works is that there's a leader schedule that has created over a period of time and how this works is that each schedule has a slot where a node can be a leader and during that period of time that node will basically operate the whole Salon Network handling and processing transaction through the proof of History generator which we'll talk about later and this period of time is called epoch and so if we actually scroll down and read further there is an algorithm that kind of shows how it is done and you can read it right here but I have made a visual demonstration of this so let's take a look at how it works all right so here's a part of our Network architecture we have our leader nodes our transactions that we have our leader node which generates the proof of History transactions and then they go into the verifier nodes and the verifier nodes will processed each transaction and make sure that they are legit mid and they reproduce a final state so how we generate a leader schedule is that we take all the verifier nodes that are actively verifying the transactions and these nodes would be set into an active set and then from this active set of nodes we would sort it by the nodes that have the greatest amount of salt staked into it and then from there the leader node would generate a arbitrary scene and this seed is used to arbitrarily pick and assign the nodes in a certain Epoch for the leader schedule how the order of the nodes are picked using the C phrase was not described in the white paper or the dock so we can only assume but on a high level this is how it works so so far so good every note has a chance and my investment is safe all right so at this point we have a high level idea of the salon architecture and how a leader selection works and now we're going to the most important part the proof of History algorithm that Islam is known for so we know that later generate transactions which are actually hashes more of that later using proof of History but what is proof of History I'm glad you asked let's find out so before we talk about how a proof of History Works let's talk about what proof of history is and what is it even trying to solve so proof of History essentially is an algorithm that creates a Time ordering to allow validator nodes to determine the order of incoming blocks that the leader generates but just to be clear proof of history is not another binding protocol like proof of work or proof of stake proof of history is more of a consensus mechanism that's used by the other two algorithms to create ordering and we'll see why that's important soon so to be extra clear Solano uses proof of stay and is further enhanced by using their proof of History algorithm to understand the purpose of it let's go back and look at how approval work works with the Bitcoin Network so to mind Bitcoin we have computers randomly trying to guess a hash so that they can be the first person or node to produce a block in the Bitcoin Network and then that block is then distributed across summary node in the Bitcoin Network and that lays the problem it takes time to propagate those blocks to each validator node because as you imagine they are literally across the road and for Bitcoin to work successfully and be decentralized every node needs to both validate the block and Converge on a certain order but as you can see if we have a lot of blocks being sent out to every node in existence there could be network problems that cause us to lose the ordering of the blocks and so how this is solved is that we make the hash that the miners have to try and guess for a lot bigger and on average what Bitcoin does is it tries to ensure that it takes roughly 10 minutes for a minor to generate a block so by setting this hash difficulty we allow the validators a lot more time to receive the blocks validated and Converge on a certain order so how does prop history solve this well proof of History creates a verifiable Time ordering though we don't actually need to wait for the validators and just continue processing the transactions so in this diagram right here you see that the leader generates you know a block to generate block 1001 1002 and it just goes on and off and because we have these number ordering or time ordering all the validator nodes know the order of the blocks even if they've receive them in in a incorrect ordering and the benefit of not having to wait for these blocks is that we don't have to pay as high of a gas fee as what you might see in ethereum or Bitcoin and because we don't have any self-imposed timeouts Salon that can process any transaction as they can physically compute giving us the high transactions per second that Salon is known for so now we know what the purpose of proof of history is let's find out how proof history works for example how do we guarantee that the chain of blocks that the leader generates is actually valid and to do that we need to understand hashing so a quick recap hashing is a way to transform a any given input to a different output so in this example if we're given the word Solana we give it to a hash function Java for example it would generate the string and so there are two important properties of the of any hash functions and that's one the same input will always generate the same output and the second important property is that the output is always a randomly distributed string value that can't be predicted for example if I were to put Solana 1 I would could get a completely different output versus Solano so while it might be theoretically possible that two separate input might have the same output the chances of that is so infinitely small that it's never really a problem all right so how does proof of History use hashing so what sign does with proof history is that we start with a c phrase or not just any starting word uh let's say Solano and we take Solana and then we run it into a hash function to generate let's say hash one and then with hash one the output of our C phase we use it as an input so the same hash function and we generate an error hash value let's say hash two and then we just continue doing that indefinitely and so we get a chain like this we have our proof history generator node uh to start with seed phase it uses the hash it generates to generate another hash and it just contains doing this perpetually and the genius of how proof of History works is that because we are using hash functions that can't be predicted the only way we can get hash one is if we were to use the Solano C phase in our hash function and the only way we can get hash two is if we were to use patch 1 as our input and so by perpetually using the output of a hash function as our input we can get create a we can create a concept of time because we know for sure that hash2 must occur after hash one and Solana because hash one is used for hash2 so okay that's great so now we have a list of hashes that can give us an idea what time how much time has passed but what how does that work with transactions is this is where the beauty of proof history comes from so you know you can imagine a node might just continue its Loop of generating hashes and then whenever a leader node receives a transaction from the user all we need to do is we just take the hash of that transaction combine it with whatever hash that we're currently on let's say hash2 and use that to generate our new output which is Hash three and then we use hash3 it has the input for our hash function to continue generating so now by doing this we have solidified that transaction one has occurred during the generation of hash two giving us proof that transaction one have occurred before transaction two for example but here's a diagram of how this actually works so as you see users can send a magnitude of transactions to the leader node and while that's happening the leader node is just continuously generating hash let's say we're a hash 100 and so let's say later no we see a transaction one when it's creating hash 100. we have the hash of transaction one and we just append it with hash 100 to create another hash I'll call it hash 101 and then just continuously generate these hashes as you can see by continuing this we can have recordable proof of The Ordering of when transaction occurs so that validator knows later on and figure out the correct ordering of blocks even if they were to receive it out of order okay so we have an idea of how proof history works now we have these leader nodes generating these hashes which are combined together eventually to form a block which then we send to all the validator nodes to verify so how does that work and what are some of the optimization that validators note have that allows group history to work so quickly let's find out all right so here's where we left off so we have our proof history leader generating all these hashes to eventually become a block and at the very end of all these hashes we have this final state and what we do with these hashes in this final state is that we send it to the valedaterals to ensure that the block that the leader generates is valid and it's not a malicious leader and so the validator nodes would go through and recalculate all the hashes following the same mechanism that we used to generate it and then it would calculate the final state which is the final hash that we receive and then send it back to the leader the leader receives a majority of the same final state that it has generated it will commit the block it has proposed to The Ledger how this whole process works in a nutshell but let's dive a bit deeper into how the validator notes work so we have a validator nodes and they just received a block of hashes and so what each validator node does is it just takes all the hashes that the block generates and it just recomputes everything to ensure that chain that we received is legitimate if it's not we don't vote for it and the Block's not committed and so you might imagine that this process can be actually pretty slow because hash functions can't be predicted so we have to recompute everything one by one well that's actually not the case you see because we're giving the full chain of hashes in our block we can actually process all these hashes in parallel with each other because we know for a fact that the first hash we've received let's say hash 100 will have to give the output hash 101 which is the second value that we received so if if the generated hash 101 is not the same as the second hash that we have then we know that this block is not valid and we reject it and the reason why this would work so well is that a validator node can run this in parallel using a GPU from the white paper it said that a GPU can actually have 4 000 cores so we can validate 4 000 hashes in parallel and generate hash functions super quick so we could easily process hundreds of thousands of hash within seconds don't quote me that way though and then finally once we have paralyze and process over hashes to ensure that the final state is correct we just send that back to your leader who will then decide to commit the block or not into a ledger to explain all this in one very simple sentence proof of History works because the hash generation process using CPU has to be sequential and is much slower while the hash verification process can be parallelized using GPU cores and as long as this restraint holds true proof of History will continue to work all right so we know how blocks are generated now we know how router you know ensure that the block is legitimate so let's wrap it up all together and see how consensus work so as we all know all cryptocurrencies are decentralized it's a sense you're trying to solve something called the Byzantine fault every node does not trust each other the very real possibility especially since we're doing finances that people are trying to scam you and take your coins away so all nodes work on an assumption that they can't trust each other and so hasselater and all other proof is taking protocols work is that they require a majority of valid error nodes to agree that the blockchain related is legitimate otherwise that node is tossed away and the validator and the node.posed it is penalized let's take a quick look at this all right here's our high low diagram again we have our validator nodes they have the final State and now they're sending it back to the leader and this is how the leader handles this in order to commit the block we need to reach something called a super majority essentially that is when two-thirds of all soul of all Stakes Soul so in this case we have a total of 300 Souls because we have three value nodes and let's say each validator node has 100 so I'll stick to it so we need to have a generated final State let's call final State one that has two-thirds of all staked Soul supporting it so in this case a validator one and two generate this day final State one and combine they have 200 soul and now we have this malicious node about eight or three would generate this final state two and it has one result so because final State one has two-thirds 200 so the majority reach consensus and this block is saved to The Ledger and then the vote from validator 3 is rejected now let's go through another example let's say router 3 actually has 200 Soul instead and it continues to vote for final state two when this happens final State 1 is also rejected because it no longer has the majority and we're kind of stuck at a tiebreaker so how do we deal with these Bad actors well we have a timeout and so one one example is assuming that a battery three is not malicious so after a couple of timeouts where about it or three didn't vote for anything let's say five it wasn't really specify what the timeout value is but it's a most likely a configurable value but let's say after missing five votes then the network will decide to remove outer three but because it's no longer participating in it and so as a result instead of having a total of 400 Solana in this network so instead of having 400 stay tuned on this network we have a total of 200 now which will allow us to reach super majority so another example is we have something called slashing so slashing is that the penalizing nodes for generating invalid blocks by taking away the stake Soul the white paper mentioned two specific scenarios one is if a node voted for multiple States so for example if let's say battery three decided to send to a leader node two votes instead of the one it was supposed to it voted for some final state two and some final state three and when this happened we about error 3 is penalized for trying to cheat the system and they would get their solanus confiscated another clever way for Solana to deal with Bad actors is that they purposely send fake hashes to ensure that the nodes actually doing the work because you can imagine that the leader node would just send the final State the validator node could theoretically instead of calculating the whole hash function chain it would just say yes and return the final state that it was given it doesn't have to spend the computer resources to calculate it and to prevent this periodically if leader nodes would actually generate a fake Hash a fake final state so in this specific situation about air one and two are you know doing the correct thing and they're calculating it and they realize that this is an incorrect State and so they don't vote but we have a malicious validator three which doesn't care and they just return the final State back and whenever this happens the validator node that does that will get slashed and have their Solano confiscated again and that's the two ways that Solana handles malicious actors okay I think at this point we should have a pretty good idea of how Solano works now and in theory that can just work in perpetuality until the end of time however distributive systems never work perfectly you can imagine that one of the nodes might go down or um the leader nodes might go down and there's just a lot of problems so we need to rely on something called the cap theorem to is that how our node cluster would work in the event that something goes wrong the cap theorem is something used in the distributed system to decide three properties of how a network works so we have consistency which ensures that users will always see the same data even after an update or a delete so if I for example add 100 to my bank account I better expect that whenever I look at it afterwards that that hundred dollars will be in there we have availability which assures that all users will be able to see a replica of the data even in case of partial failures so an example would be let's say I send that 100 back into the bank if I were to look at again potentially depending on which node I talk to which server I talk to actually I might actually see my old account balance the difference between available and consistency if I try to do that in a consistency system I won't actually even be able to view I'll be just waiting for the screen to load until that update has been successful the final letter in our cat theorem is p partitioning partitioning ensures that a system continues to work X as expected even in presence of a partial Network failure in cap theorem we have to essentially pick two of these three and because we are working with a distributed system and we don't and we want to always ensure that our system continues to work we always need to ensure that we have partitioning so essentially the only choice that we can have is we can either have a consistent and partitioned service or available in Partition service so let's dive into the only two viable Solutions CP and AP City or consistency partitioning essentially an example of this is let's say we have our server and we have three databases so our server updates the data to database too database 2 will then propagate that same update to database 1 and database three and let's say another server comes along and it's trying to make a read request for that data while it is also being updated and there's different levels of consistency in this case we're assuming we have strong consistency so when this server tries to read the data it would fail to read or specifically it would just kind of stall be blocked because we don't want to read any stale data that's how CPU works now let's talk about AP or availability partitioning and so when you choose availability let's go back to that same scenario okay we are in the middle of updating data that we send to database to it's being propagated we have two servers and they're making every request the first server to database one and the second server to database three and so we can actually have two results so server one might actually read from database one which has already updated itself with the newest data so we'll get the new data but server two might read it and database 3 might not have received the newest update yet so it will receive the old data and so that's roughly a cap theorem in the theory uh so that's what so that's roughly speaking cap theorem in a nutshell so which one this line I decide to use let's find out to upgrade the white paper in a session three Network design if we scroll down this is the lovely picture I stole uh if we read this last paragraph in terms of cat theorem consistency is almost always picked over availability in the event of a partition so that solves that Solana uses CP and then if we first scroll down to section 5 5 10 when reading about availability it says a cap system that deals with partitions have to pick consistent consistency or availability our approach eventually picks availability but because we have an objective measure time consistent is picked with a reasonable human timeout and then it kind of goes into how the proof of stake works with the different partitioning side of the network don't worry about reading all this I have a diagram to show how this works all right so before we continue on I'm going to give disclaimer that I am not a distributed system expert so there are many people that are much smarter than me but I do have a a bone to pick with the white paper saying that's on is is a CP system I don't think as long as a CP architecture I feel like it's more of an AP if for example if you're just a salon to someone else I could theoretically imagine that that while the transaction is being processed you can quickly go look at the receiver's balance and see that they haven't received anything so maybe they have a weaker form of consistency I don't know but I feel like saying that's a strongly consistent is not exactly accurate but then again what do I know continuing on all right so when what to do when dealing with partitions we need to always ensure that our network is available so in this case availability what happens if we have a network issue and not all available nodes can vote now what slime essentially does is it slows down the block generation process and wait for the nodes to recover so in the white paper there are three states that were proposed so the first state is if the number of verifiers nodes that we have available in the whole network is greater than two-thirds of the network so when this is happening we can have a very quick unstaking process using a low timeout so in this diagram we have our Network we have five validator nodes trying to validate block sent by the leader and validator and validators two through five are setting the state and let's say in this instance validator one died and is no longer responding back to a leader and because we have over two thirds of our network still working for example these three doesn't matter we can quickly just unstake and remove validator one all right so in the second scenario let's say what happens if the number of validator nodes we have is less than two-thirds of the total network but greater than one happened that work well at this point because we no longer have super majority we can no longer reach consensus so what we have to do is we just have to have a longer timeout and just have the leader just generate more hashes and we'll try to wait for the nodes to come back we don't want to immediately just kick value ones and two out because if we do that we can quickly reduce our pool of steaks Alana which I potentially could lead to a malicious person taking over the whole Salon Network because they just happen to have the only set of valid nodes left and so how Solana handles this is that it slows the timers so we give nodes more time to recover and so the final state which is when we the number of validator nodes we have is less than half notebook it's very similar to the previous date that we talked about the difference is that once we are below half Network we have to wait even longer for these valid error nodes to come back because we don't want to try we don't want to consolidate but the consensus to just the two remaining nodes that are four and five but of course even if we waited for a extra long amount of time and the validator notes never come back we have no choice at the end but to unstake them and just allow powder 4 and 5 to carry on so now we have an idea of what calf theorem is and how Solana handles Network outages it'll be actually very interesting to see what brought down this line Network in the past is it actually an architectural problem with proof of history or is it something more because of just bugs that people wrote if I'm a betting man I'm willing to guess it's probably something related with bugs because so far at least from what I read from the white paper and the other documentation proof of history and Solano seems pretty sound to me so there's actually one last thing I want to talk about in this video and that while scaling the salon architecture how do we handle more users using the salon and Network I came in with some negative assumptions but was very pleasantly surprised by uh what was detailed the white paper let's take a look okay so to understand scaling and how Solana handles a growing amount of users we need to understand the concept of vertical scaling and horizontal scaling so what saw does is it uses a vertical scaling and what vertical scaling is is you can imagine we have a node it's our leader node generating hashes using proof of History and what vertical scaling is is whenever we hit a cap limit to the number of transactions we can process we just simply add more RAM CPU and GPU to our computer into our servers and this is the flat that Salon always gets all their verify nodes are super expensive and only rich people or people with a lot of soul can actually afford to even run these validator nodes but at least from a distributed System point of view on this this is actually a more ideal situation versus horizontal scaling which we'll talk about later so there are two benefits of using vertical scaling one is that it's far far far more simple than using a horizontal a horizontal scaling solution we'll see why we get to take advantage of technological advancement as you can imagine as the years go by we'll have better and better Hardware that we can add and ensure that all of our nodes can just automatically process more transactions without doing any extra work and of course as you might imagine that goes back to our main complaint which is only the rich people can afford the validator notes so what are some of the detriments first we already talked about new hardware is expensive it's not just as simple as just adding another four gigs of RAM if we were to move from 16 to 32 gigs of RAM is exponentially more expensive so we have expensive servers and the second problem is we are still technically bounded by our Hardware limitation no matter what we do until we have better Hardware we're basically capped at a certain level of transactions per second which is not good for Solana's aspiration to one day take over Wall Street so how might Solana handle this well I'm happy to report that going through the white paper it actually was mentioned of a horizontal scaling solution so let's understand what horizontal scaling is the backender for example we have one liter note so if we want to scale horizontally instead of just getting a better CPU we just add more servers so when we have multiple servers each of these nodes all act as a leader and they can all process different transactions together and so in theory it allows to handle unlimited amounts of transactions from the users and so the benefits of doing this is two volts one whenever we have any bottlenecks in our transaction per second just throw another server no problem and then we just handle X more users and their benefit is that these servers it doesn't have to be high-end server it can just be the computer that I'm using right now and that will allow the salon Network to be much cheaper so now that brings us to the complication this all sounds great but now that we have multiple leaders using proof of History we have a big problem this breaks proof history because each server acts independently of each other we now no longer know the actual ordering of the blocks that are being produced and when the transaction comes in and so how do we solve this problem well in the white paper there is a hypothetical solution that talks about this which is great here is an existing diagram of user interaction with this lawn Network so we have one leader so in a horizontal scanning solution we would have multiple leaders so let's say we have uh you know let's say we have a liter a and later a generates hash 100a and hash 101a so that's the basic diagram so now let's imagine we have multiple leaders now so we have liter oh let me move myself over here so now we have leader a b and c and each leader will generate the hash with their own letter appended onto the end of it so each of these leader will continuously generate hashes and use the hash as the input for the next hash so according to the white paper with each of these later leader node does is that every once in a while they'll actually try and synchronize to each other uh what so how we would synchronize is that let's say leader a would send its current hash its current state and time to leader B which then we'll hash both that value that it receives along with the hashtag it's currently at to generate the next hash which in this instance let's say it's just hash water B at this point let's say we send hash 99a everything that happened prior to Hash 99a has to have occurred before hash 100b so with this we have a way to reason about what occurred first now we continue this and let's say hash 100 leader B synchronized with leader C and send hashimotid b to it and then we combine hash 100 B with hash 100 C to get hash 101c and you know as a result of this because the transitivity uh we all we now know for sure that everything from hash 100b and before it must have happened before hash 101c and also because hash 100b is made from hash 99a over here we know that everything that happened from hash 99a and before that must have happened before hash 101c and so there are still some you know dependency issues like well what about hash 10c and I'm sure there must be a lot more study done into this so it kind of understands proof of history is a vertical scaling solution so we don't have to deal with the complexity of this but I imagine one day as we hit the limits of vertical scaling we might start either looking into a horizontal scaling solution like was proposed over here or maybe we'll look up some sort of other side chains that will help handle this it'll be interesting nonetheless to see how this all plays out in the future so I'm very excited for that all right if you made it to this part of the video well first thank you for sticking with me for this long but you have finally understand a little bit more Slaughter yes a little bit because while I was reading through all the salon documentation and the white paper I realized there was just so much more that I was not able to cover with the implementation detail I felt that I was able to cover a pretty good relative Deep dive of how Salon Works um but not on its whole entirety which I imagine no one single mortal can ever accomplish so before I end this video I just want to give a couple of thoughts uh going through the whole process so overall um I actually started this whole entire thing with a pretty negative connotation I actually had no idea how uh crypto Works in general to be honest no idea how proof of History Works proof of staking proof of work but after going through all this research and making this video I've actually have a much better idea how the road works and to be honest I'm actually surprised when I first came in I came in there I came with the assumption that cryptocurrency in general is basically just a public database like honestly why are we even using this besides you know watching the coin value go up and getting rich but after finally understanding some of the core problems that we're trying to solve like the Byzantine fault and ensuring that we have a true decentralized uh cluster of nodes I can understand why certain decisions have been made and that makes a lot of sense and this actually clarify one big that I have with Salon which is proof history is it really as great as we say it is well answer is yes actually it is um not being able to have a order a timestamp order of nodes being created and not having to wait actually it seems to be a big game changer but what I read compared to the other blockchain Technologies so coming out of this I actually am a lot more bullish now on Solana in general versus when I started coming in thinking it was just a load of smoking mirrors the one thing that I was actually very impressed about was just having a horizontal scaling solution because what I did understand there's only this whole process was that we just had a vertical scaling solution with just one node and I thought well that's not sustainable eventually we're going to hit a upper limit and we can't continue anymore well it turns out that's not the case uh Solana does at least in theory have an idea how they would scale horizontally if they had the unfortunate circumstance to ever reach that point because because I can tell you from a normal distributed system in web 2 that is in nightmare so if you found this video helpful I greatly appreciate it if you give it a like and if you're not subscribed nope consider maybe hitting the sub button but until then uh stay tuned for my more regular scheduled content if I ever get that far alright see ya