How to Set Up a Cosmos Validator Node: Full Guide with Sentry Architecture

[00:00:02]hello everyone and welcome to our fifth edition of the call with us um we have marco today with us from intel gmbh and he'll be running to how into a journey of how to set up your own cosmos validator uh he has experience in this field as he has been running his own validator for a while and he'll be sharing with you um what is the best setup to do that this is a session which will last almost 2 hours that is being recorded so everyone who like to watch the replay you can find it on our cosmos youtube channel and also on the cosmos dot network code as web page this being said if you have questions please send them here to me in the chat privately or also in the discord code as uh now that we have um [Music] and um next workshop is on the 21st of july and we'll be going through uh building on uh ether meet so check out the new workshop we have listed you'll find them there so we're good to go now so thank you marco for joining us and for sharing your experience and running a cosmos via data with us thanks for having me um [Music] okay yes so today we'll be walking through on how to run a validator the recommended setup with tendermint and so the agenda is kind of we'll cover what is a validator what does it do and what is its role in a net in a quick network the sentry architecture and what it does and what it enables this place server configuration and then we'll start up a server from scratch and we'll set up a sentry node and a validator node and then we'll walk through a simple couple steps to secure your [Music] um there's many ways in how to do this and then last set up metric through to prometheus server um hopefully we won't last the entire two hours but doing stuff live is always interesting so what is a validator um so a validator helps secure approval sync network it had the participation consensus in that it proposes blocks and it votes on proposed blocks from other validators a validator in a proof of stake network has something at stake that's kind of the mistake and proof of stake it has something that it's willing to lose if it acts maliciously and this is kind of the hardest part to protect against there is countless attacks that can be done but in the case of the cosmos hub they we use a variant of proof of state called delegated proof mistake in which a user um anyone in the network you me instead ever having paid in participation of the security of the network we can delegate our tokens to a validator who will um securely participate in consensus and then from that there's a block reward because proof of stake networks are inherently inflationary and there's a transaction fee of course associated with every transaction that the proposer and the validators that voted on the block will receive and these this rewards and transaction fees will trickle down to you as a delegator to this validator in the cosmos hub there's many different uh sources and different cases of getting the full reward the cause will sell actually rewards validators for waiting the entire uh waiting for 100 of the validated signatures this is of course um if a validator is down for some reason [Music] then the validator won't be able to receive their participation rewards in the block and now a valder can also be run as a full node but also as a signing node what a signing node is essentially um is a node that only has the most current state and it only and it drops the previous date so or that's one case and another case would be that it only receives the votes that it needs to sign it doesn't actually keep any state so it's a very light and efficient there's multiple examples of this but it is not built into protocol currently of course if you have any questions just being adriana and please ask away so in a century node architecture we have essential we have a couple nodes that are communicating with the public network the validator is private because and this is primarily due to ddos protection because in this case a if someone's trying to attack the validator they would first have to break through the century and sentries are protecting the validator's ip and node id and so they will not be able to find out what is the ip of the validator to try and attack it now this the photo that you see here is kind of one example uh a different example of a century node architecture could be a single connection to the validator instead of three connections to three centuries now um this three is kind of a recommendation that we do but of course you're kind of free to do as as different configurations as you see fit um what i have seen when i was running a validator was that many people would run one seed note and a seed node is essentially a node that only only feeds other nodes other addresses and so essentially if your sentry or cno were to go down and needed so if your century or if your century node were to go down then you would only need the seed nodes ip because the seed node would have a list of peers that would it would be able to feed your sentry to catch up and get back to where it was and protect your validator now one of the main reasons why this ddos mechanism was thought of is kind of it was it was a security consideration to the prediction of the next proposer so currently in tendermint and in many proof of stake protocols the proposer of the next block is actually deterministic and so to protect the validator who is going to be the next block proposer a firewall essentially a firewall of other nodes is needed out there who are working on um switching keys between centuries instead of having a validator so there's just a kind of a server just feeding keys to different centuries depending on which one is the closest and the fastest and now this kind of century of architecture it could go away if it was random leader election so in random leader election the leader is essentially non-deterministic you cannot predict who will be the leader who will be the proposer of the next block and so essentially the ddos mechanism the ddos attack is more or less not possible okay so um tendermint was built and so telling this underlying technology that is running the cosmos hub and other cosmos-based blockchains and it was built in the age of low memory environments and so we recommend that you run with at least two gigabytes um there have been experiments of running uh tendermint on raspberry pi's and running it successfully um without any issues and of course um if the more ram you put on the server the quicker it'll be the more processing throughput you'll be able to do um and so in the in tendermint and in the cosmos sub of course your state grows quite quickly for every block you um even if it's an empty block with no transactions there is some state there and so at um so it is recommended to put a large amount a large hard drive an ssd or normal hard drive and to start out 80 gigabytes um 60 gigabytes it's 80 gigabytes is recommended and of course at some point you will hit full disk and you will have to migrate to a larger disk of course when you write when you're picking a server any os is possible different os's have different security guarantees and just different environments for what the user would like to do um you can run in cloud providers you can run on your own hosted machine on your own personal machines um it's kind of up to you on how you want to set up your validator configuration and what you believe to be the most secure so also if you are using a cloud provider um like aws and google cloud many of them come with a way to preset firewalls in the dashboard so aws you can preset this with security rules and um in that case on the validator itself you wouldn't actually have to need to configure a firewall but a nginx balancer it's always recommended especially for the rpc connection so of course when you're starting a server so in our example today we'll walk through setting up gaia um there's already one node running like i touched on earlier and we'll be connecting to this node and so we'll be running the same software that's being run in the cosmos hub currently so the requirements for starting the server are you need going 1.12 or greater and of course you need make because meek is the build tool that we used um for configuration gcc this is used for the ledger technology um like ledger nano and so you'll be able to interact with your node and kit of course a clone of the gaia repo and then when when we solve and get through all kind of tireless then of course in the genesis file before you start when you start a new node you always need to get the original genesis file there's no way to start without it and a quick way to get this is if there's an existing rpc port available that people have provided to the public you can just go to their slash genesis page and then route and then in there you'll be able to find the entire genesis.json so there's a there's a lot less questions that i thought there would be so we might get through this a bit faster um okay so um for our sentry node it's kind of the front end to our entire network so where we want it to handle peer exchange because we want the address book within the century to be quite large so it can act fast and send messages quite quickly that it receives from the validator persistent peers um this is the peer that we want for the sentry node to always be connected to and so in many cases this will be your validator or other sentry nodes the nice thing about this is that you can set your your other sentry nodes to communicate through a virtual private vpc a virtual private network on your cloud providers so in many cases aws provides a private ip to communicate between servers and you'll be able to use this to communicate between sentries but also to communicate with your validator the private peer ids is consists of the peer ids which are meant to stay private so this will be your validator primarily in some cases some people run multiple validators and this is for failover protection um in our case we'll just be running with one validator with a sentry in front of it to display how is it and we want the address book settings to not be strict this is primarily so it can be of course not strict but then it can provide different ips so we can play around with it a bit more so i'll kind of walk through the slides and then we'll go through the server and start playing uh playing around with that side of things so the validator node is our backend it is the node that does the handling of the logic it verifies the commits it signs transactions i'm sorry assigns votes it signs proposals and this is what secures the network so we want this to be especially safe for our validator node we don't want the peer exchange module of tendermint to be enabled this is because we don't [Music] and we don't want um it to be able to try and search for other notes because we only wanted to communicate our sentry notes just one second uh i would like you to if you can close the camera maybe second out of the bandwidth for the sound and just leave the slides okay if this is better now thank you okay perfect should i repeat anything okay all good um and of course we want to mimic what we did in the century node of setting the address book um set to false at the strictness of the address book now um this is kind of where it gets kind of interesting um of course in a follow-up session we'll be walking through setting up bkms this is essentially our key management system we will remove the private key from the validator node because um in our example the value key will be saved on the server and provided for quick access to the validator but this is not the most secure as people could break into your server so yeah securing your server from dos taxes key and so this is kind of the most important thing because if you're running a sentry node or if you're running a valve data you want to protect from outside communication other than what you expect in the validators case we only want to communicate with a sentry node and possibly a key management or an hsn system for our century many people and in many cases we reduce the sea connection to only be available to a specific id so we can only communicate with certain people because the rpc has been known to provide a mechanism for people to slow down notes and if our people are able to slow down your sentry nodes then it slows down the communication to your validator of course for this a nginx proxy can be used a load balancer and of course a simple firewall and we'll walk through setting up a simple firewall using ufw so now let me switch to different [Music] okay all right okay can everyone see my screen yeah yes because we haven't forgetting that no one can hear me okay we have a question from anjana who i think it's new to the cosmos days she's asking what the difference is between cosmos i believe cosmos sdk and thunder mintar so uh i want to make sure that everyone understands that that tender mint is the consensus engine while cosmos sdk is the framework for building applications and yeah and if everyone has more uh questions about this i can send um i can send like a presentation in the chat here yeah kind of a short description it could be like a um there's many many different acronyms that people use but um what uh i kind of stick to is kind of your framework is kind of like your front end the application that the stuff that the clients and users interact with but tenderness is like your back end where all the heavy lifting the logic is handled so the front end could be something like react in the back end could be something like node tendermint does the heavy lifting and the communication between peers and handling of consensus while the cosmos sdk handles what you want to do do you want to make a governance proposal do you want to submit a transaction for for sending tokens to a friend and all those types of things we also have a question here uh from torch she says um what is your opinion on a sentry relayer connection i think this is like a private network or a file editor setup instead of a sentry so i have just copy pasted his entire questions and question in the chat he said he had he has like mixed feelings towards the relayer notes but would like to hear your opinion or experience on that like why are validators using this architecture and you have the full question in the chat let me just double check on the forum i don't recall the relay node [Music] and also while you look for this clint is asking if each sentry node needs to run on separate hardware or not um for this i think he asks if the sentry nodes attached to the validator need to run on separate hardware yes yes i was just thinking about the really note so in my experience um the only two types of nodes i've kind of interacted with for validators is and i've also experienced with uh failover protection um kind of signing hsms in the case that you have let's say force sentry nodes and you don't have actually a validator node and your hsm will choose that random one sentry node to send its key to and that key will sign on that on that century so that was kind of the most interesting thing that i've run across and played around with but um i haven't looked into the layers out and then to the next question doesn't does each sentry node need to run on separate hardware yes so it is recommended to run on different hardware um to run on the same hardware is kind of putting all your eggs in one basket if that server has maintenance or is down um for any number of reasons then entire century node architecture will be down and your validator will be able to communicate to the outside world so i think it's important for people to understand the sentry nodes are like a backup also for the validators so it's very important to have them in separate locations in case something happens to um like one of those or the validator you have like backup running securely somewhere else if you put them all together on the same hardware if something happens then you run into the risk of your validator completely shutting down or not responding anymore um so i just got a question from leo asking can the validator nodes share the same set of sentry nodes so if you're running multiple validators you do have to be very careful for not double signing um and this is something that is pretty scary because if you do double sided then you essentially will get slashed but your value notes can't share the same set of century notes there's multiple different types of kind of mistakes that people can make there's only i believe currently been one example of someone double signing a block double proposing a block in which they were stinking a new validator and using the same network as the other net as the running validator and the participating object and he forgot to set um the validator to not not sign blocks and not propose blocks and so while he was sinking his node actually proposed a block that was in the past and so when the network recognized this they essentially blocked his um validator key and so he could not use that validator key ever again and i believe a certain percentage of his entire stake was burned and the rest was returned to its delegators okay so okay more questions for now okay we're good perfect okay so now i'm just going to sh into a server okay so of course there's no go so in the first steps um of course we want to upgrade the server and update any dependencies it has and so to do this we just run sudo apt-get update i'm i usually for these examples it's for me it's the quickest to run on an ubuntu server and i'm using digitalocean but any cloud provider can be used we can also get started on a different server okay so if you can walk us through what you are doing now yeah um so the top server is just currently updating its dependencies um and um anything that's out of date um digit teachers depends he's up to date there has been no attacks to service because um like at the equifax i pack i believe and many attacks hacks out their world primarily stem because they did not upgrade their dependencies all right so this server has been completed so we'll do the same thing oops down here so this is just running the same thing as above as we just ran on the book and now we need to of course install go for this example i'll just be using go i believe is the latest um and you otherwise if you do it through to installer um you can get the old version um i'm not actually sure what version is actually through active right now um since every time i do install go i use w get so now since it's installed um okay so we have connected to the server and now we're installing go yeah and so we just pull down a tar file and so now we'll be kind of fun carrying um i don't know if that's actually a word um the file into a set of files that will be the code source code and so here you can see here's the tar file that we downloaded and then when we ran the last command we got the go and we want to that go to our user local and then when we of course when you download the go we need to export environment and variables and so now we'll be able to use go and go is the language that's used by tendermint and the cosmos sdk so since the top one's going a bit faster i'll just go along with that let me just grab the clone [Music] url for gaia so now we're loading gaia gaia is the software that runs on the custom stuff and recently recently we migrated from master to master to main and so if we check yep we are on main and so right now the next couple steps is to install make let's do that so make is of course the what we use for to providing commands for users to install software and to run tests linting a whole plethora of things and then the next thing we need to install is gcc this is used by ledger so you'll be able to interact with nodes with the software using a ledger device and i'll be doing the same stuff we have run through in the below server so does anyone um is everyone anyone able to follow along or is anyone also trying to set up their own note you can do it on your laptop on your computer or if you are able to sh to a server then we can kind of play around and get you connected to the testnet as well jump in with your questions if you have some so now back in the top um server we'll be installing the gaia software and this installs two binaries one is gaia d and the other is gaia's tli they both do two different things gaia d is the daemon that runs the software so you can call gaia d start to run the software and gaia cli is how you interact with the chain so this will handle your transaction to your queries and transactions uh modules saying that they are using docker for this yeah docker is a another way to do this i i'm a huge fan of docker and development environments um i'm not the biggest fan in production environments of course everyone has their own opinions [Music] run into some weird networking issues the best for production environments so now the software is installed on the top server and so now we'll be able to run gaia d and then we'll be able to see the software and then um this these are the kind of commands we will be able to run i'm with gaia d and of course with gaia cli you have a status configurable parameters the query sub commands and the transaction subcommittees so before we start the way i go through this process is actually before i start configuring the genesis and and stuff in the test and start taking my node i actually go into mike id config whoops tacos and then we'll go into gaia d into the config folder um this will house the genesis the prior validator key and configurable parameters for tendermint and also the sdk for right now we'll just be touching the configurable parameters of tendermint so the first thing we want to do is change our prometheus flag from foster true and this one we lost you a bit here go back to changing the can can you hear me yeah okay perfect um yes so right now i'm just changing the prometheus flag from false to true and this will automatically on when the node starts exposing the port to triple six zero for prometheus to pick up and uh for meteor server to pick up and collect metrics from um the exciting thing about this actually the exciting thing about the prometheus metrics um that is coming in upcoming release if you are a user of the cosmos sdk or a node operator the next release will house metrics from the sdk directly so modules you'll be able to get more what's happening in the application so here here's the rpc port so the rpc port is exposed over 26657 and by default tendermint sets it to the local host ip and so for in our case we wanted to export we want to expose it to the wider internet so we'll be able to query it from anywhere in the world and then that's the main things we want to change for now we'll come back to it um leo's asking should index um should index all tags or keys be set to true for a validator it's entirely up to you as a operator um at least for me um in this example um it's i'm going to leave it as true but as in the past when i was running a validator i would turn off the indexer and turn off as much as i could just so i can have the run to have the node running more efficient um in the future um this is another thing that is changing in the future this will kind of be a default or depending on the application on how they handle it because then these flags will not be able to be set in tendermint so now we have created um we have set up our node um we want to next thing we want to do is grab the genesis that was already created when i created the network so i'm just so in the genesis file you have all your parameters um and then of tendermint and of each module and this is kind of what sets up your application um what is the most recent knowledge of a few applications so in the case of the cosmos hub if it were to do an upgrade it would actually export the current state of everything into a genesis file that genesis file will be quite large because it has everyone's accounts everyone's delegations and everything like this and then on the new chain um on the network upgrade they would use that genesis as beginning part of getting blocked so let me just [Music] find the node i was using earlier so what i'm doing right now that you can't see because of the sharing screen stuff is actually i'm going to the slash genesis of the other node and once it loads i am going to grab the genesis and then put it in our genesis json and then we'll be able to start the node so justice um asco is asking uh he didn't uh quite understand where the genesis file was located if you can explain again yes so the genesis file in many yeah so the genesis file in many cases um i'll show again um so when you create a node you essentially have to create a genesis file and that's what this kind of output right here is um as you can see here it's kind of like the gold bonded amount the previous proposer the validators accumulated conditions and so this is the genesis and so but the thing is when we want to join an existing network we have to actually go grab that genesis um that network's genesis because this genesis won't actually work for that one and so what i did that is went to a rpc and that she so i can actually curl it from here and show you kind of the output that would be received so this is a different note that has already been running and so we would go to this rpc endpoint and type in slash genesis and then it would output the genesis file and so essentially this to one of i believe one two [Music] so it would be here to here and then we would go back to our local genesis file and just output it here now oops let's hope that will be correct usually grabbing stuff without double checking with this can cause a bit of an issue i'm getting the brackets wrong of course it is json and so now since we have the json the genesis file and we've configured our node we're missing one last final step and that last final step would be to connect to an existing node so to do okay did this answer your question about the genesis thank you yes he said perfect um yeah so to add a node to our um to connect for us to connect to we have to go back to config.tunnel file and then we can search for persistent peers and in here we're going to be out we're going to be inputting the node id of of the node that's already running the ip and the port that it's exposing for p2p communication now in the next step we want so now um some people the only question that i've run into and kind of the issue that i ran into when i was first thing to play around with cosmos nodes is how to get the node id and i do believe this has come up um some people will have run into this question and the simplest place the quickest place would be for to go to the tendermint command of cayenne d and go show node id and this is your nodes id it's a hash of i believe the moniker and some other information and so this is um what your peer what your node will be as identified as so now since we got since we inputted the needed information now let's actually quickly see this is going to be first try [Music] we lost there's a chance i got the brackets wrong on the genesis um can you hear me now yeah i was just saying if we get this right they're copying the json from here because i think i copied one extra bracket but let's see if i was able to get it right yeah i got i got the wrong i got the wrong number of brackets one the bottom should have one two three four so doing stuff live is always fun there we go should okay let's see if it's able to find the other okay and now as you can see the other note has been running for a couple hours and so right now it is catching up to the most recent height of the other node and so in and i believe it's already caught up um so this is one example of syncing a node um in the future um there'll be different and quicker ways um this is kind of an issue for some people and syncing large um chains and so we'll be using this node as a sentry node and then we'll try to use this node that i'm hitting enter on this terminal as a validator node so let's see what if we had it installed now we're going to run through the previous commands of the other uh the stuff we did on the other node and so first we need to install the guide software and in many scenarios you'd run the node in the background for this i'm leaving it open right now just because we will have to stop the node when we want to configure it to run as a sentry node and to configure it as when this validator is able to connect to it okay so now it's like catching up so the top note has already actually caught up and now it's actually participating um [Music] receiving blocks that the other node has proposed and finalized since since that node is the only node of the network then then there's uh it's not it's only receiving information that has already been finalized okay where were we guyadee in it let's say the other one was tacos let's call this one tuesdays and so now we're going to go to into our config.tumble again like we did in the previous one and configure it to how we want a file data to be configured so as i said earlier for the validator node we want the peer exchange to be set to false so x so this is the pure exchange reactor so in in our example we want this to be false but if let's say you're running a normal node you would want this to be true or if you're running a seed node you'd want this to be true and then for a c node you'd actually want other configurations set to zero something like the memphis size so you wouldn't actually be processing and receiving and sending transactions you would just be feeding other nodes information about other peers so we set that to false now oh i actually forgot to grab forgot to grab the id of this mode and go back and it's already caught up so in this on this node we'll go back to the persistent peers here we'll be inputting this node's mid and let me just grab its ip the 2665 i always get mixed up on it and then of course we want to set our previous to true and now this is not required for for a node operator i do it just so i can ping the servers from my laptop or from a orchestration mechanism so in many cases for docker kubernetes and these cases you could use rpc to check the health of a node and people used in this case or if you're just using something like ansible you could set up a script to ping it to just me so now that we've configured the node to so that we've configured the node to how we want it to be now just like in the past node we need to fetch the genesis so curl genesis this has to be four while you are copying this i want to make sure that if we have questions or not i don't know if this is too high-tech for people who are watching us and if they want more information or more details on the commands that you are using or the steps that you are taking so please feel free to send us the questions you have so don't be shy there are no stupid there are no stupid questions i want to make sure people understand the steps to uh opening up the validator and i want to make sure that even people who haven't try this on before uh understand like what is happening here and they'll be able to do that on their own also after watching this tutorial so stop us anytime you have questions no matter how how they are and we will make sure to answer them yes please please ask as many questions as you can the interesting part is going to happen pretty soon when we have to interact we have to go back to the server that's already running and send tokens to this node so we can actually create a validator so okay can you make us um a summary of what you have done so far in just a few words marco yes so currently what we've done so far is uh we walked the different uh node architectures a sentry node architecture the different nodes uh available in this type of architecture um kind of some a quick glimpse into how to work and we walked through setting up a node from scratch with um gaia and now we are that went from copying the genesis from an existing node connecting to an existing node and exposing prometheus and exposing the rpc port to the world wide web and now we are doing the same thing on our second node and this node has been configured to be run as a validator meaning it will not participate in peer exchange and it will not be its ip and node id will not be broadcast to the entire network so this is going to be interesting again if i'm able to get it right okay so now [Music] now before we start this note something that we have to do is we have to go back to our sentry node the node that we'll be using as a century and add it as a persistent peer because we do not want this century node to broadcast um our validator's ip to the broader network but you want to use the sentry node um to catch up to the network so to do that we'll be getting the id of the node um as i shared before okay so in easy terms you are hiding the ip away from being broadcasted yes so right now um i'm going to go back into the con fig uh solomon is asking if since the audio can kind of break some breaks up from time to time sorry solomon we're doing our best sometimes the internet is not our best friend uh if you can find like a step by step guide or document somewhere yes there there are multiple um so there is let me grab your one i just have to remember we have too many docs sites okay we'll search for it solomon and we'll share it with you here in the chat and also in the discord channel so so i'm sending you kind of the um uh one from the documentation you can send it to everyone okay sorry okay great and then there's also um a [Music] introduction to century architectures and i deeper dive into it at the docks um from tender so so if if i do cut out that i apologize um please just think um and ask me to repeat i'm happy to do so okay so you have to share with us this um [Music] uh via the setup from hub.cosmosa network and also the one on the violators and from docs um can everyone see those miracle has posted them in the chat for everyone to see and i'll also take this here and add it to the discord channel for everyone to have them there in case they like to browse through them i also haven't checked the discord in a bit don't worry i'm on top of it all right um okay awesome so where we where we left off is and now we're adding the validators node id to the private peer id list of the node that we will be using as our century node so we did we came into file and this file is the config.tama um this file contains both lots of settings for 10 minutes configuration [Music] so okay we have a question from denny and asking how does seed mode work can it be on while syncing um so seed mode is um it can be on while syncing you'll essentially just try and start try and crawl the network it's usually usually run as a specific node type so you wouldn't i mean you can run it as a full node but let's set up a nicer configuration when you're able to run seed mode when you have stuff turned off [Music] okay um and so in this case if we wanted to run a node as seed mode then the recommended setup is actually to go to the memphis configuration options and set the number of transactions in the member to zero and this way you're informally turning off the mempool so you will not your mempool will not be working essentially it will be on but not processing anything maybe it would be like a very good time here to explain a few words what the mempool is yeah for sure so the mempool is a pool of transactions that are submitted by users and so you can think of it as when a bunch of people submit those transactions are not executed immediately there is of course a delay based on the block time and so the transactions accrue in the middle and then when it comes time for a block for a validator to propose a block he looks at the mempool and says what transactions should i take from the menthol and then he picks them up from the mental adds him adds them into the block he's about to propose to the network and then he proposes it to and then once he does so then he's able to propose it to the network in many cases memphis um are a prior priority um ordered list and so the gas fees that you set on a transaction will set its um urgency so if you we're kind of seeing this in ethereum um that many people that the transaction fees are going up quite heavily and so if you actually pay a lower gas fee then you will your transaction will take longer to execute because you're fighting a lot more people to get into a block but if you pay a higher transaction fee then you'll be bumped in front of the list and your transaction is more likely to be picked up by the miners and also i think it's very important to highlight the fact that the mempool has to be clean from time to time yes and so um of course the mempool in the current design of the mempool um not all transactions are picked up some can be lingering around for a longer period of time and so um if other transactions are coming in front or the the application is calling one transaction above another transaction and non-stop then the mempool would can get filled up in a bit cluttered and then that you're able to call via command to flush the mempo essentially you're flushing the transactions and then it will start to fill up again with the messages that are sent from other nodes now there's many um excuse me yeah sure i i said i think we i hope we have answered the question regarding this yes um is are there more questions not for no thank you perfect um yeah so a lot of the a lot of the configuration will i can kind of actually cover some of these configuration options um to better explain them uh let's go where are we okay so um you have configuration options for [Music] almost everything in tendermint um which is the best thing just because as we've seen on the cosmos sub there are validators that will misuse these settings to propose blocks or to move forward on a block a lot faster than other people can communicate their votes now this isn't really detrimental to anyone except the person doing it because they are not getting the reward for actually waiting for everyone to submit their votes or waiting the full timeout so let's start at the top okay so here here we have the proxy app in tendermint um the idea of french tendermint is you can write your application in any language um and so this is kind of where your app lives it doesn't have to live kind of together it can actually live on separate servers and of course if you want the fast sync um now of course i don't really know a reason why you wouldn't want to fast sync um but this is a configurable configurable option to sync um not in fasting currently would mean that you would be syncing while participating in consensus which is very slow and so it is actually preferred to run with this on now when running a node or even running an application um you may not want to use the default database so the default databases actually go leveldb but um the code for the sdk and tendermint is designed in a way that you can actually run with different databases this won't affect um the network so individual nodes could run with uh separate types of different types of databases the sub types um here are go level dc level db old db we recently added support for badger tp and the list is unfortunately growing but we hope to do database testing and provide a faster environment for everyone so if you run into an issue kind of the only times you'll have to worry about these two the log level the log format is when um you run into an issue or you want to debug something that's going on with your node um if you're running into an error into some sort of weird error or you just want to see more so what's going on in the note um how often is throwing errors how often it's um doing un unor unorthodox things then you would change the log level then the pathogenesis file and then these two are are fairly important so in your priv validator key file what this is is um the type of key that the network is using so in many of these cases in 100 of the cases of ed25519 the address and the private key now this is a very important file to keep secure and that is why key management systems and hsms are recommended when running a validator now what the state file is is the last known state the last known height that your validator that you validate your key signed at now this is to help to try and help prevent double signatures because if you were to restart your node for any number of reasons you want to have knowledge of hey i signed at this height so while you're syncing you should not try and sign or do anything until you get to this height and if you're caught up so this okay we lost you about 23 here in the um the pre-validator l a can you hear me now yeah okay we'll go back back okay perfect um so if um yeah so if you would um point the port and the ip if it's a tcp or if it's a tcp i'm ip or unix software and you would provide unix socket address and this tells the tendermint node that hey you should not use the priv validator key json but you should use an external signing device to sign the votes and proposals that come to you and then this is kind of more information about um the app and if you're running socket and so the abc is of course run over socket in our case the profiler port is in golding there's the notion of profiling an application and this will show you where an application spends the most amount of time most amount of bandwidth most amount of memory usage and so this is useful again mostly for development purposes to help us debug how we can speed up tenement to provide a faster environment for users i'm gonna go through a bit go through some of it faster just because this file is 100 or so lines so here we have the rpc connection the rpc server configuration options um in the beginning you saw that we changed the la laddr from 127.0.0.1 to 0.0.0.0 to allow external devices to query the rpc port now when doing so it is recommended to have a nginx configuration in front primarily because this can cause your node to slow down if it's being hit too often and then on that we have all the configurations you have the grpc um address jpc is used can be used for submitting transactions and then of course this is kind of um gets into the nitty gritty of the rpc can you hear me yeah it's kind of breaking down from time to time okay um so this is getting to the nitty-gritty um these are many things that you as a node operator do not have to worry but if you want to get into heavily customizing your note um you can um you can diminish kind of the max open connections for your node uh max subscription clients and the amount of subscriptions per client this is kind of these these are all configurable options for our pc and of course you can provide a tls server certification certificate and a key so you can protect your rpc connection now on the p2p configuration options you have the port that p2p communicates with any seeds and so you can put a seed node in here so instead of you having to [Music] always connect to a pier here is where you put your seed node in persistent peers we already covered a bit upnp port forwarding this is for nat again um it's not something to worry about um your address book um setting your address book to strict um the number of inbound outbound peers now the max number of inbound outcome peers does limit the amount of of course those limits amount of peers you have in your address book and this you can raise it to a larger number just 40 and 10 is kind of the number we chose with to be extra safe in the software we chose safety and liveness over-performing and so many of the figures we sound like a robot chosen as defaults are [Music] can be configured to higher if you do so if you do please okay when you break down you sound like a robot a robot yeah yeah i'm not sure how to better the internet here yeah it's mostly tiny breakout and this is when the robot part comes in at that point it's just like you just yell stop and then i'll pick up again okay let's pick up like 10 seconds before perfect um yeah so as it turns from before um the max inbound and outbound appears then after that we get into some more configuration of the pdp now this is all stuff that's heavily configurable the majority of people do not actually configure these settings because they are set by default to good and safe numbers the main stuff that you kind of have to worry about is the rpc um androids exploding wider web um your prometheus metrics your peers your private peers your seeds and then some of your metrics and of course if you run the c note then you get into the c note section and a bit of the memphis section so private peers we did cover um that's the mempo can be configured quite quite a bit um you can also say recheck to false and this will diminish the kind of the load on your server and you can also set up broadcast to false so if you receive if you're in seed mode then of course you don't want to be broadcasting transactions um but if you also set your number or transactions in the mempool then of course you're doing essentially the same thing you're just saying i won't accept anything um then you can increase the size of the memphis this is the number of transactions of uh in the mempo the recommended is of course the default but many people do increase it if and many people actually and some people actually do decrease it on their validators just to reduce the load on their server now like i touched on before um tendermint was written for low low powered environments so environments with low memory footprints and so this isn't the large concern but kind of it's the cherry on top if you want to run on even a smaller server or run one as only assigning node so now we get into the some fasting so fasting there's actually two different versions um we're currently in the middle of uh we actually finished the p2 of fasting [Music] v0 is default because it's just another longest and v1 is a refactor for testability and v2 is a refactor of v1 and we hope that v2 will become the default and it will provide increased performance so now the consensus configuration options these aren't um i do recommend trying to stay away from these there are validators that do touch these and one is um the main one is if you want to move forward when you receive two-thirds of the votes then you would end up touching these settings adding them to zero because then you don't want to wait for everyone to send their you just want to move forward um the interesting thing the interesting one um in this um this grouping is create empty blocks so in many environments on how many blockchains you kind of see the tendency that they're always creating blocks even if there's no transactions on the closed tab when there's no transactions there's still a block that's created um what i'm saying is that the uh even on the cosmos side when there's no transactions created and there's still a block created and in many other blockchains and many other ecosystems this is also fairly common but um for some applications they actually just want to produce blocks from this transaction within the memphis or multiple transactions and so this is essentially that feature so um if you have one transaction on monday and then let's say it's a slow it's just a week and so you only have one transaction you have another transaction coming on thursday well in those four days you would have created two blocks but if you had this set set if you had it set to false but if you had this set to true then you'd be creating blocks um let's say every eight seconds in between monday and friday so next is the tx index section i believe yes uh leo asked about this section uh previously so in tendermint um and in the cosmos sdk you have this event system in which when a transaction is submitted you have these events that are outputted so you can query specifically to this event um an example of this if you want to search for transactions that are over a certain amount of atoms that is also sent by this particular address this would provide you a way to do that now this is on a per node basis and so some nodes that are exposing the rpc won't always have their indexing set to true indexing all tags or they might even just index specific facts now um there are cases where people will not index at all this of course reduces the increases in efficiency of the server because you're not running a process which you're trying to index events and so this is something that you can change you can't don't have to worry about many nodes that are running as full full nodes archival modes we'll be setting this to all text to true in order to provide a better user experience for the people using their pc endpoints and now finally we get to the prometheus um instrumentation the metric gathering um for tenement like i mentioned before it exposes a bunch of metrics um they can be found the metrics that tender exposes can be found here i just posted in the chat a link to the docs.tenement.com page that goes to our goes to our documentation on metrics this is kind of an ever-growing list and provides information into the nodes and so it can be provided so you can see what's actually going on and of course the other metrics can't be gathered through this okay uh that was a mouthful um i hope someone was able to take something away from the list of configurable options are there any questions in regards to the config file in particular i have a question there is a sound in the background is it like from the servers or something um it's from crickets outside because it's really hot here and so the crickets are chirping it's got crickets and birds okay got it okay except crickets and birds any other questions on the configurations that mark going through i know we had some slight issues with the um the sun breaking up but i hope people got like uh the most important things out of this but it is somehow you fail to understand something from what he went through just let me know and we can go back to that again so let's see okay so we went through the configuration and what do you plan to do now um i have to actually remember where we left off right here ideas has been added okay perfect okay and what you want to do now yes so now since the private prg has been has been um before we start the note up again let's just double check that we have the node added as a persistent peer here so we're going back to the country.tumble most of the work you do will be in the conflict so this is out here stop here okay perfect um this is all correct so i'm actually going to i believe malaya is in the chat here is are you here yeah he is victor is here i'm actually going to um one of his articles that he wrote a while ago on using systemd for uh running your servers and so here is a link to the article and so this is kind of systemd is a tool to help manage processes and so you'll be able to use this to run your id instance in the background for me to not have to go through setting it up right now due to time constraints i'm actually just going to use no hub no app is kind of a similar thing um and so we're going to do that and so what no does it just runs guy dstart in the background and produces a log file for us to follow along or see if there's any errors and perfect the notes actually already caught up so let's see if we got let's see if we got everything right oh i have added all resources in this court so um make sure to follow up there in case you need more information including the um how to manage the atom node with systemd the tutorial that victor rolled i have also passed that there so make sure to follow those information in case you have questions okay back to you marco marco can you hear me mark i think you muted all right so now we're just waiting for the valve video um what will become the validator node um we're just waiting for it to sync and i believe it's caught up yes okay perfect i think it's hard to mention here that you have open it catch up yeah i was saying that uh for the audience i think it's important to mention that i believe can you hear me yeah i can yeah i can can you hear me can you hear me yeah i can can you hear me yes sorry about that okay don't worry i was saying that i think it's important to mention to the audience that i believe you have opened up the sinking a while back and you're just catching up now because it it kind of takes you a while to sync um on the current setup of the network yes um so right now the the node was actually able to catch up quite quickly and so right now we're able to proceed with math first we need to the next steps of logging into the original value the network yes we're losing you because he has funds that we want um he has money he has tokens that we want to receive to be great can you find like a better spot because i believe it's something to do with the audio or so you have to go back again can you hear me better yeah perfect okay go back uh a few minutes uh intro in which you're discussing the uh catching up and then i have some questions that we need to address here perfect um yes um and for the next steps [Music] we're having a hard time with this [Music] sending tokens from his account to an account receives the tokens um okay let's can you hear me better now yeah let's try can you uh uh you hear me yeah i i was trying this uh can you leave the meeting and come back maybe you reset something okay let's see if the leaving and coming back works you okay waiting for marco to just join again you wow okay marco you're back on okay oh my god oh no okay yeah let's try we have no other solution for now i mean um i can try and answer the questions before i share the screen okay perfect so we have here um some some uh some topics um having two validator servers running and be sure not to double sign how to do that okay so one many different configurations on how to do this um one possible way is to you either run two instances of the kms but the kms's are aware of each other meaning that the kms is hsm's will to communicate and so if if the node if one kms is receiving a is not receiving a response from the validator then it will notify the other kms to sign the transaction and send it to the other validator now it is it gets a bit hairy because of configuration you have to be very careful this configuration because you can easily accidentally propose two different blocks when is your time to propose um there i touched on a couple times uh there's a couple projects out there who are actually working on instead of having a full node as a validator you only have a signing mode so your node is essentially just a let's say just like a mini server that receives votes receives proposals and and commits and just signs them and sends them back and so you essentially you don't because the validator essentially doesn't have to have the previous state it only has to have one previous blocks state to participate okay i hope we have answered this question but i want to make everyone aware of the fact that we'll be running a second workshop um only on the how to secure evaluator with the kms because the second questions are regarding the chemist and one is um which kms is good and why so there are a couple the one that i am the most familiar with is the [Music] and this is supported by uh maintained by conclusion um and a kind of cryptography and arrest house and so they um of the problems that hsms and key management co-author the kms library any signing transactions so this is the one that there's also there's another one that that is a light signing essentially they do is you you give it the key and instead of running an entire note they just participate on the edp layer and on a simple consensus signing layer so you're essentially um and a mini memphis and so those are the only things that you're running and so the efficiency or the size of the server that you need to run your validator is actually a lot smaller an interesting thing that the people at enigma secret network did recently was they were able to extend this um light signing node and so this is a secure enclave um i believe how it works is you would store your key inside of it um i actually do believe uh if ian begins from secret network he's here no he must have left um but he was here earlier and he was uh and he is on the team um that or on the secret network and make one it's kind of you have to be careful when choosing an hsm um but not only when you're choosing an hsm or a key management system but also configuring it because misconfigured hsms and canvases are quite common okay and the last question is how to use canvases when using a cloud server and if it is possible to do that or not yes um it is possible to use that to do that um the recommended um for i mean there are multiple configurations to do it so in the case of a um the recommended path is to have the kms server um either um sharing the server with the validator or in the same um server like server rack in the same uh region of the validator this is primarily due and with of course the internet isn't as efficient as we would like as we're seeing right now um so if you have your kms in a different region of the world than your validator then you run into the potential of missing blocks due to um high latency of the key site tosh can we i know we've got uh we've been interrupted by the backlog of the the noise but hope hope we did okay thanks yes okay good any last questions before we move on um what else do we have to cover merkel from the tutorial um well i was going to walk through the actual creation step so we'll need to log to the other server and send tokens from that server to from that account to a different account and then run the create validator transaction um does that should should we um save that part for the second part um just because we have 13 minutes left or uh how long does it take to do that um it shouldn't take long but of course doing stuff live is always error prone okay let's try since we have promised people to spin up the entire validator from scratch okay so right now before i share my screen i'm just grabbing the ip of the original validator to sign into [Music] okay so you should be seeing my screen now yes so we're going to exit this server and ssh pin to this server okay oh okay um so from this server we also need to what i'll be doing is creating a key and taco okay here's the address the pub key the mnemonic all the information that you shouldn't share on a live workshop but for now we're going to be using we're going to be doing it so gaia cli so right now i'm going to be going through the steps of sending funds from one server to another [Music] sender address and it's the name of the keystone send from test one to this address that we just created and we want to send a lot i'm not gonna try and claim back how much i'm typing in specified michael you have a typo in your command it's got a clean set of guys oh yes yes thank you solomon okay oh whoops that's a big steak there okay so we have a txh here and let's query see if it went through successfully okay transfer recipient here are the events i'm like i touched on earlier these are kind of like the things you can index signatures the message transaction the guest wanted the gas used perfect so we can see [Music] so right now i'm creating the account just to see that money got the money for the tokens yes perfect okay this one's going to be a doozy i'm going to actually open up the editor so we can walk through the command create dodge okay so this is the command that we'll be using to create a validator so um we're creating a staking transaction and the transaction is the amount um is how much we want to specify how much we want to stake with ourselves kind of self-delegation our pub key are um the moniker that we want to use the website this is kind of if you're running a validator you want to have a website so you can not necessarily so you can entice and show users show people who may delegate to your validator what how secure you are kind of for them to get to know you we want to specify the chain id so in our case the chain id is code with us and the commission rate the commission rate is the rate that we will be starting out with this is shown in a percentage and so that's why you see it as point zero one um and then we want to also specify the max trade the reason why we have a max rate just so um so when you create a validator you have a max amount of commission that you will charge so you can this will notify delegators of how much is the maximum amount you can charge some validators do set this to 100 and so it's just kind of um zero to a hundred percent commission rate and so the max change rate this is the rate that you specify that you will change your commission so in many cases um in our case here it's five percent so in one period we can only change if we want to bump our commission the max amount we can bump it is five percent and then the min self delegation is how much we want to set as the minimum self delegation for ourselves the gas is just setting it to auto the gas price and we want to specify it because it's from okay so this [Music] what's the oops for oh i just typed something or did someone else type oops okay where are we at so right now where the transaction that we went over and before that we walked through together is the transaction now that i'm creating and so i want to confirm the transaction before signing yes i want to pass in the password and now we've submitted a transaction to become a validator so now let's actually like text oops you so let's specify yes so just double check right now i'm just double checking one of the my notes that i have everything correct thank you there she i think that was there this success log if we got it there's nothing that always gets me specifying the chain id again with the out of gas i'm teacher let's see if that is enough okay that's used seems like those are inverted now i know we'll just fat well i think that was a bit too much of a fat finger don't do this live people watch out with a fat finger it's very dangerous i believe there was a case in the beginning of the chain of the cosmos lab correct exactly i was referring to that someone lost a lot of money on a fan finger issue so be very careful when doing this live the gas just seems to be bumping every time i give it more let's just okay let's see every time i'm not actually sure what's going on with the gas right now yeah i said even though you you're giving him more than he requests there's still an error tosh says maybe you should try like 200 000k on the gas [Music] let's try it [Music] maybe tobias will be the winner now with 200 000. let's see let's cross our fingers looks successful [Applause] great it was a 200 000 then your bias was a win thank you perfect so now we can actually let's stop sharing this and go and let's see what the most recent height is i haven't actually [Music] it's good um we're losing you so can you hear me agenda yes i can now okay perfect yeah so um we were able to see my when we queried to see how many validators there were i was able to see uh last the logs um but i haven't seen what you have pressed out for that okay let's i'll just show you this guy so if you can see my screen yeah um so here as you can see um there's two validators so we are able to create one validator um as we did right now okay perfect does anyone have any questions um the next uh code with us will be covering the setting up the kms i don't think we have questions so far so in this session um you are able to to to see exactly how to set up a validator from scratch and in the next one miracle will show you how to attach um a kms to your validator and how to secure it better so be prepared for our next session and if you have any other questions regarding what miracle has done today now be the best time to do that because if not okay one more woo-hoo um i i promise to bring better wi-fi next time yeah please do okay we have one from tobias again uh what does the mac commission maximum change rate referred to and in which time interval does maximum change um changes actually the maximum change um rate for commission refers to the rate at which you're allowed to change commission um so if you um in our example we set it to five percent so if we wanted to change our commission to six percent that's the max amount we would be able to do so so we'll change it to six percent and then we would have to wait a period or we're able to change it again this [Music] um can differ from network to network that is using the cosmos test can you hear me um we kind of lost you for the entire explanation of the maximum change rate and the interval okay so we have this right now so maximum change rate refers to the rate at which we can change the commission so for example if you have if you have set your commission rate uh to uh 10 and you want to um go up to 20 and you have a maximum change rate of maximum let's say five percent that means that you can reach your 20 in two days because you have a maximum change rate of 5 per day that means you have to go from 10 to 15 and then from 15 to 20 to be able to reach your um commission um maximum change interval i think it refers to like uh the interval that you'll need to set up in order to um how often you can actually make the change correct correct marco yes and this will vary from network to network okay cool i guess we got it covered uh we are so sorry for um not being able to offer a better connection this time but we'll make sure to uh get marco in the best uh internet location possible next next time for this uh for the next workshop to not have this um problems again um i'll be passing a recording on youtube and also on call with us webpage another cosmos network if you want to replay this again and marco if you have any more thoughts or information for the people who attended today you can drop them off in the discord channel the codes i have also passed all the information that were linked here and i want to make sure that everyone knows that our next workshop is on the 21st of july um we'll have people from chainsafe presenting the workshop and they will walk you through on how to build on ether mint which is currently actively developed by them through a grant by interchange so see you all next time and if you have questions don't hesitate to add them in the discord channel and we'll be happy to help you with everything you need and thank you marco for having the time to to host this workshop and we'll be looking for for the next one thank you for having me and sorry for all the wi-fi trouble don't worry it happens okay thank you everyone for joining and see y'all next time bye [Music] cha-cha