EVM Internals: Solidity, Bytecode, Memory, and Storage

[00:00:00]okay hello and welcome everyone this is the etherium engineering group Meetup today David and I are going to talk about the EVM solidity and what solidity looks like as a program in the a VM and how it uses code storage stack memory and other things David you want to introduce yourself hi David hardwood from consensus I work with Peter and the Pegasus business unit on the team X and all of us are here at team X are pretty much present I think so if you want to speak with any of the others doing this area of work please either ask questions or introduce yourself to people that you may not know Peter say thank you so before I start I'll say that David and I were brainstormed this talk and we're going to work on it and do about people part but David's got super busy so um you won't hear David talking as much as I am but he still though there's a lot about this stuff so yeah all right let's share my let's do this so if I go play there just a sec as Google slides catch up catch us up and so does some let's move that over there and go present on that screen sorry just getting my own screens to work and now what we'll do is we'll share screen and we'll share desktop too okay so you should all now be able to see my slides can people see the slides yes all right brilliant thank you okay so this is a talk on the EVM and the yep put will put together the talk and we've had a lot of feedback from these people which has been very much appreciated we're going to walk through an introduction to the overall topic and then talk about aetherium transactions and how they relate to it all and walk through slowly but surely if we get to about one and a half hours in and we're not finished what we'll do is we'll reconvene this time next week so we'll see how we go so the color there are some examples in this talk and there they can all be found in the etherium engineering group solidity examples the bytecode output that you'll see was generated using some tools that we've created and the stack analysis was created using some static code analysis tools so when you think of the solidity language you've got a file that something-or-other soul and that's your source file you put it into the compiler and normally you get out the ABI in the bin you can specify parameters to say what you want to get but those are the two files you normally end up with and your binary file has your is your output of the actual bytecode and the ABI is an interface file which tells applications how to interact with the code so essentially what functions your code supports and what parameters they take though it's worth remembering that solidity is but one EVM language so for instance there's a language called Vita which is in alpha at the moment and it's a compiled with the VIPRE compiler all of that said we've done some recent analysis of all of the deployed contracts on main net and on the numbers are indicating that it's well over 90 percent of the contracts on main network created with various versions of solidity yeah and so I'm not going to give you exact numbers because our analysis needs just a little bit of tweaking so once you've got those that binary file and the ABI you then put it into some sort of salinity wrapper generator and that all generator say a Java wrapper which essentially is an interface that you can call from your Java code to call the solidity code inside of the etherium node if you were using javascript obviously you end up with a Jas file or maybe you can just use the library interact directly and so once you've got your application code and your wrapper code you use a library some sort of web 3 library and then that talks to your solidity code via the Internet and your solidity code will be running inside the a VM inside an aetherium client inside of the etherium network so an aetherium client is an instance of software running on a node in the network and a client given we're in a peer-to-peer network is both the server and the client so it's the essentially the machine that's operating the etherium network and so yeah so you've got to think your application is interacting with the solidity code that is now bytecode inside of the a VM okay so I think I've just skipped you so in this talk we're going to focus on that byte code in that code dot bin so that's the code that gets executed by the EVM so in aetherium you have transactions and these are the things that are sent over the wire between the application and the etherium node and and also then around the network and so you've got a nonce and so that's the value that starts at zero you've got the gas price which is how much you're prepared to pay for gas in terms of ether or in particular we where if we is a unit of 10 to the minus 18 of ether you've got the gas limit which is the amount of gas you're prepared to use the to address or who you're sending the transaction to and the value and the data to be sent to the to address and the signature so if you're just transferring money between accounts then you have it set up like this where you you always specify just twenty one thousand because that is how much gas you need to use for a base transaction and the address you're sending the ether to and the value and the data is usually empty but it's actually worthwhile pointing out that the data doesn't have to be empty it's not mandatory and so if all you want to do is record a commitment on to the blockchain you could put some data there and then you'll have that as part of the overall blockchain so it's important to remember though that it's not going to be part of world state so it's not going to be part of the state tree but it will be part of information that gets linked into the overall blockchain because the summary of all the transactions so the transaction route is part of the block header if you're going to deploy a contract the to field has to be empty and that indicates to the to the etherium client that you want to create a contract and the data field contains the initialization code so essentially a code fragment which is going to set up this state of the contract and deploy the contract and if you want to do a function call then the data field is the other parameters so what function do you want to call and what are the parameters and it's really up to the EVM code to process that so code that conforms to the ABI specification takes a function selector and parameters however EVM code does not have to conform to the ABI spec it could do anything and as such that data field is just a blob of data that the EVM code can do stuff with yep and there's a link to the ABI spec so stack memory storage so all of the locations you can do stuff within the AVM so the EVM so aetherium is a stack based processor and what that means is that there aren't registers you've only got the stack and so when you have do an operation you use stuff from the stack and then you so you take input from the stack and you pop that input off and then you push on to the stack the result so and is popping off the stack stack Cesaro and stack one being seven and two it's adding them together and pushing the result nine onto the stack and so inside of the EVM you can access and store information from six different places so there is the stack so you can push stuff onto the stack and keep it there for a while as temporary storage the stack isn't persistent so at the end of the transaction the stack is gone you've also got full data so you remember the data field that there was in a transaction so that information is available to the contract so when that transaction comes in that call data is a read-only memory area memory is a temporary place of storage where you can store stuff just for the duration of the transaction so it's like a scratch pad then you've got storage which is a persistent data store you've got code and that's the code you're executing but you can also use that for static data storage as well and then you've got logs so this is a writer only output area and so when you are thinking about storage and how it all works you've got to think you've got a block header and it's got you know the parent hash the AMA is the block number and all that sort of stuff and it's also got something called the state fruit and so this is the root hash of a local Patricia tri of all of the accounts and if you don't know too much about this area have a look at the YouTube video from two weeks ago where hurrah Theo and I did a talk on stateless and we do talk about the merkel patricia tries and how they store information so anyway it within this try you've got a key which is 160 bit account address and so then within each account you've got account state and so that consists of say the nonce and the balance and the code hash and a storage root and so within the storage area you've got another mill called patricia try and this now try has got 256 bits of key so each key is 256 bits and so when we talk about storage this is what we're talking about we're talking about something stored against the account ok over david to talk about the yellow paper thanks Peter so the etherion virtual machine is defined in the etherium yellow paper it is and has been kept up to date since the first version and so there have been changes to the yellow paper along the way but if you're going to github at this URL or just to a search for the etherium yellow paper it'll pop right up to the top the paper was originally written by Gavin wood at the beginning of the etherium project but since then many people have contributed to this paper on next slide please Peter toward toward the back of the yellow paper you will see that all of the authority mop codes for the etherium it's all the thurmond virtual machine op codes are defined this is the the canonical location of this information so what's here is the definition of the etherium virtual machine it's a virtual machine that implements the totality of these op codes and the way they're defined next slide please Peter look at the first few here you'll see that you've got values in hex that start at zero and they just increase so the very first opcode is this is zero X zero zero that's stop next one is zero one that's add etc on down the list all the original all the arithmetic is modulo two to the 256 which is the same 256 number that you saw in that local petition try repeating on Peters last slide and the description of each opcode goes through you'll see that stop doesn't take any it doesn't take anything off the stack doesn't take any input it just halts the execution add Peters already showing you in a previous slide where it takes two items off the stack stack item zero and stack item one adds them together then pushes back on to the stack opcode to MU L or from multiply does exactly the same thing as add except the operation of the computation is to multiply things together instead of add this is pretty much the way this level of any machine code looks like and it's going to it's going to be very familiar to all of you who had this level of computer science it's it's pretty straightforward stuff if you if you look through all of the op codes you would find that it's all the typical sort of things you get in a compiler class with the addition of some things that are very specific to the etherion virtual machine Peter already mentioned that their stack operations it's a stack virtual machine and the higher op codes are the ones that are more specific to aetherium operations Peter thank you David yeah so now I'm going to talk about contract deployment constructors at a knit code fragments so this is some solidity code and so we've got a constructor which is public and we set one of these storage variables to three and so the Anette code includes code to deploy the contract plus set up the state so in other words setting var valve two to three is part of setting up state and so you know you compile the code using that to produce the dot bin file and you get something like this and so in this code here you've got the anit code fragment and the actual code to be deployed and so it doesn't have to necessarily be this arrangement but for solidity it is so as I've said before contracts are deployed using transactions where the to address and dress is not specified and the data is the annette code fragment and so the unit code is not stored on the blockchain that makes a lot of sense because it's the constructor it's only ever used right at the start to set things up so it doesn't need to be stored on the blockchain and this for this particular type of transaction the data is actually treated as code so you might remember I said that you know you can access the call data so for a contract deployed transaction the call data is actually treated as if it's code and executed as code so if you look at that binary there you start out with 6 8 and so 60 if you looked at the yellow paper is the opcode for push and pull push one in particular meaning push just one bite and so that's pushing 0x80 onto the stack yeah and so if we completely if we decompile a bit more turn it into opcodes this is what you've got so we pushing 8 here and so push the word onto the stack so the stack has a depth of 1,024 so you can have a thousand 24 things on the stack and so when we use the term stack 0 that's the top of the stack and stack 1 is the 1 just 1 below the top of the stack and the stack items are one word wide where that one word is 32 bytes so when you push in one bite on to the stack but I push 1 then the top 31 bytes are set to 0 and so after this has happened you have 80 as the top of the stack and then we push 40 onto the stack and so the stack now looks like you know 40 at the top and 80 is the next one and then we have something called M store so M store is store a word to the temporary storage memory so that to that memory area and so it takes two values off the stack and so stack 0 is the location to write to and stack 1 is the value 2 right and so what this is going to do is it same will set memory location 0 x 42 value 80 and so what this is doing is setting up what's called the free memory pointer and so the idea of the free memory pointer is that the solidity code that runs after this can load it up and it'll know that anything after 0x80 is something that it can use and so if it allocates some information into memory then it will need to update the free memory pointer and so that's the idea it's interesting to note that 0x 40 is a byte offset and not a word offset and so though we're storing whole words it's actually a byte offset call value puts onto the stack what the value field of the transaction was so how much we has been sent and then you have dupe worms so dupe 1 duplicates the top of the stack and so now we've got the stack looking like this where you both the stack 0 and step 1 contain the value that was sent with the transaction and so there's also dupe 2 3 4 all the way up to 31 and what they are doing is they will duplicate different levels into the stack and put them onto the top of the stack and that can be used for RAM stack manipulation so is 0 pops a word off the stack and if the word is 0 it puts one onto the stack otherwise it puts 0 and so what this is saying then is or if the value is 0 put 1 but if the value wasn't 0 put 0 so then we push 2 bytes of the onto the stack and so this so push to means push to bites and then jumpy is jump if and so what that is all about is it saying that if stack zero is Oh jump to stack 0 if stack 1 is not 0 and so it's going to jump to a location or go straight through depending on the value of stack 1 and so yeah so if that value wasn't 0 then jump otherwise don't and then it pushes 0 onto the stack and then duplicate 0 so then you end up with the stack that looks like that stack 0 and stack 1 both being 0 and then we have a revert instruction and so revert helps the execution and indicates that a revert has occurred and it uses stack 0 as a memory location and stack 1 as a length to indicate the revert reason so the reason why the code reverted so the stack had zeros at the top so in other words we're saying there is no revert reason so all of that code together is really saying look this contract this constructor wasn't payable and so if someone sent value to the construct as part of the unique functions part of this contract deployment then we should revert because the contract is not supposed to have ether sent with the contract deployment so when you have a jump whether it be a absolute jump or a jump I am if jump I you always jump to a jump destination and so the program counter so we had pushed to and pushed zero x10 onto the stack and then we jumped to it if the value was 0 and so that's going to get you to program counter zero X 10 and so yeah if you jump to an invalid jump destination then your code will stop and you'll have an error pop pops the top the value that's on the top of the stack off and just throws it away and so now we're yep so it's worth recalling that we were setting Val 2 which is the second storage location to the value 3 so what we can see here is we push 3 onto the stack we push 1 onto the stack and then we store it and so s store stores a value into storage and so it takes the stack 0 is the location to write to which is going to be 1 and it's going to take the value which was 3 and it's going to store up there so storage location 1 becomes 3 and so that matches what we're seeing in the salinity code there so now what we do is we push some values onto the stack and so and what we're doing is we're getting ready for code copy so you might remember the code the runtime code followed the ANU code fragment and so what this is doing is it's getting getting ready for the code copy operation so we can copy the code [Music] from where it is at the moment to the right location so we set up 24 and then you can see that here on the screen after the invalid operation at PC 23 we've got PC 24 which is push 180 and so that putting it from 0 X 24 that's the start of the runtime code and so code copy copies from the code that is currently executing to memory and so stack 0 is the offset to write to stack one is the offset to read from and stack 2 is the length of the Vikes to be copied and so the stack ends up looking and so the stack looked like this so what we're saying is let's write to memory offset zero and start at offset 24 program counter 24 offset and do it for 0 XC 1 bytes so which is the length of the runtime code and the thing to note is you remember how we set up the free memory pointer so we're just wiping that out so we set that up in memory but now we're just going to copy straight over the top of both and that's not going to matter because we're going to exit but it shows you that setting up that free memory pointer just for this piece of code didn't actually help and it was a bit of a wasted few operations yeah so yeah so we've copied the code yeah ok so overall that block was copy the contract's code to that's going to execute on the blockchain that we want to have on the blockchain to memory and then we returned and when we returned from a init:function so a contract deployment transaction stack zero is the start offset in memory where the result is and so stack one is the end of the result so that's the same for any return function but the results that's returned with the return function for a contract deployment is the code that should be copied into the blockchain yeah all right which is what that just says there so store this code into the blockchain against the new contracts address so there's an invalid opcode there too which should be on which is unreachable because you've already returned and so the returned followed by invalid indicates that this is the end of the unit code so in summary we set up the free memory pointer which in this case wasn't used we calls to revert if any wean was sent with the transaction because the constructor wasn't payable and we set up a storage location to a non-zero value and so we could have arbitrarily set up lots and lots of them and cutting complex structures as well and then we return to point it to the code and the length of the code that needs to be stored into the blockchain okay so what about function calls so when we look at this code quiz question Lukas how many how many functions are we going to end up with we gotta wind up with three oh yeah cause I do that go there yeah sure good one P you got me the fields of functions yep the fields of functions so you so when you're writing code solidity code you want to have a very close look at this so if you've got variables you can have them private and then you're not going to have X access of functions in your solidity code so if they're private you don't need to access them programmatically then just have them private if you want to access them if you just have them public then essentially getter functions will be automatically created so what this means is that we actually have three functions that we should expect to be created for this code one to fetch Val one one to fetch fell to and want to execute set so at the start of the code we're going to set up the free memory pointer again and then because none of those functions are payable it does the let's check to see that the value is zero again and if it isn't let's revert and then we jump to the next block of code and so we caught the young the code automatically copies the value onto the stack and I think that's just in case any of these functions were payable and you need to use that value but so it pops it off so called data size is a opcode that tells you the size of the transaction data field and less than is going to tell you whether stack 0 is less than stack 1 and it's going to pop the values off and then push on put one onto the stack if they're less than or 0 otherwise so what that push from one for down to the jump I are all telling you is if the transaction data field it has less than four bytes then we want to revert and the reason why it wants to do that is related to something called function selectors so a function selector is a truncated message digest of a function signature so you can think we had a set brackets you int 256 function and so if you message digest that and then have truncate that down to 32 bits then that's the function selector for that function and so when we're doing this revert if less than for what we're really saying is if we don't have a function selector you know so if we don't have enough byte still at least have a function selector then we should revert and so then if you did have enough in more than four bytes then you go to a jump table and so you execute called a load and so called a load only loads 32 bytes at a time so one word at a time on to the stack and it loads starting it off set pointed to by stack zero so you can see we've pushed zero and then called out a load so we're loading up the start of that call data so load up the first 32 bytes and so she sure is shift right so shifts the stack one to the right stack zero times and then pops them off the stack and then put some the result back on the stack so the we've had stack one so stack zero had the yeah yep sorry should have had a little diagram here saying what was on the stack so call down a load has loaded up the value of the called a diversity two bytes then we're going to shift by 0 X a 0 to the right because that's the top of the stack and so essentially what that's going to do is it's going to shift the top 32 bits from the top of the word to the bottom of the word and so what this is really doing is it's loading up your function selector and then once we've got the function selector we duplicate it and then we say let's push a value on to the stack a essentially a function selector and we say well does the function selector that was supplied in core data equal this and if it does well let's jump to offset 41 and so if you see in code function selectors you can go to this website here and paste it in here and search and it'll tell you if it you know what it thinks that function selector might mean now because it's a 32-bit hash or truncated hash you're going to have collisions so just because you see that it comes up with a certain thing it might not necessarily be there but if you have your own if you if you know if you have your set of functions that you think might it might be and it could help you differentiate between say two or three different functions okay so and then with this next bit of code here it's going to jump to 5d if the function selector is that and another jump and so if the function selector isn't what are the ones that we know about so if if the the function selector passed it in as a parameter isn't when we expect then we don't know about this function so we revert okay so this is oh yes so then mention something that I didn't realize so when you're looking at the offsets in the bytecode they're put in ascending order and so then we're saying well you know what you could do is you could try and work out a function name that produced a function selector with a low value so it was going to be higher up so if you're important and oft used functions would be higher up but a extra fun fact from myself is that you've got to be careful with that because if you have more than about four functions so maybe eight functions it starts to do a binary search so essentially it says if it's greater than this jump to here otherwise go straight ahead and I think I'm seeing a few that have got say thirty-two different functions you know solidity contract and I'm pretty sure they actually do a binary search until they have only about four functions or so so yeah you'd want to actually look at your code before you started doing micro optimizations like that okay let's focus in on this set function so we've got a parameter on that we were expecting to be passed in via call data and we're going to set it to that very first storage location so the first thing code does is it sets up a return address or a end address and so what this is is that the code invariably has some code that it's going to call right at the end to exit whether it be just to stop or maybe to return a value and so invariably the way the compiler is written is such that you when you enter a function you push this address that you're going to go to at the end onto the stack and then what we do is we work out the core data size minus four so essentially what's how much call data do we have if we exclude the function selector and another Before we jump into analyzing the next block of code something to understand is so if you had ever seen LT followed by is zero you've got to think that what we're doing is so lton less than would set one if it was less and then is zero we'll set up to one if it's zero or set it to zero if it's not zero so essentially LT followed by is zero is going to give you not less than or greater than or equal to so what then so we had the call data - the size of the function selector and then we push 20 which is our X 20 which is one word onto the stack and we say well look if we don't at least have so greater than or equal to one word then we should just keep on going straight ahead and revert but if we do go to location 55 so then we pop the value off the stack get rid of it and called out a load so what we want to do is we want to load the word at offset four from the transaction field onto the stack and then we push 7b and drop to there and so that's essentially 17 and jump is just a hard jump so yeah which you can think of this is saying is push underscore per am on to the stack and then jump and then once we get to 7b we push a zero on to the stack and then explore so we're storing at location zero the value that was next down on the stack which was the per am so store underscore per am at location zero and then we jump to the location we've set up earlier five B which is there and then we stopped so stop means end execution indicate indicating that the index acute in status is successful and don't return any data though so this is stop is what you'll get if it's a transaction that doesn't you know which doesn't return any data all right what about Val two which was returning you int 256 so focusing on that so it's just a vehicle that's auto-generated and its signature is just dull two brackets all the view returns and brackets UN 256 is not part of the function signature so if we jump to there first thing it does is it sets up its return address which is 63 and then it loads the storage location 1 and duplicates it actually to 2 which is so to do is give me the value from not the top stack but the one below the top stack so if you recall 63 was the return address essentially so we've s load will of pushed fell to onto the stack and now we want to get sixty three and so we put that there and then we jump and once we get to a location sixty-three now we've got this on the stack and you'll notice that the stack one is never going to be used so but fell two will be so stack zero so we load up the free memory pointer and then we manipulate the stack and then we duplicate that free memory pointer so we swap the free memory pointer down and then duplicate it back to the top of the stack and then what we do is we store Val two to that location pointed to by the free memory pointer and then what we do is we load up yet their free memory pointer again so we load the free memory pointer again so we've got a copy of it and we do a bit of some stack manipulation and then what we do is we say well how big we're essentially creating how big is the return value that we're going to be creating and what's it off its offset and then after all of that manipulation we're then ready to return and so remembering returns says stack zero is the offset of the result and stack 1 is the length of the result and so what you're doing is you're saying all right and where the free memory pointer was pointing so 0x80 i've got one word and please return that which is the value so fell to is at that location so putting all of that together for that block of code what it was really doing was it was returning a UN 256 that was put into memory so let's focus on actually with this is Val one this is an error in the slide and you're now focusing on Val one not Val - because we've already looked at Val - oh sorry so what this is showing you rather is that what we were trying to do is return Val - which is what we've just done so now focusing on Val one okay so the first thing we're doing here is we're setting up the return address again and something to note is the zero X 63 is the same return address that we used for returning Val - so we use it we're reusing the same code between two different functions because that piece of code all I was doing over saying I've got a u in 256 and I want to return it as the result and so because it's common code it can be reused between the two functions the two view calls and you notice that in when it's getting to zero X 87 we're doing a nest load from location 0 so very similar code and it's just essentially reusing code which is interesting it means your byte code will be smaller so what about you know I ever have a question for you is that the result of the compiler optimization if you would turn optimization off when those have been separate with the duplication have been present in the bay code that is a fantastic question I suspect the answer is correct yes true a true statement is that the code has been generated with optimization set to on yes so you may you're probably correct that there will be things that duplicate code if you don't have optimization on I suspect though yeah reverse engineering or you know like getting the opcode from the bytecode would tell you for sure so what about if you want to be able to have people send either to the contract and so what you do then is you know so you could have a function which except is payable but if you just want to have normal transfers normal non function call transfers you need to have what's called a fallback function and it needs to be payable so as well a fallback function is the cover all someone's done a function call to your contract but the function didn't match any of the ones that it's got so you still want to accept that function call and do something and in particular you want to accept the ether sent to it yeah so make some breaking changes on v6 around callbacks now and I own a compilers for but I think I've changed their fall back into two functions now oh yeah the link yeah please do I compiled this using version 5 point something of other yeah alright interesting to know I know that there have been a few chain breaking changes with respect to fallback function so for back functions are one of these things that I know when I first saw the I thought Oh yuck you know I mean the idea of someone doing a function call to your contract and the function not matching anything you've got that you accepting it anyway seemed a bit dodgy but yeah having a payable for call for back function makes sense if it's just to accept a direct ISA transfer but a function call that didn't really work should just fail yeah all right thank you Lucas yeah so if you do have that callback function how does stuff change so now if the called data size is less than four then we're going to jump to a zero X 30 which is now going to rather than being a revert is just stopped which means that the eath will be accepted and additionally you've got your normal function block um but if none of them functions match you'll just fall through and again you'll just stop and accept the ether another side effect of it is because the you've got a payable fallback function but the other functions are non payable now each and every function needs to check whether the core value is zero or not so you you will end up with more code okay so the next area of of the talk is about storage so storage is set up as 32 byte words and variables that are smaller than 32 bytes will be stored in the same word and so you end up with something like this where the byte is stored by itself in storage location 0 and then you've got a whole word so that's in spiritual location one and then you've got storage location two and then three and so you've because they perfectly match the right size you the UN you went to 32 64 and address are all pushed in together if you had a bit of spare size that you can imagine you could have moved your storage around your room you're ordering of your variables so the by actually could have been part of that word that storage location too so remembering as well that I'm setting up these each of these memory locations to 10 11 12 13 14 and 15 hex so his first block of code is going to set that initial bite to 10 so what we do is we load up the location so it's even though the only thing stored in this storage location is that bite the code isn't smart enough to work that out so it's still going to assume that this is just one bite of a word that is being used and so it loads up just one bite pushes the tens or x10 the value that we're going to store there and then it loads up FF and does not so that'll end up giving you zero X ffff with zero two zero zero at the bottom so essentially a mask which you then and with the current value in that storage location and then or in the value that you want to put there and then you store it so you've got to because it's only one byte you're masking it in if you compare that if you if you had have had that same value as just a you went to 56 then you would have only had these three opcodes here so you would have used up less fuel up combs I mean it should be of course noted that push and swap and other codes like that only use tiny tiny tiny amounts of gas compared to where store so it has very little actual effect because their story is twenty thousand or five thousand depending on if the memory local storage location has been used before whereas some of these others are only say three or five or something like that so for storage location two we've actually got quite a lot of code and so what's happening here is that we load it up at the top of this code block we mask in each of those fields each of those values and then we store it so this is important because every time you do a nest or you're using 20,000 in gas for 5,000 250 locations already being used so just doing 1's Dorf all that is important it is also so you're so having these three variables in one location means if they're being manipulated at the same time you're going to use fewer s loads and s stores which is going to affect the amount of gas you're going to use quite substantially and again this one here is just a word so what this is telling you is that well laid out storage variables will result in fewer locations and fewer location well I say will but it's probably cord to save you money it will depend on yours usage patterns because it really comes down to how many times do you call that s store and s load operation so in the example there there was no way of having that Val bite that first bite in with the other storage locations but if there hadn't been a spare room you could have pushed it in with them so that you would have had fewer storage locations okay bites and strings so bites and strings are variable length byte arrays and they behave in the same way so you can do something like this where you're pushing a byte on to the end of an array and so the bytes is allocated a storage slot and so given the code and we only had so here we've only got one value that we're in having in storage it'll be stored in storage slot zero and you'll have to excuse this diagram a few people have pointed out that by it to zero off to the far right here should be as big as byte one and all the other bytes so it's not that biro has got more bits than by one it's just that it was harder to draw it but so if you've only got 0 to 31 bytes then you have your byte zero has the bottom bit set to 0 to indicate this is a small number of bytes and it has a length field which is 0 to 31 and with the top bits being 0 and then it alik eights bytes starting at the top of the overall word and going across and so all of this is stored in the actual storage location for the variable however if more than 32 rights are stored then the market sorry about that Susi's obviously got some thoughts on variable length byte arrays yeah as long as well she should Peter yes so um the marker bit indicates that you've gotten 32 or more bytes you know you know and so what happens now is the storage location zero just holds the length of the overall bytes array and then the storage location which is given by the message digest so kick act 256 of the storage slot number so we were in storage slot zero so that's going to be kick ACK of zero and that is going to be where the bites are stored and if you've got say more than 32 bytes then by 32 and more will be stored at kickoff the storage slot +1 and so if you had 2 bytes arrays then one of them will be saying storage slot 0 and another one will be in storage slot 1 and so if you do a kick out 256 of 0 and 1 you'll end up with a completely different part of the number space and so you're not going to have collisions so what does this actually look like in bytecode you know in opcodes so we push that return address on to the stack to start off with we work have the get rid of the function selector from the call data size and then what we do is we check the length of the variable that's been passed to the parameter so you might recall we're only after one bite however in solidity you need a bi all parameters are passed in as words so 32 byte words so even though we've only got one byte it's passed in as a whole word and so what we do is we check that we've got at least 32 bytes and if we don't then we revert and so then what we do is we load up the we load up the core data the word and we create a mask and so you could go push 32 and 0 X 0 0 and lots of F's or even push 31 and just lots of F's however these up codes here 44 we should be right now everyone my kids at home yes so if you look at that you the fact that we are run from 44 to 4c we're setting up a mask and the reason why you're doing it that way rather than pushed 31 all ifs is because this ends up being fewer bytes and so again to David's point earlier about optimization I am pretty sure if you had the optimizer off you would be pushing 0x ffff so we mask off using that end operation to mask off the parameter and then we jump to 53 Higgs okay and then we do some stack manipulation and we load up storage location 0 and we mask off the length part of storage location 0 so three F so three F is thirty-one shifted by one and with the one extra bit and you end that in and so if you can think I'm going to go back a few slides here if you can think of this length field we're trying to mask off the length and the bottom bit and so then what we do is we push three e onto the stack and we say well is the length is this length Plus that bit perfectly equal to three eggs if it's perfectly equal to three e then that means that the actual length is one F and the bottom bit is zero and so if it is then we jump here and then we've got the stack looking like that and then we store the bite or which we store the yet we set up the memory location to be zero install the memory 0 to be 0 so we know we're using the location 0 as a scratch pad so you remember how we had 0 and then we had 40 hex was the function a free memory pointer so location 0 is a scratch pad and so we use that scratch pad to record the off they essentially the storage location that we're going to use and then we use kicker 256 which is denoted as char 3 so though the opcode says char 3 it's actually doing a kick act 256 and so what that's doing is it uses the memory location pointed to by the stack to do the message digest and so it's going to do message digests of offset zero in memory for 20 bytes but 20 hex bytes which is that one word and then what it's going what it does is it gets rid of the length byte that currently exists and it stores the current information that is currently stored in the bytes into that kick hack of zero and then it stores the updated length and so the reason why it's doing this is that now we're in a good state to actually process the BIOS so all of the work that we've done before this has just been saying alright we've gone from having 31 bytes now we're going to have 32 bytes so we need to change the way the length field operates and so you can see there's an awful lot of code there to manipulate and get bytes in just the right location so that's why it's another thing we've got is variable-length arrays so storage arrays and then you can have you in 256 arrays byte arrays you can have arrays of pretty much any other types including complex types so values for an array are stored at a memory location which is given by the message digest of the storage slot number plus the offset so I say key but it's really the offset in the array and so you store the value into that offset and so the number of elements in a dynamic array is stored using that formula so storage and then slot number and so if you've yeah so as you make the array bigger or smaller you put it into that value there you enter that into the storage slot 0 so for the array 4 mappings mappings is a way of mapping between one type and another type in a sort of key value store where the value can be arbitrarily complicated so for mappings what we do there is you have the key and the storage slot number and they're concatenated together and so if and then you do the kick out of that and you set the value to that location so you should note that for mappings nothing is actually stored at the storage location slot number itself so you can't have mappings of mappings and mappings of arrays arrays of mappings etc and so what you do there is you do I recursively do that kick at 256 to one work out the location where the actual data should be stored and have a look at that link for details so we've talked a little bit about memory as being a scratch pad and you might be thinking well I could write some arbitrary code evm code and I could push I know a huge number onto the stack and I could say store 0 at that location and go in store and then you know these are theorem clients would have to try and allocate that memory and then they trash because they'd run out of memory so each time you use an extra memory location it cost gas so you have this memory expansion thing and so if you know probably but again you will gasp and so before you cause not a memory error on the etherium client you're going to run out of gas and so I guess what this is saying is that you want to be a bit careful about say allocating huge arrays or huge other variables on as memory because they cost gas so the layout of memory is so the first 40 hex bytes are a scratch pad and they're used for hashing methods for when you're doing things like creating the working out the location variables should be stored in storage for mappings and arrays the next location is that free memory pointer then you've got 0 and then you've got your location that your actual memory starts at so if you say creator you int array in memory then it'll be stored starting there for instance so just quick question on on memory before you go on yeah so I don't see any protection in aetherium unless it's implemented at the client level that I'm unaware of or maybe I guess potentially you know at maybe the web 3 layer I don't you know if you were to create a client and you had your own implementation of solidity wrapper dot Java right or or your web 3 library you could inadvertently or advertently create a situation where memory just was terribly corrupted and it wouldn't be hard to do that I guess Lucas is about ready to find out how hard that is right and so it's really up to the it's really up to the client to keep the keep any memory corruption issues happening I guess is my point right I know it's I'm so memory it's the solidity compiler so very much in Lucas's domain to make sure it's ok one thing that would be interesting I think if you tried really hard I reckon you could get base 2 storage locations to overwrite each other so you know like if you've got a haven't tried it and so maybe there's some protection against it but if you can imagine that you've got a kick act 256 of say storage loose lot zero is an array starting at that location and then you've got another one starting at a different location in the overall storage so if you did a certain offset into that array that just happened to match the other value I do wonder whether you wouldn't overwrite it so that's storage but you would have to try hard you know you have to surely yeah what sorry go ahead sorry there is an attack known attack like that turns out that there is no protection against and they're flowing the index of an array so even though it's it can be in principle they are very far away one from another because of the hashing of the of the other says but still it's doable and I think it was done in some even in some known attack so yeah in the stories it can be done not not because of hash collision but just because of using the way in which the index of the array is moved around yeah yeah and so I guess what that showing is that yeah you could yeah you could create if your solidity code essentially took in an array index via the API and then fed that in then yeah that would be an interesting attack to to do it we should do a talk at some point where we specifically write the code to daughter yeah I'm sure we could get that to work shouldn't be too hard at all well shouldn't it all right it'll take a day take a day to actually get it to work and write the slides how's that yeah but as far as memory goes though ah no look I used a fair bit of Thursday and Friday on the slide so I've done my slope writing anyway I've got another talk in two weeks time so that's going to time but now we and we'll do that as another talk for sure but you know I think in memory though you don't have dynamic sized arrays and so it's all fixed size so I think if you for corrupting the actual memory compared to storage I think you're gonna have to have a bug in the solidity compiler yeah okay actually before I go on there is one extra point to make and that is you might recall how each account storage is completely separate so what that means is even if your contract or someone else's contract is a bit buggy and allows people to won't do all sorts of nasty things their contract cannot trash your contract because you've got a completely separate storage area which is really important okay so yet another location that you can access stuff from is code and so you can use code copy or external code copy so code copy copies information from your contract into memory and we've seen it was used in the net code fragment but it can use it for other reasons and you can use external code copy as well which copies from a different contract and so I've seen this used for revert reason there are messages and so I daresay if you've got any static strings then they will would likely be stored in code and not saying storage yeah also though if you wanted to have your contract as a contract that deploys another contract on demand you can imagine you would load up all of that from your own space and the reason why you might say use a have some static data stored in a contract rather than s door and s load is because s store and s letter dramatically more expensive than code copy and writing to code okay before we end there is one extra area of to look at and that's aux data auxilary data or metadata and so modern solidity and I'm not sure which version this is from but has optionally as metod are added to it and it's bytes that are added to the end of the contract in C bore encoding so that's a binary encoding format and what it does is it indicates the swarm of ipfs message digest so Northey so it's the message digest of the source code and it indicates that this in this code is stored on swarm or ipfs at this location and of course there you've got to upload the code yourself but the idea is that in combination of the compiler name and version which are also there and whether it's an experimental version indicates to you how to compile the source code to get this bytecode and so you can use it to automatically verify that the deployed contract matches the source code because you know you can have these tools automatically scan so if you have a look at the orchestrator you you've got something like this where you've got a map and then you've got a length and so bees edit our 0 we're swarmed that you could also have visited our one or ipfs and then you've got a length of your message digest and then you've got some characters which might say say soul c and then the version of the compiler so there you are and Lucas always using 5.10 yeah and have a rate there of the docks so in summary the EVM is a stack-based processor that has access to all sorts of stuff including the stack and so called data read-only it's the transaction parameters memory it's a temporary storage area and it's not persistent storage is persistent and it holds the world state or is part of world state code you can have your you've got your sourcing your actual code your executing plus it's got static data and also output wise you've got a write only event log and there's so much more to cover that wasn't covered and you know like trust contract calls what do they look like and why don't you know why do you use the various types of cross contact contract calls what about error handling so exceptions and has have been added recently we've also got asserts and requires and revert so how do they all interact I didn't really talk about logging and what do they look like at the bytecode level are they efficient how do you make them efficient I didn't talk about bytes 32 or bytes 1 and compare that to you went 256 and so I probably there's more work to be talked more to be talked about about memory variables so we we've been doing a project on code localization and so we've been analyzing all of the contracts on main net and there are some really weird and wonderful contracts out there and you could almost do a talk just looking at and trying to understand what was the developer thinking and what does that contract actually do and why it might be interesting to look at say a real contract that people use a lot like ERC 20 and yeah or maybe some of the other important contracts and we've already added another bullet point to the list and try and write a contract that has a storage collision will a storage overwrite but it would be interesting for people to have a think about what else that would like talked about and so this will appear on YouTube soon enough and when it does please tell me if you've you know if you've got new ideas put comments in is already a decent decompilers Oh from byte code to solidity do you mean yeah yeah I I haven't seen anybody I haven't looked so I've written some code that does static code analysis which at least will tell you which of the byte code is part of which function and you know gives you the core path and also shows you what the stack looks like at every point of the code execution so it's not a decompiler but it certainly helps you understand what's going on and it also takes byte code and gives you that printout that we've seen during this talk but not solidity but yeah I don't know I do a web search maybe someone's got something but I think yeah I don't know i think given it's on your 32-bit space you could scour the internet for all of the solidity function names from all the open source projects and i reckon you would get a pretty big data set that could be used for reversing what the function could be and then based on the parameter usage that would give you an idea as well yeah okay so we're going to be doing a few more talks and in two weeks time we're going to talk about code mobilization and then in four weeks time Ragavendra is going to talk about polynomial commitments and point proofs so code localization is part of a theory in wonder X and it is related to stateless and so essentially how do we have the ability to not have to send all the state so part of the state is the code that's going to execute so how do you work out which bits of code to send and so that's what code Merkle is a ssin is all about polynomial commitments is they thought that maybe there's a way of replacing the Merkel patrícia try and to do proofs and so they offer the promise of much smaller commitments then on the 13th which is a Monday we've got a conference coming up and so we've just finished the call for speakers just yesterday and so the early bird registration which ends on this Friday is Australian dollars 20 and standard registration after that will be 40 so we've got a whole stack of potential speakers and David Center and I've got to work out who the speakers are actually going to be sometime tomorrow but it looks like it's going to be an awesome world-leading lineup so please consider registering for the conference in August the fifth we've got Tim is going to give a talk on IP 1559 which is all about fee markets and so he's going to talk about the proposal and the various alternatives and just how this can be approached so Tim actually lives in Canada so we're going to do the a bit earlier than normal so three hours earlier so please consider coming to those talks are there any questions how that Lucas has dropped we can probably safely say it'd be nice to have the future talk and Lucas on his compiler if he gets some progress made on it Fisher Hall Fisher so was that talk all I can say we've got a ton of stuff that's gone flying by in the chat anyway hopefully my audio didn't break up too often yeah alright cool great - all right thank you alright well everyone have a great day and I'll talk to you all later goodbye