Understanding MEV and Flashbots: Ethereum's Dark Forest

[00:00:14]foreign foreign arbitrage august 28 2020 dan robinson research partner paradigm wrote a seminal medium article calling ethereum a dark forest he cited the 2008 novel by chi chi nu which introduces the concept of a dark forest an environment in which detection means certain death at the hands of advanced predators in this environment publicly identifying someone else's location is as good as directly destroying them in dan's piece he sees those predators as arbitrage bots that monitor pending transactions and attempt to exploit profitable opportunities created by them in this episode we are venturing deep into that dark forest to discover the truth behind the mysterious mev and lean into the magic of flashbots this is the defiant before we go any further just a quick word about our sponsor balancer who are up to some pretty funky stuff to help ease the pain of high gas costs on ethereum trade as much as you like and recoup most of the gas costs damn straight and in their new ball for gas campaign traders are receiving six figures worth of bowel tokens every week and with version two just around the corner balancer is becoming a one-stop shop for defy liquidity version 2 brings stable pools and weighted pools tightly integrated under a single protocol with flash loans lending via asset managers and much more check it out at balancer.finance now back to the dark forest it's hard to imagine when you're faced with uniswop's friendly ux and unicorn flavored design that lurking in the shadows are some of the most savage predatory forces you will ever encounter anywhere in your life and you'll probably never even know they were there as dan describes it arbitrage bots typically look for specific types of transactions in the mempool such as a dextrade or an oracle update and try to front run them according to a predetermined algorithm generalized front runners look for any transaction that they could profitably front run by copying it and replacing addresses with their own they can even execute the transaction and copy profitable internal transactions generated by its execution trace so if you want to get deep into the weeds here then you have to read what has become the essential textbook on crypto arbitrage bots flash boys 2.0 front running transaction reordering and consensus instability in decentralized exchanges it's a dense paper 23 pages long but you only need to read the abstract to get the big idea like high frequency traders on wall street these bots exploit inefficiencies indexes paying high transaction fees and optimizing network latency to front run i.e anticipate and exploit ordinary users dex trades we observe bots engage in what we call priority gas auctions pgas competitively bidding up transaction fees in order to obtain priority ordering i.e early block position and execution for the transaction we explain that such fees are just one form of a general phenomenon indexes and beyond what we call minor extractable value mev that poses concrete measurable consensus layer security risks and we show empirically that mev poses a realistic threat to ethereum today dark forests threats this all sounds spooky to the spooky to the a to the f what wtf is going on here and how does it affect you well simply put miners determine the order of when transactions are processed on the blockchain and can exploit that power to their advantage since each block can only contain a limited number of transactions miners have full autonomy in selecting which pending transactions in the mempo which is the place where unconfirmed transactions go first they will include in their block now while miners usually order transactions by the highest gas price or transaction fee in order to maximize their profits they don't actually have to mev is basically extracted from the regular user i.e you and me but often in ways that may not be immediately apparent until after your transaction is processed it might look like there was an unexpected increase in the network's transaction fees or sudden additional slippage on trades and yes you you poor sucker you foot the bill but the whole thing happens invisibly out of sight it's a most mev is performed by third-party bots hunting arbitrage between indexes now with the increased adoption of defy and growing liquidity on dexes profitability and frequency of this type of mev is going through the roof the bots engage in bidding wars for a place at the front of the queue in a block continually raising the transaction fee to win the slot we are robots now look this may not sound that bad in fact but the net result is that the network's bandwidth becomes completely clogged up by increasingly vicious arbitrage transactions which then raises the cost of transaction fees for everyone else trying to use the network another type of mev comes from bots who front run trades now because all transactions must first go through the mempool the bots can monitor for large trades entering the mempo and execute some pretty snidey maneuvers for example let's say a nice big juicy trade is spotted a front-running bot will copy that user's trade and then pay a higher transaction fee to get their transaction process first this moves the market price of the asset being traded then there's a chunk of slippage and after the user's trade is processed the market price of the asset moves even further in the front-runner's favor which allows them to take profits by selling their initial position this is what's known as the infamous sandwich attack basically you get screwed by a sandwich finally robotic beings rule the world now you might think this is really only for a very small group of very motivated and very technically competent traders like market making firms but if you want to get a feel for just how much revenue has been collected through mev then the place to go now is mev explore version 0 is a dashboard where you can see at a glance all the mev activity that's going on and trust me when i tell you it's kind of startling so we start here and if we look at the total extracted mev since the january 1st 2020 this includes successful and failed mev transactions and their associated transaction fees the number is 367.1 million dollars i don't know about you but that seems like a lot and in the last 30 days alone there's been 46.5 million dollars and in the last 24 hours 1.3 million dollars of mev so these are just bots going through and looking for opportunities to just just [ __ ] you up basically man and then as is the way with crypto there are some astonishing looking graphs they just seem to be going parabolic so this one here the cumulative extracted mev is just going through the roof this is insane the daily extracted mvv goes up and down as you would expect but the trend generally if you look at all the bottoms of the the uh the peaks here that is up like seriously up something else that's fun on this page you see here they say mev equals minor and then they've crossed out minor and change it to maximum extractable value there's a reason for this and that's because well firstly ethereum's moving to proof of stake which means that we won't have miners anymore and also the idea of mev is not exclusive to ethereum it happens on bitcoin but it can also happen on proof of state networks and it's basically just to describe this moment of front running or using the mempool to goose trades effectively there is some more data here um you can see that the majority of mev happens on uniswap no real surprise there and that most of it happens in arbitrage but there are some liquidation statistics there as well uh well that all looks very nice and fun the bit that i really like is the leaderboard so we go on the leaderboard and we look at the latest mdv transactions not that sexy numbers are not that big but if we look at the top gas spending mev transactions in the last 90 days oh it gets so fun because you might think that the bots are always winning but they're not bots get wrecked as well uh because bots are not clever so for instance if we look here i don't know how many far down this is but there's a transaction here where the mev tx sender income was 3143 and they paid over 13 000 in fees so a net loss uh further down though there's a oh there's a big one here so the m e v t extended income was 188 867 and they paid nine and a half grand in fees uh and that's not the only one there's another one here a hundred grand and paying oh wow six thousand dollars in fees um but then if we look down further there's one that paid that got 893 in extracted value but paid 5783 five dollars in fees and it will be interesting to see at the end of the year how well a bot performed and whether it came out net and profit because yeah these are not always working out and there is some big wins here but there are also some some pretty big losses as well so it's not as clear-cut as you might think and the team themselves say that the data as much as they believe that it's correct it it might not be 100 verifiable and true but of course they're working on developing it the whole time but no doubt this is a page which you will get a lot of information from they also go into how the data is accumulated how they put it together and there are some faqs where you can also find out all about flashbots and what they're up to but yeah generally this is a very helpful and very alarming dashboard to look into it's kind of fascinating the team behind mev explorer is flashbots a research and development organization which was formed to mitigate the negative externalities and existential risks posed by minor extractable value to smart contract blockchain the idea being that if we accept mev as inevitable then the best way forward is to encourage users to do it according to at least some rough rules of combat in their eyes mev represents a genuine crisis and their approach to it can be broken down into three parts one illuminate the dark forest two democratize extraction and three distribute benefits now mev explore is their first step in illuminating the dark forest giving you and me and anyone out there the opportunity to see pretty much what's going on not all of it but most of it to democratize extraction they've now developed mev geth an upgrade to the go ethereum client which enables a sealed bid block space auction mechanism for communicating transaction order preference got it and at the start of the year on january the 6th the team launched flashbots alpha a two-sided marketplace composed of miners and mev searchers in their words any miner or mining pool can run mev geth and receive additional income from mev strictly more than what it would earn from running vanilla geth without the need to enter into any bespoke deals with traders mev searchers predominantly bot operators are able to send transactions through a flash bot gateway called mev relay to improve their chance of landing on chain without the need to pay for gas on failed transactions or causing chain congestion and enter into gas war that hurts other ethereum users which all sounds wonderful as long as people are using it but almost immediately creative white hats began to find really novel ways to use the toolset one of the most interesting being account abstraction now because one of the very cool things flashbots allows for is submitting zero gas transactions that makes it possible to submit transactions which are paid for with any erc20 token and this even got vitalik talking so how would this work well you've probably hit that situation where you sent too much eth out of your wallet and then can no longer pay for the next transaction selling an erc20 token for ethon unit swap it's also out of the question because you need to pay for it you can't send it to a centralized exchange because oh yeah you need eth for it this would help you out but that's just one piece of it imagine your private keys are compromised and a hacker steals your eat but you have tokens staked in another contract for instance now the hacker's bot doesn't see those but in order to rescue them you will need eat in the wallet to power any transaction however the moment you send any to your wallet the bot will swipe it now that's where flashbots can come to the rescue and this is exactly the situation a friend of mine found himself in and he managed to code a bot to front run the hacker and recover his funds so how did he do it you had your wallet compromised and then by some kind of weird miracle of i don't know what you managed to front run the hacker so tell me what what happened there and how were you able to to do that because it sounds like science fiction to me so how i got hacked was pretty stupid i just left my private key on github some crawler found it and he had my private keys and then the first thing that i tried was getting my getting ethereum into the wallet trying to get my tokens out didn't realize that it would instantly snipe the ethereum so then i already lost like half of my token but for some reason it left also half of the tokens in so what i did then was i basically i just had ethereum in one wallet i sent a transaction to send aetherium to my hacked wallet i sent a transaction right after that from the hacked wallet with a nonce that like what's the next transaction basically so it couldn't overwrite the nonce and then i used up all the ethereum that was in my wallet so it could never overwrite my gas price meaning that my transaction was the most profitable for the miners and that's why it would be included in the block instead of an overridden transaction from the hacker so i think the big question here is what are you doing leaving your private keys on github come on yeah no i i made a liquidation box and it was a private private github repo and i was just cleaning up my github trying to make stuff public like ah some other people might be able to use that and well no one used it except the hacker to get my private key from there because i didn't realize my private key was on there do you think people underestimate just how much there is malignant activity just hunting everywhere for you to make a mistake yeah for sure i i underestimated it for sure because i didn't know that that like there are multiple crawlers out there that just crawl github and other stack overflow and other websites where there might be codes including private keys and just crawl get the private keys and then those same bots are extremely quick in getting out the ethereum so as soon as i send in ethereum the split second that it arrived it was already out what can you show me today what can you demo for me to give you know the people who watch this channel some kind of insight into this tangled forest this dark forest of the mempool and everything else that's going on there all right so i'm going to demonstrate how to get tokens from a compromise wallet to an uncompromised wallet using this github repo right here so i have this wallet right here which holds 50 chartx tokens and a bit of ethereum but not enough ethereum to make a transaction to send away those chartx tokens and a bot also has my private key so it's been compromised so if i would send in ethereum it would instantly get sniped to transfer all my chartx tokens so you lose both my chartx tokens and my ethereum so what i'm going to do instead is i'm going to send a transaction from here to transfer the chartx tokens to a different wallet then i'm going to send a second transaction from here which will pay for the gas fees for the first transaction i'm going to bundle these two transactions together and i'm going to send them to a flashback note and once it's mined it should i should safely receive my chartex tokens in here so i've cloned this github repo that takes care of all that to my local computer and i'm gonna run it from here and see what happens and as you can see it worked so the chartx tokens got transferred to this wallet which is this one our save wallet and we paid zero in gas fees and then our second wallet this one paid everything with all the gas fees also actually paid zero cash fees but just bribed the note it interacted with the flashbots smart contract and we saved the chartx tokens so you can apply this technique to basically any crypto asset that's transferable so you can save your erc20s you can save your nfts but you could also make custom code to save your tokens that are stored in a urine vault or something else when you look at that graph of mev extraction since the turn of the year it's startling just how steep it goes up at a fundamental level that graph represents technological and informational asymmetry the odds being stacked higher and higher against you and against me and normal users and when we attempt to proselytize others on the virtues of permission as trustless systems over centralized ones there's a real danger we're advocating for simply swapping one evil for another because there's no way any of this benefits the humble user and the more volume that sweeps into d5 and crypto in general the more money there will be to be made from mev but the thing is that might account for some of the rise but it doesn't account for all of it now lots of very smart people have now written about mev and when you sift through all of that material one prevailing idea recurs a fundamental belief in the good intentions of miners in the natural altruism of miners because you see while trading bots can overbid for gas in pgas they don't ultimately hold the power to reorder transactions only miners do and that is an awesome power that they can wield without anyone being able to stop them the defining feature of ethereum's current era is that most miners are not attempting to exploit mev themselves yet writes charlie noise in an article paradigm yet the future is only going to get scarier writes down in the dark forest miners today leave money on the table by not acting on these opportunities in the future they will reorder and submit transactions in their memphis for their benefit even worse they could re-org blocks mined by other miners in an attempt to steal mev which was not claimed by them resulting in chain instability the fear is that miners have already been tempted by the fruits of the mev tree and are not only dipping into pots of additional revenue that traders can't touch but also actively colluding with traders to exploit mev to their fullest advantage the reality is that miners are already implementing the mev in their latest report the flashbots teams state that there are five mining pools running mev geth accounting for over 12 percent of total ethereum hashrate which collects them 0.13 each per block of additional revenue doesn't sound like much but when you add it up when you scale it up it is so the question is what has provoked the miners to suddenly sit up and take notice to stop being so altruistic well isn't it obvious [Music] eip1559 the alteration to the ethereum blockchain that's now scheduled for inclusion in july now we've covered the proposal in previous videos but essentially it's this gas fees will be sent to the network as a kind of burn with only an optional tip paid to miners so the miners don't get it so is it any wonder that they're moving rapidly towards adopting strategies that keep mining profitable for them over and over again i've seen criticisms of miners calling them greedy and conniving but in any clinical clear-sighted analysis pieces you chew through researching this subject time and time again you hear miners called rational economic actors that is to say they do what's in their best interest financially because more than anyone else they are invested it's business so let's not be so quick to vilify them shall we now look in another article the flashbots team state we believe mev extraction is unavoidable and they see their work as an attempt to define what constitutes fair and unfair mev but the key word here still is mitigation so what can be done well it turns out that the various layer 2 solutions now picking up steam do offer interesting ways to combat mev but that's really too deep a rabbit hole to go down for today shutter protocol promises to combat mev using threshold cryptography now i read the introductory article and um it made me feel extremely stupid wingardium leviosa stop stop chain link is developing what they call fair sequencing services fss an mev solution that uses a decentralized oracle network to fairly order transactions sent to an on-chain smart contract by separating the ability to order transactions from the ability to produce blocks malicious value extracting such as front-running can be prevented through pre-defined ordering policies sounds cool huh and if you've tried to enter an ido recently you could well have been front run by a bot getting in ahead and buying all the tokens before dumping them on unsuspecting retail which means you and that looks like this all this the team at ferrum have developed what they call an inverse fair launch protection system iflps which identifies bots addresses that attempt to infiltrate a listing blacklist them and then prevents them from selling or transferring tokens and i was sent this absolute zinger from the totem fire launch where a bot attempted to buy 150 each worth of tokens and was subsequently blacklisted and lost all that money oh i'm so sorry so that's it for this week there is so much more to learn about the dark forest but hopefully it will give you pause for thought next time you interact with the blockchain there is so much more going on than you will ever know flashbot is an incredible resource i highly recommend bookmarking and adding it to your d5 playbook be safe out there this was the defiant [Music] you