Building Substreams with Spyglass Labs: A Web3 Developer Guide

[00:00:02]welcome to episode 2 of devs on the graph in this episode we get to know Alexander gusev of spyglass Labs we get to know a little bit about his background of how he got into the ethereum ecosystem build Guild also how he started building with the graph and now how he's building with substreams and Spyglass Labs at the end of this episode he's going to take all of us through a workshop where we can understand substreams and start to build with substreams so if you're a builder and wanting to know more about substreams stick to the end of this video if you're listening to the audio version of this episode be sure to switch over to the video version of this episode at the end and you can check out that link in the description so you could watch his Workshop as a final little bit of housekeeping just know that Spyglass Labs has recently went through a name change from soulbound labs to Spyglass Labs so in this episode we're discussing soulbound Labs just know that it's currently spy glass Labs so without further Ado I give you Alexander gussev good afternoon Alexander gusive also known as blind enabler Goose he's got all of the different names and whatever you want whatever we want uh hi everyone uh this is Marcus and here we are at devs on the graph episode two and like I just said we are with Bland nabler Alexander gusev and he is with soulbound labs and has been so for a few years now just building building building I want to get to know more of that story but before we get into that I want to understand you and what brought you into becoming a developer today so let's start back at the beginning when did you first start getting interested in computers and building software yeah so I think that when I think back to like when I first started getting into software development it was not actually with software development um my first taste of like engineering and building stuff was definitely when I was like a little kid so I I'm trying to think I was like probably about nine when I was playing this game on the Playstation 3 called The Little Big Planet like Little Big Planet 2 are you familiar with it a little sack dude jumping around and stuff yeah um so that game had a like the thing that was super cool about that game was not actually the game itself it was like the game had a really cool level editor and so basically it was effectively like a whole game engine inside of a game that you could use to like create and publish and share games that like other people could play and you could download people's maps and like customize it yourself um and it was that that like totally hooked me on this feeling of like creation and like I mean I was like 10 years old and homeschooled and I had a little notebook that I carried around to like sketch up ideas and never like my brain would just pop up with new ideas and like oh I can't wait to like go home and go build that but then that kind of like lied dormant for like eight years um you know I didn't really gravitate much towards towards like computer science really like I inevitably had some experience because I've been like into Linux for a really long time so I was like writing bash scripts and you know playing around with my computers in like interesting ways but I didn't really like consider that programming um but it wasn't until uh like 2020 um towards like the very start oh it was actually a little bit before I think uh quarantine hit but I started getting really into ethereum and this whole concept of smart contracts and I was like whoa this is super powerful um let me dive into that yeah what was your first impression of ethereum they want impression when you can look back and say wow that was an important day what kind day was that for you I mean honestly I think my first day hearing about ethereum I like did not like understand it it wasn't until I just started spending more time in the ecosystem and like seeing stuff that people were doing because like I don't know at first like a global computer is like a kind of cool idea but it's not it's not like immediately interesting at least it wasn't for me like when I first got into it but then I was like okay you know this is like super powerful and I guess also that's like within the context of General like cryptocurrency because like I've I've been in you know really interested in cryptocurrencies since 2013 when Silk Road got shut down it was like my first exposure with it I was like okay that's really cool we have like magic internet money we can send each other but um it took me it took me a little bit to like fully like click with ethereum so I don't really have like a day where I was like oh my God I get it but it was more like a slow like frog in a beaker type deal or I was like okay like there's like a lot of really cool ideas here that could be super super powerful like you know D5 is like a really obvious one that I was like okay this is cool like there's like hyper efficient stuff with no middlemen that are just working off like code that's really killer yeah tell me so am I like allowed to swear on this thing you can do whatever you want okay we are just for the audience we did get a chance to work together back with soul bound Labs doing devrelations back then and the first I heard of you working with the team was through scaffold eth and that was really how you got started and that is how I got introduced to you so what was that like going from scaffold Ethan using scaffolding and then also using tooling with the graph in subgraphs yeah I like learned how to do ethereum development on accident like I didn't actually go into it being like Oh yeah I'm gonna go like learn a theory of development today I was like it's like I had some ideas and I had to like piece together the ways like how to do that um but like yeah getting into scaffoldee from the build Guild was definitely like a huge uh like parabolic moment I don't know I just like my learning like accelerated really rapidly because before I was kind of like fumbling around didn't really understand how like anything worked I was like all these tools are like kind of cool but they don't like click at all um but yeah like with scaffoldeth a super forkable framework for like building ethereum smart contracts was like a perfect uh playground for me to experiment with and like learn learn through experimentation as opposed to like learns like reading which like I I don't do particularly well like I like to I much prefer to like get my hands dirty in uh this sort of thing and when it comes to this sort of stuff so um yeah basically like yeah it was through scaffolding that I was able to like learn smart contract development and like front-end stuff and you know put all these together um and then I started working with the graph a couple months later because I got hired to do a little bit of contract work which I got exposed to sub graphs with and I was like oh this is actually really cool you have like these graphql front end you know or graphql apis I was like I don't really know what graphql is but I kind of understand like I kind of make these queries which can like fetch different sorts of data um and then you know through experimentation I like learned how powerful they were I guess very cool what has kept you going right now you are hopping from uh you know fundamental ethereum concepts and now you're looking at different ways of using apis that are sub graphs and now you're building with soul bound labs and I do want to talk more about sub streams which is something that you've been building with lately what's a thread about yourself that you find is wow Alexander gusive I'm going to keep on going every single day is there something that is like a North star that you would like to see in the world or in the ethereum community or is it for yourself I'm just curious what motivates you yeah I mean like so I'm very involved with the build Guild which is like very closely tied with scaffold it's a group of developers who will use scaffoldeth um and so you know I'm very motivated and like propelled by my peers there because it's like nice mentoring new developers it's nice seeing what other people are doing and this sort of thing um but like I think mainly it's just like I'm just having fun like I'm just building things like I don't put too much pressure on myself uh to do anything like super crazy like I just kind of I'm like okay you know this is a pretty cool idea I'm gonna go play around with this I'm gonna go work on this um I tend not to put too much pressure myself in terms of like I have to go change the world today because like you know that's that's like tough that's a lot you can do one thing at a time yeah exactly so like I feel like I can perform optimally like if I'm playing so to speak like I'm a big proponent of like learning through play and doing experimentation those sort of thing so um yeah whatever kind of helps me okay but yeah before we get into sub streams I want to ask if you could share with any new developers in the ethereum environment what from your learning would you like to share with them about your journey and maybe something that might accelerate them in terms of developing with the graph developing with scaffoldeth yeah uh I think that developments um it's like a loaded question like what can I what kind of advice maybe maybe what would you tell yourself that would be a better question yeah I think that I would say like again yeah don't don't put too much pressure on yourself like learning is a very incremental process you don't wake up one day and you're like oh I did it I know how to develop smart contracts it's like no you like have a problem you need to solves like you learn about what it takes to solve that problem um and yeah like just figure out what you want to do and then find a way to make that happen and then learn from there you know it's like very easy to get caught up like learning and like gathering information because it feels very productive But ultimately I think it's a lot better to have a goal and like work backwards from that seeing like what you need to do um and you know in that same vein is it's really useful to have um you know I guess like references so I I'm a I really like this dude Paul Graham are you familiar with Paul Graham so he he's like a founder of the Y combinator startup accelerator and like he also had a startup you actually had a number back in the day but he has really good essays on his website everyone's good check it out if you haven't checked it out already um but he specifically has a pretty interesting story because he actually went to art school after getting his uh undergrad in computer science so it was he like you know went off and did his own thing at art school and he has this really good um essay called hackers and painters in which he describes hackers and Painters as seemingly two very different things but actually they have a lot of um things they could learn from each other in a lot of ways in which they overlap and um I think one of the really killer pieces he said there was like artists don't learn to do art by just doing art like some of them do but very very few do a lot of them do art by mimicking greats and like copying because you learn about the intricacies of the development process and the intricacies of like painting I think a lot of that ties over very well to um you know learning learning development because like it's really easy to kind of like reverse engineer something it's like okay I want to go you know build a clone of uniswap like okay I can work backwards from there super well and I learn about defying you know I learned about all these different things and then you know you go build your build your clone so I guess you know it's very useful to work from references and work from other people's stuff so like it's yeah find good examples of really cool things that you would want to do and then try and clone them and modify them in slightly different ways I love that and play I love it and play I love that attitude and don't put too much pressure on yourself and that's for myself and everyone else out there not too much pressure on yourself let's go into sub streams you started correctly I'm wrong you started learning rust yeah just because of substreams yeah yeah yeah tell me about that journey and tell me now that substreams are now the docs are now live on the graph.com and then also you can go to substreams uh streaming fast website and they have the docs there as well go ahead and dive in if you'd like but I want to hear about your journey where you started years ago maybe one year ago a year and a half ago yeah it was about a year ago yeah shortly after graph day gotcha shortly after graph day we got demoed I graphed it and I was like oh wow this is like that's cool that's very cool I can't wait for that to be a thing cool so where are you at with your substreams development journey and what would you like to tell the audience in a few words what could substreams provide for them oh boy substreams can provide a lot they're they are really really cool and you know I guess there's like two ways in which I could talk about substance so I can talk about it in the context of like an existing subgraph developer and like how substream solves so many of the pain points right sub graph development and substreams as is are substream powering the sub graphs yeah so they pipe into subgroups yeah I guess actually a really important first thing is like let's define what the hell even a sub graph is perfect um so or sorry a substream so many sub words um so yeah sub streams are just a data transformation layer for I guess they're a transformation layer for blockchain data again what does that mean that's like not a very good that's like some mumbo jumbo right so basically it is a way to take an input of some block data and modify it to either extract certain types of information that we care about or aggregate certain values and basically a way of finding a needle in a haystack so to speak I like this analogy where um you can kind of think about we have some data that we want to find but it's hidden in a bunch of other data you know within block block data and whatnot send data a lot there but um you know this is also very similar to kind of what sub graphs are doing but sub streams are not really concerned with how to store them and how to serve this data so they are simply concerned with we have data over here and we need to manipulate it in some way because you have to do something with it at the end so they're just worried about transforming data so you can use substreams to spit data out into a postgres database to use in a graphql endpoint and so this is what the graph does with substream or substream powered subgraphs which are super super cool um but yeah sub streams at their core they extract dude they just transform this data and manipulate it in some way to be consumed by some endpoint right and correct me I'm wrong but the substream availability of the data is coming from firehouse and you're getting that and then from there that's being piped into the subgraph and you're being able to get that data much faster in terms of the speed yeah of indexing and then also you're getting a much more granular data how has that changed your development with soulbound labs well yeah it's changed it a lot so I guess you you mentioned something that's also pretty important to talk about with a piece of how substance work um and that is fire hose so substreams are built with a streaming first or I guess they are built to be have data streamed through them so traditionally how things worked with subgraphs was we had an instance of Geth which is the ethereum node or ethereum node client sorry ethereum client written in go um and you know within Geth you have RPC calls so these are kind of like you know API calls you can make that will get certain information so an example this is like youth get balance which can take a parameter of an address and we'll return the balance of an address um and so this is very simple you make a request you get back your data but the problem with this is RPC calls are slow and they can't be like parallelized very well um fire hose is oh much faster honestly actually I guess not well and for what I understand not to interject yeah from what I understand fire hose allows for a flat file representation of the entirety of a blockchain where the blockchain itself is not having to actually be interacted with you're actually just interacting with fire hose and that allows for the parallel rapid indexing to be piped in through sub streams okay that's interesting yeah I know I appreciate that explanation yeah yeah I just kind of consider it this magic black box that allows me to get data very fast yeah no that's that's yeah I think that that does sound super correct but yeah basically um the the way that substance works are they have restrictions but these restrictions actually are not restrictions they are not restrictions like in a bad way they they kind of force you to write better and more easy to understand code in my opinion um so sub streams has a concept of modules and so the whole architecture of how your data flows has to be deterministic so what does that mean if you're you're familiar with like math or anything there's a concept of like pure functions which just gives some output based on some input and determinism is if something has the same inputs it'll always produce the same output right so if I have a function you know f of x is equal to 2x it'll always return four if I give an input of 2 because it's multiplying two times two right it's not there's no like sense of side effects there um and so with substrings all of your modules which you know start off initially ingesting the some raw block data have to filter it and do something to it but the nice part here is that we're not concerned with like ordering of blocks or how this works we kind of just shove the data through this pure function and it gets manipulated in such a way and so you know as a result we can't do the same we don't have like the same level of Freedom as far as like oh I'm gonna go grab this entity at this ID you know like we do in sub graphs but as a result our data or our code is much easier to read and much easier to reason about because if something breaks we kind of more or less know where it broke and we can also you know visually see like because actually substreams is really nice uh graphene utility that we can actually see how subtrins are like how modules are used to feed other modules in this chain and so yeah with soulbound Labs I know you guys have been working with subgraphs for a while and now with sub streams how is that providing you guys the data in a different way and how is that making your dap different yeah so because substrands are able to be so fast because it's all deterministic and it can be parallelized super well it enables us to get a lot more creative and a lot more powerful with like what kind of data that we can index so traditionally subgraphs have been pretty slow um but with substreams it's a lot faster and so you know we can now do things that we were just technologically limited to before which means we can do a lot more complex calculations within subgraphs to give people like reputation score or even just more complex stuff in general and so like you know this example isn't really specific to any of my stuff at soulbound this is like my own personal play time working with the sub streams um but I built a substream to index every single contract that was ever deployed to ethereum mainnet yeah which is like you know that's like a ludicrous thing to possibly do um but alas you know another good example is like I built a ERC 721 balance thing it's like kind of um I guess it's like an erc721 tracker but it tracks every year c721 ever deployed on mainnet um and gets like everyone's balances and stuff so you like you know can use that to fuel your nft Marketplace or whatever else and these things are just not possible with like normal subgraphs right and then that granular data coming in Faster it allows I mean from what I understand what you guys are tooling with and you have a presentation by the way if anyone's listening there's a presentation coming up on June 15th at the house of web 3 that you'll be presenting at surrounding substreams and what he's been building with subgraph powered substreams the GUI that you have at soulbound Labs what's that GUI doing and how is that presenting sub graph substream powered subgraphs yeah so hmm yeah so there's a couple things I I could talk about here so I guess simply to answer your question um this GUI is built the intent of making substreams and specifically substream development much more user-friendly um because there's no real reason for it to be you know access limited to a small group of people because like people have data that they want to access and we should make it accessible and easy to use right and so we I absolutely love that not to interrupt but I mean ease of approachability is so critical it's the same thing that got you into scaffoldee yeah exactly that can be a launching pad for so many developers out there who let's say they want to understand subgraphs substreams sub graph sub stream powered sub graphs to be able to visualize it and say that's real time that's happening and being able to actually have that that's such a powerful thing that I personally think can provide a jump off point similar to a scaffold these type of moment for you yeah totally and I think like you know it's not necessarily that these new developers are maybe incapable of writing the code to do this because you know it it really the substream module code isn't anything too crazy but there is this like Panic that sets in when you're a new developer and you're staring at a blank terminal you're like right and it's like you know it's a lot nicer to have a little visual playground than like you know have suggestions like what you can do like okay I can take this block data and I can filter it out or like oh I can put in this nft contract and I can say Okay I want to see all transfers where the from address is vitalik.e through the two addresses metallic.eth and you can kind of like visually piece these sorts of things together um it's it's really important for I think substring adoption hopefully anyway it'll get more people using this really powerful really powerful technology awesome that's awesome that's awesome so I think I've gotten to know you once again Alexander we've talked for a while now not just now but for the past years we've gotten known each other so I really just want to thank you for your time yeah plus the next steps for you next steps that you're excited for what's something that excites you what you are building and then for the coming months coming years what's something that you're just like man I'm really really excited about it doesn't have to be anything about substreams sub graphs the graph what are you tinkering with that you're just like oh my gosh this is fun right now I know you're drumming right now yeah yeah yeah that's awesome I do I do play the drums it's super fun my poor hands are just like getting demolished with all of my hobbies like I was in the rock climbing and drumming and also you know furiously programming for many hours a day uh you know I would say like substreams honestly are some of the most exciting things for me I would say within within the space right now is like a lot of cool azk stuff that's going on and like scaling stuff but you know I think that we're approaching a point which is rather I I don't think anyway that we've had which is that these decentralized technology options are becoming better we're achieving like decentralized Supremacy so to speak where the decentralized technology options are becoming better than the centralized options which is pretty crazy because traditionally it's been a sacrifice it's like okay do I care about my data correctness and do I care about this being good it's like okay I'm going to choose a decentralized option like the graph because I really care that my data is correct um you know but if you don't care so much you can go to another provider or something but with sub streams it is so fast and so composable which is something I'm also really excited about it's like because the substance modular architecture you can import people's packages and use their modules so you have this whole ecosystem where you don't have to write everything from scratch you can just use people's packages and plug them in and it all just works you know um but yeah with substreams basically data indexing is going to become so good and so easy to do and so fast that we're going to have a Cambrian explosion of indexing data on the on on chain which I think is going to be really really cool because we're just going to be having so much good data available out there to help developers build things better which is like super exciting to me so I love that I love that you are bringing up Cambrian explosion and how the actual decentralized layer of so many different types of technologies that are in this space web 3 decentralization including the graph including all of these different blockchains it's now in your opinion comparable if not better than certain options and I really am excited for that yeah no I I think it actually is just going to be better um because like you know for instance I think like Alchemy has some nice uh stuff where you can you know they have like an nft API where you can get all nfts a certain user owns and whatnot um but now we're going to have these equivalents and more be decentralized and like you know anyone can add on and make their own forks and it's great it's like we're we're getting really powerful apis built on top of raw blockchain data which is sick so love that I think it's a great place to close it so that was Alexander gusive with devs on the graph I've been Marcus and I thank you for taking the time for joining us and if you stick around uh Alexander is going to be bringing us through a little workshop that he has prepared so if you could stick right through this go ahead and uh continue the learning with Alexander and once again thank you for your time thanks hello hello my name is Alexander Gustav I'm from spygloss labs and this is now the substreams overview Workshop section of the talk um so we're gonna be going over substreams and as a result we have some prerequisites for what you guys got to know so you have to have some level of General web 3 knowledge you don't have to be Pros but you also don't have to be a noob um and you're also going to have to have some understanding of programming you don't have to be an expert developer But ultimately this is a programming talk and as a result you have to know you have to know a little bit of programming um so you guys are going to walk away from this Workshop hopefully with some pieces of information um these pieces of information are what are sub streams so by the end of this you guys should be able to answer this question pretty clearly you guys should be able to answer why substrings are so epic because they are super Epic you should also know how to build a substream kind of the process of building a substring and then you should also know that decentralized apis have in my opinion now reached the point of Supremacy over their centralized apis and there's a couple of reasons why I'll talk about this but we'll talk about this in a sec um but I think it's really important when talking about any sort of technology is technology exists to solve some problem and so let's go over kind of the problem statement for why sub streams which exist and why you should care in the first place um blockchains are open and all the data is there out in public for any of the node operators to have however digesting the raw blockchain data is hard and complicated and finding the data you care about let's say for whatever your application is can be pretty difficult and convoluted to do um and so let's kind of put this in more common developer terms so let's give an example um we have our developer Bob and Bob wants to build an app that shows all the nfts that a user owns um so how does he get this data well he has some options so he can run a full node and query the chain directly for the event logs which is fine and it's really the correct quote-unquote decentralized way to do this because he has the nodes and he doesn't have to trust anyone with his data and he knows that all of his information is correct but again this is a very complex to do and then B you still have this enable the haystack problem where it's just difficult to get the data you directly care about um you can also use a centralized provider rather than running your own node which is fine for the most part they have you know centralized providers are a legitimate risk for the ecosystem because if left unchecked they could become the next AWS where we have three companies controlling you know 95 of what the internet runs on but for now they're fine you could also use a centralized data indexing service so this is kind of like a blend of Uh custom uh you know Data Tracking as well as said charge providers and kind of examples of these are things like Dune or some of these other data analytics Services um or you can use a decentralized API but before we get to that let's talk about centralized providers so companies like inferior and Alchemy they provide traditional API endpoints that allow users to access information from the chain um so these are you know basically easy to use uh endpoints to do things that developers care about the developer experience is pretty good with these things I mean I've used them I think a lot of people use them most people probably use them um you know if these aren't sufficient they are ultimately at the end of the day providers and so you can definitely use them to just listen to event logs as well some examples of like the nice kind of rappers they have to get the data you care about or you know for instance getting all the nfts that a user owns so you send an address and it spits back some big array containing all the nfts they own or also getting token balances for the user so you don't have to process this directly you just have these ready to go endpoints um and so why would someone go with this well the reason why is because one is this is the primary reason is it's just easy so they're very easy to use because everyone or you know most developers are used to using apis to get their data and so this is very intuitive and also I guess it's cheap as well however there are also a number of cons um the first is after opaque so we don't really know how they're getting their data we don't know if they're emitting certain data or whether or not they're sending you bad data you don't know um they also can't be customized so you can't include like a white list or a blacklist or add some extra info or add some extra derived info um what you see is what you get and you can't really play with it at all which I don't like I like customizing my things to make it kind of work perfectly for me um they're also centralized and so I spoke on this a little bit and so they are centralized providers which definitely could become the next 80 best or Google cloud and I'm not going to talk on this because it's not the primary focus so this talk um and then the last piece is kind of there's no guarantee over the correctness of data because again there's no like signature they're being you know you're receiving in accordance with this uh you're just kind of trusting that they're sending you the correct data and most of the time it's not critical if they send you bad data but nonetheless still uh still an issue them um and so now let's talk about decentralized API services so there are protocols that exist such as the graph um that allow for decentralized apis to be built um and so the way the graph works is it's a network of people that are called indexers and these indexers are node operators as we can kind of see here in this diagram right you know the center person and basically a users Define custom ways to index blockchain data they deploy it to the network and any number of indexers can pick up this specific configuration and start tracking the events that we care about um and so an example of this could be you know getting all the owners of a particular nft collection and deriving some extra data in there because really with subgraphs you can do just about anything you can do really complex subgraphs and index a lot of data and then it's served to the user through a graphql endpoint um and so in my opinion this has the best developer experience once the subgraph is created because working with graphql really is incredible the queries are really simple you just describe the data you want and then it delivers it to you and it really is killer so like here's an example of how you would get the all nfts that a user owns so you would query for a particular user you would pass in their address and then you would grab let's say all their nfts you would grab their contract address as well as the token ID and this is awesome and so it'll return an array of users or it'll return that particular user with an array of older nfts they have so you know let's have the pros you get fantastic developer experience at the final end of the stage um I am going to talk about this a little bit the graphql is awesome super cheap to query the graph is the cheapest Network per query I believe um I don't think there's anything cheaper uh it's super configurable because at the end of the day it's user defined so if you want to modify your subgraph to do something or add another contract like you can totally do this the how you what it indexes is 100 up to you um it's decentralized which is always good because basically we just don't want big centralization of you know data and then also we have a very high data correctness on the graphs Network because there are penalties in place for people serving bad data so if a graph node indexer can be found to be surveying false data or bad data or malicious data they get penalized because in order to be a graph indexer and serve data you have to put up an actual stake of your cash but it's not all sunshine and rainbows let's talk about some of the cons um assembly script is terrible to work with I say not fun but I would yeah it's really not fun um the compiler errors are just really rough in that language and in general it feels like it's just not really production ready frankly um the as well as the the tech can be really hard to learn within the graph it's ultimately a pretty complex protocol and understanding how all the different parts work together is hard and requires a lot of info it's it's a lot more complex than querying an API um complex subgraphs can be a nightmare to write safely because of data mutability there is some things that make this easier to do such as the really cool I believe was by float Capital they made a unbreakable you know quote-unquote unbreakable subgraph package which you know added some nice helpers to write sub graphs in a way which was a little bit safer um but ultimately they're not they can just be pretty difficult to write like handling large complex mutable state is always difficult in any sort of programming problem but especially in subgraph land um you also can't compose it with other sub graphs this is a con compared to substreams this isn't really a con compared to API stuff but nonetheless I still think it's a you know it's one of the downsides uh and then also like traditional sub graphs can be really slow to sync because they sync linearly um and so we'll talk about parallelization and why that's all possible with substreams here in a second but now let's talk about substreams a little bit so let's talk about rewriting all of your subgraphs in Rust because that's what you should do subgraphs were not all Sunshine rainbows the developer experience was far from good but then about a year ago substance was born it was a new method for indexing blockchain data it was built for parallel processing in a streaming fashion written and rust by some super cool people at streaming fast which is a core Dev team from the graph and you can clearly tell by this picture of two astronauts sitting next to each other with their appropriate logos there right um so subtrins brought with them a number of interested things so they brought composability modularity and insane speed increases for data indexing upwards of 50 to 100x in certain cases um if we want to give an example of how insane 50 to 100 x is let's assume that your work commute takes 10 minutes to get to work it's not like a super long time if we were to 100x your commute speed that same commute that took 10 minutes would take six seconds and that is a crazy speed up and so let's dive in and see what makes sub streams so epic so first and foremost substreams are fast um actually I just gave this I forgot that it was in the slide um but we have uh like you know 10 minute commute well 100 Exit commute speed and it takes six seconds so this means that things that were bottlenecked by performance are now possible a great example is that you just want V3 sub graph which took three months to sync linearly but when we parallelized it with substreams it was able to only take 50 hours which is really cool um they're also modular so subterms work by having little module sorry wasm modules that are composed to extract your final data that you're looking for and for those who don't know Walzem is short for web assembly which is a very fast Assembly Language for the web web assembly um and specifically this is kind of like the environment that the code is written in uh and inside of substrings there are three types of modules there are sources things such as blocks and its data you can say that they aren't directly modules but I personally consider them modules so I'm just going to call them a module then you have Maps which are modules which take input or you know n input or multiple inputs and they output something so this is very similar to a pure function for any of you that are familiar with functional programming um and then stores so stores are key value stores that enable you to aggregate values from Maps um so let's look at a simple diagram so here we can see up the top that the source module is a streaming fast ethereum type V1 block that is just a long way of saying this is an ethereum block this ethereum block is being fed into the map withdrawals map module as well as the map with or map deposits map module which are then being fed into the store module of counters and so you can very clearly see how the data is Flowing from the initial block and what parts it's touching along the way um this is really cool for a number of reasons so I talked a little bit that subgraphs when they get complex can be really difficult to write because of mutable state this does not have this problem because all side effects of your code are restricted to the appropriate module so the map withdrawals module can never talk to the map deposits module because they're just two different things the only thing the map withdrawals module is going to be working on is the block data it's getting in and the only data it will be affecting is the data it's spitting back out and so everything is very contained and so the developer experience when working on complex substroms is much more enjoyable in my opinion um so substreams are also composable and because substreams are you know all modules under the hood we can import other people's modules and use them out of the box so this is really cool because this means that I can build a substream that extends another user substream without needing to rewrite it myself so in subgraphs if you wanted to do something similar you would have to rewrite or reuse all of their code directly whereas here we can use an import statement and just import their you know s package which is a special file format meant for stream and fast packages or substance packages and then we have access to all the modules directly so we can use the unit swap V3 modules in my substream if I felt like it so let's uh go over a quick overview of a simple substream now you guys are substream experts let's actually dive into some code so uh if we go over here we can see oops if I pop over here so this is what a module looks like in Rust code and so let me make this a lot bigger so we can see uh right now you guys only need to focus on this particular block of code so this line up here is we're saying this is a this function that we're you know follows this declaration is a map module and so this is a function that takes in a block which is an ethereum block and it's going to return a contracts object in this contracts object I defined in a protobuf over here so I can actually just pull this up um and we can see in here that I defined a message called contract which is very similar to an object or struct or any of these other type of data format and so in my substream we're tracking all contracts which have ever been deployed and so for me I only care about the address of the contract that we posted as well as the timestamp of when it was deployed so going back to the module and how this all works we have our contracts variable which is the result of all the blocks transactions we then are going to iterate over the transaction within the like all transactions within the block and all function calls that are within this transaction it's a little bit complex but basically you can think about this as just looking to see every type of transaction that was ever called within every block we then iterate over it until we find a call which is a call type of five and so in our case this call type of five represents a new contract being made um don't ask why it just is how it is in ethereum and then from there if we find a call we're just simply going to map the call data directly over to then form our contract type so we take the address so we grab the call to address and we paste it in here and then we just get the blockchart timestamp um and then we collect it all into an array and we you know export it all and we say okay this is all good these contracts are all of these contracts um and so that's cool this is a really simple substream and it all works beautifully and we can even output it into a substrate or sub graph here um but that's not super important to discuss right now this is really kind of supposed to be a high level talk um cool uh hopefully that doesn't make sense to everyone now we can move on to one that's a little bit more complicated so rather than our you know 30 line long substream let's look at the unit swap beat these substream which is 1300 lines long almost 1400 so we're just going to go line by line I'm just kidding we're not going to go line by line um this is a perfect time to Showcase one of the really cool things about substreams which is how polished the developer experience is um let's say you're doing some work on this uniswap substream and you're like wow this is a lot of code that we're looking through because it is a lot of code it's very complex you only care about one specific thing well it is a very cool thing that you could do called um the graph command so if we pop open this and we just go to um we have to go into the module real quick let's go to sub streams and Google swap E3 uh subscriptions three we can grind sub streams graph let me make this bigger real quick okay um so we can run the substance graph command which will read the yaml Declaration and it will generate this file for us and so we can pop this open in editor we can see very easily this is pretty complex sub graph but we can see how all the modules are connected so we can see this substream's block is feeding into this mapping pool is created which is going into this map tokens white list which is going into here and blah blah blah and let's say I care about this store tokens module if I want to find some data from the store tokens module I can now go back to my editor and I can pop this open and I can look for store tokens and there it is this is the module that I care about it's online 124 and I can modify or touch this part of the code base all fine and all of a sudden I'm able to very quickly read through a 1400 line program because I have a visualization of it all and that's really cool um and so let's um pause this year all right um and so kind of the final part that we'll talk about which is just a really cool feature that I've been playing around with a lot with you know some of the stuff we're doing it's uh spy gloss is parameterized modules so modules have inputs that like we said uh however modules can have or map modules can have inputs of either sources such as ethereum blocks they can have inputs of stores which are those aggregation modules they can have inputs of other Maps which again are just more data Transformations however there's actually a very very cool new module input which came out recently thank you streaming fast for building this and that is modules params of well I guess it's module input of params um and so what does this actually mean params are just a string which is passed into a module that sounds really not that cool but it is a string and what does that mean is we can actually do almost anything in these param strings so a really good example of what this enables us to do is have really flexible and dynamic modules at runtime within a substream so rather than having to rebuild the substream every time that I switch a contract that I'm indexing we can get creative with how we're using module params and you can see here we have this public function called map events and this takes in a param of a string and a param of a block and it returns something called a hot dog so hot dogs are kind of a joke that we wrote um I mean it's not a joke I think it's actually like reasonably clever um let me pop over to hot dogs so hot dog types are everyone knows what a hot dog is coming to the shape of a hot dog it's a cylindrical piece of meat that you eat however no one really knows like what's inside of a hot dog because it could be beef it could be pork it could be possum you don't know um and so we called this type of hot dog a hot dog because it's effectively a dynamic type inside of sub streams um and so rather than strictly defining a protobuf for all of her different types such as say transfer event um we just instead have these modules take hot dogs as inputs and outputs and so we let the users that are using these modules customize how to handle running with hot dogs at runtime within their modules as opposed to being really strict about it and so what does that really enable us to do well it enables us to do some really powerful things so this map events module takes in an input string as you can see here of type or oven address separated by two ampersands with a contracts API and what does that enable us to do so basically we can create a hash map which goes from an address to an API um by just parsing the string that we put into it and so all of a sudden this module is able to map every event that a particular contract has and the contract does not have to be defined until we go to run the module and so this means that by introducing a param of string we get super flexible modules and again we can compose these modules so you can use this map events module inside of your subgraph or substream to kick-start the process of building it um and this idea I think is very interesting and really cool and very powerful especially because of all the composition that we're talking about and so so one of the things that we've been working on a lot at spygloss is building out a big repository of reusable modules and so I have it all inside of this literate programming document but you can see that in here we have all these different modules and these are kind of um this is a very small selection but we have modules to let's say track all contracts that and see how much gas they're using we have modules to mimic the overview of etherscan we have modules to count unique users that a contract is working in we have modules to get random um nft prices based on blur we have nft price modules of all sorts of kinds and generalized event Parson modules and this is all possible because of the ability for us to pass in params because we can customize how these modules operate at runtime and so we can give a little example if I pop open uh this little run command um and we can actually go use some of this so to for instance to get an etherscan overview of some contracts I actually I don't know exactly which one this is but we'll find out if we go to run it we can see that we will get a list of all the transactions that a particular contract has interacted with so we can see an ether CN overview which has a from value a method that they called as well as a two so I believe this is tracking uh the blur contract address and this is just getting all of the times that people have interacted with it so this person Xerox ECC 953 blah blah blah they call the execute method on the blur contract at this particular time this dude did as well and then there was a four block Gap and then these guys did and people's call execute this guy called bulk execute But ultimately we're able to parse the API completely at runtime and do really cool stuff like that and if we were to say let's say we didn't care about the blur contract let's say we wanted to modify this to be the um I guess let's pass in let's see I don't want the blur address I would like the lady address let's say herb yeah board ape Yacht Club address as well let's just paste this in real quick set a bit like this and then we will rerun this and we'll see that again there's no compilation that's going on here we're just modifying how the function is being used and we can see that somebody called a safe transfer from method on the board API Club contract and this was the guy that called it um and I think that is really powerful and I hope to see a lot of people building really cool stuff with these parameterized modules um so yeah I have plenty of other things that I could show you guys but we'll have to wait for another video or something like that because at this point I think that we're gonna run out of time so thank you very much I hope you guys learned a lot um and let me know if you want any help building substance or yeah so thank you bye