The Graph Subgraph Development and Querying Guide | ETHGlobal Workshop

[00:00:00]foreign [Music] yes I'm Marcus I'm working on the graph I work for a company called Edge and node and welcome to this presentation an introduction to the graph quick start to sub graph development and querying so let's get into it first and foremost oh we got some internet issues there we go let's try that again interesting well we're going to be presenting here that is totally fine there we go and for some reason that's going okay let's keep going all right so if any of y'all want to hop into our hacker dashboard we've got a few different things on this dashboard that would be of interest to you we've got prize in information we've got the telegram group also and also this presentation as well as all the repos so we're gonna be doing a little bit of live coding you're free feel free to scan that let's get going okay so first things first a overview of the workshop we're gonna go an introduction to the graph a high level overview of what's going on uh when you are getting your data from the graph we're also going to talk about what subgraphs are all right that's really important we're going to go through that we're also going to go through deploying a subgraph and then also we're going to compare queries between two different subgraphs the subgraph that we deploy as well as a well-developed subgraph okay let's get into it first a introduction to the graph so when you go to the graph.com first thing you're going to see is this access the world's blockchain data let's go ahead and parse that let's go ahead and see what that means I want to go all the way down to the smart contract layer first things first a smart contract really is just a bit of data storage that's all it is and if you're looking at this this is a crypto punks example kind of a mock-up where it's just some data that's all it is now if you have this data what about the data in the past what about how that has interacted with other data points well what you're talking about is this type of History all right we're looking at historical data and you really have to understand in smart contract land everything can interact with everything so we have to understand really all those different types of interactions and seeing how really are we able to get that information to you in your dapps and this is really a problem really because the blockchain is Right optimized all right what I mean by that is it's easy to write to the blockchain relatively speaking we're able to write to the blockchain especially with l2s now in a rapid and affordable way now that's really creating an indexing issue where now we have to organize all of this data and when we're here now we have to read that data we have so much data we are sending our information to the blockchain and then now we need to read that who here today has interacted with uh Instagram or some type of social media app on web2 there we go all right have you posted on Instagram recently okay so that's right that was a single right and then with as many followers as you have you know hundreds thousands of followers that's the read and that's the exact same thing we say on the blockchain but there's a lot of reading going on relative to the actual writing that's occurring so in web 2.5 there's a certain amount of solutions for this where anyone here can spin up an indexer on their own they could do it that's totally fine there's other ways to index as well that's totally fine we call those centralized indexing okay and so if you have a data point that you're looking for and you want to get it and it goes to your front end awesome all right that's a really awesome thing however when that goes away your dap goes away all right and that's really really a problem the web 3.0 solution is to have decentralized reading and decentralized indexing from the blockchain and so what you have here is a single data point going to lots of nodes and all of those nodes are able to populate your front end and if anything should happen to any one of those nodes you're still intact you're still good to go and that in a nutshell is a high level overview of the graph we love this so Let's uh think about what this means because is that too too loud there we go we good yeah Okay cool so what that means is we have a read layer that is amazingly uh we take the advantages of the blockchain and it's decentralization and we apply that to the read layer because the blockchain is very much you're writing to it all the time if you're reading in a centralized way well that's a problem because now you have the Web 2.0 type of issues and then from there if you decide to decentralize also your front end you can actually have truly decentralized apps and that's a really powerful thing so to overview the graph Network which is the decentralized network of indexers the world over we have over 450 permissionless indexers working worldwide there's redundancy between them if anything should happen to one another one will pick it up it's fast cheap and reliable and it's a global open API Okay so let's get into sub graphs who here has deployed a sub graph okay got a few people awesome so let's get into subgraphs what are subgraphs why do we need sub graphs let's talk about it we are swimming in blockchain data all the time all right being able to write all of this information to the blockchain all the L2 capacity it's amazing a subgraph essentially is a way of organizing that data you're providing instructions to indexers around the world to say this is the information that I need and I need it organized and I need to be able to query that information that's essentially what a subgraph is that's a set of rules that you give to the indexers around the world and say hey I need the data prepared in such a way that will be useful and queryable okay that is a subgraph so like I said subgraphs are instructions for indexers let's break down the pipeline in a high level overview of what that might look like you might be looking at on-chain data you might be looking at off-chain data specifically ipfs that's currently supported there's also a push to support more off-chain data but right now it's just ipfs you will Define your subgraph in the subgraph Manifest also known as the subgraph yaml you want to think about that as like the rules and overview of kind of the shape and structure of your subgraph from there you're going to give that information a little bit of logic and that is in the mappings.ts file and you're going to say hey we're pulling this information in do we want to put any logic on it do we want to actually start to transform it that's in the mappings.ts and then from there we've got this schema.graphql where you can actually have your data in nice easy buckets also known as entities and you can send graphql queries to those buckets and get your data on your front end it'll look something like this all right instead of going through etherscan instead of parsing through you could send a simple query in graphql that's very easily read and from there get your data through an API endpoint Okay so that's a high level overview of the graph we are seeing around one to two billion queries per day there's over 1 000 subgraphs published on the graph network with the indexers all over the world and there's 450 plus nearly 500 at this time okay so if you'd like to scan this feel free I love QR codes they're easy way to communicate uh where you have learned about graph about the graph protocol and sub graphs uh and once I see the phones now we will continue going there's a phone there awesome Osmos okay got it okay okay finally part three let's go ahead and deploy a subgrass so if you would like here is the repo that you can star and reference you don't need to actually clone this this is just for reference I'm kind of just getting these overviews of what I'm talking about today um and there's also going to be another repo that we'll look at in just a little bit so once I see phones down I'll keep on going let's do cool all right so first things first you're going to want to go to the graph.com if you want to participate with this you're more than welcome to I'll take my time is anyone going to be live coding with me you're welcome to yes move okay kind of sort of okay yes awesome saw some hands also you're going to want to pull up miniscan https colon forward slash forward slash mini scan the XYZ those are really the only two things you'll need and then also of course etherscan that would be helpful too okay those are the three tabs you'll need and what we're going to do in this deployment is we're going to deploy the subgraph and then we're going to trace the punk transfer event because subgraphs are optimized to look at events we're going to look at this Punk transfer event and we're going to see it on chain we're then going to see it in the subgraph Manifest we're also going to see it in the logic and then we're going to actually be able to query that in the schemadow graphql that is the pipeline we're going to follow from beginning to end okay let's go back and we'll start so like I said we'll need some tabs open first things first we'll go to the graph.com and studio so you make it a little bigger very nice and we're going to create a sub graph and we'll say live at yeast Global Water Loop and we're going to be looking at crypto box all right we're gonna be looking at the crypto punks smart contract so what we're going to do makes it a little smaller there we go it'll be on ethereum and we're going to create a sub graph now this is the subgraph dashboard we have yet to deploy our subgraph it is not published onto the network it is just something we are working on when we do get to the point where we want to query it this is our development query a URL where you could actually query that in the front end just getting you guys familiar with this you're free to fill out description information all of this is all great just providing information I'll tell you why this formation is important in a minute let's keep on going forward first things first you're going to want to install the graph CLI all right so open up your terminal copy there and paste internet yes now we'll see you're right as that's spinning up right there there we go yay awesome so we've got the graph protocol graph CLI installed from there we want to initialize our subgraph we'll copy this and paste it right into the terminal a lot of copying and pasting that's all it is so here you're presented with a selection of protocols we had neuron earlier so awesome we index on near that's also there for you we have Cosmos ethereum and rweave now we know our smart contract is on ethereum but any evm compatible chain that we also support is through ethereum we Index right now I believe 40 chains 40 41 chains um on and they're some are evm compatible some are not so just go ahead and dive in since we know this is ethereum we'll click on that subgraph Slug it automatically populates from the CLI and directory automatically populates and here we're given all the chains that are evm compatible we know it's on mainnet so there it is and then contract address this is where we can start doing a little research so let's go to etherscan and we'll type in crypto punks there it is and we're going to get the smart contract address not too small I thought might be too small that's fine we'll be here just for a second and then we're going to go to miniscan.xyz and here I love mini skin it's incredible because you're able to select a smart contract address and find relevant subgraph information really fast all of this information you may need for your subgraph some you definitely do for depending on the smart contract depending on the Chain there's different interactions that you will have depending on what is supported on a certain chain but a lot of that information is just right here do you want the ABI do you want the code start block all of that is here do you want the name it's all here why would you need this let's go back to the terminal and find out we're going to copy the terminal excuse me the smart contract paste it in it's automatically getting the ABI and this is okay sometimes it fails just due to internet issues just to the chain this literally was working all day today and now it's not no big deal because what I did was I have the ABI here so I'll pop this open very nice there we go control a control C and I'm going to pull up a new window then we're going to folder let's do 12 repos whatever the demo that's where we are at and we're gonna open that up and we are going to create a new file and we're going out with abi.json this is what you do if your ABI is not able to be gathered and now we have it right there right click and then you have the path so that's what the CLI is asking for here and that's oh nope so close so close copy path there we go there it is okay now a lot of times the start block is also populated but just in case it doesn't populate depending on the situation at the moment you have it right there copy and paste into terminal okay now the contract name subgraphs can index lots of smart contracts all on the same chain all at once so it's important to name your smart contract by its actual name you don't want to just have the demo name of contract because if you have two three ten smart contracts not helpful so I'll go here hey there it is and copy and from there paste a lot of copying and pasting very convenient from the scanner right there from miniscam now this is the I think impressive thing the most impressive thing is when you actually see this occurring this is the index contracts events as entities any events being emitted remember that little blocked pathway that I showed you where it had subgraph.yaml and it went all the way to the schema.graphql this is where that connection is occurring all right when you just press true right there you're having this scaffold out and it's connecting all the things in the back end automatically for you to look at all the events in historical data that you can query using graphql so that's all doing now right now while this is occurring um let's see come on any questions so far while this is spinning up question yes so as I'm sorry a single subgraph so that's not there right now we have a single subgraph for a single chain you could index as many different smart contracts we want on that same chain if you have a different subgraph that'll be for a different chain and you can separate it like that you could pipe all those different apis into your front end and work with it like that but good question the initiation early this issue that the second so wherever you want to start gathering information you can do that if you want your sub graph yep correct yeah the initial deployment of exactly right where that uh a smart contract was initially deployed it automatically finds it for you and um there it's perfect so now we could add another smart contract we're not going to do that right now so we'll say no and you are welcome to run through these steps here I like to go back to subgraph Studio at this point and here it's just more copying and pasting super quick super easy you're going to authenticate first things first very nice and you're going to CD just copy paste copy paste and then graph codegin and graph build graphcogen is type safety making sure all those connections that you've built up let's say you make any modifications to your code it's important to keep those connections uh fresh and then graph build compiles everything and you can read a lot about that in the documentation totally fine to do that we're just going to go ahead and graph Cogen and graph build what has been scaffolded on the back end we're not doing any changes so that's fine and then we're going to deploy now just so you know when you deploy here you do want to have incremental versioning and also when it is deployed you should see the front end of the dashboard also respond nicely there it is and it responded very nicely so the build is completed it has been deployed we have our development endpoint that is active right now it is currently syncing at 73 so it's Gathering more information takes a little bit of time just so you know when you deploy a subgraph for the first time it takes about five minutes before it's queryable because there's a certain amount of caching going on in the back end at five minutes it is going to be active so just be a little bit patient right now though let's go ahead and head back to the presentation as this is getting ready and let's keep on going and the presentation as that's going so where are we right now okay as that's spinning up let's go ahead and do a reference for subgraph patterns you may see these sometimes you may not see these sometimes but these are relatively common and then we'll see if these exist in the actual boilerplate subgraph that we spun up with our graph CLI and then we're also going to compare that to a very well developed sub graph as well and you'll be able to compare and contrast something that's relatively simple that is a boilerplate one to something a little bit more advanced so if you are indexing a crypto Punk if you're trying to gather that you're going to pipe that into an entity and an entity is in the front end really what you're going to be querying so what buckets do you want to create generally speaking you're going to create a contract entity an account entity nft entity trait entity metadata and event interface these are like different buckets that your subgraph will create and pipe the data into and let's go ahead and go back to the actual code and see if that's the case so we'll head back to visual studio code and there it is nope okay so let's look at these buckets right now everyone let's say hey schema.graphql hey these were automatically created for us we have an entity that is an assigned entity a transfer entity a punct transfer these are automatically created and these reflect the events that have been emitted from the actual smart contract just for the sake of this demo and just for the sake of being focused I'm going to focus on the punk let's make it big so everyone can see the punk a transfer entity let's just focus on that and we're going to trace that all the way into our actual root level of the smart contract so we have this smart uh we have this entity right here this Punk transfer how did this get here how did this Punk transfer entity appear here well remember if you take a one step back from the different blocks that we were just at that went from the schema to the mappings we got to go to the logic and that's going to be in the SRC folder so let's look and see how this Punk transfer event is actually getting populated there it is okay we have export function handle Punk transfer so when a punk transfer occurs this handle Punk transfer Handler triggers and there's the event right there you can see everything light up right there very nicely thank you vs code and an entity begins to be populated and we see this event right here this event params from this is an event right here that is being emitted and the from from the chain is going to The Entity also named from very nice go to vent params two and then that's also going to the two event params Punk index that's also under the punk index also we have event Block events also carry block information that's carrying the number it's all just being piped right into this entity and it's being saved all right that's all that's happening is simply coming from the event and being piped into our entity right there the entity is being saved I'm gonna slow down I've got a few minutes any questions so far am I going too fast are we good we're good question but they didn't understand yes yep yep there's just a way of looking at the entity and saying hey this is how it's organized I'm going to send a query to those entities yes great question we'll get to that in just a second so here we are we have our Handler that is we're going from the front which is the schema now we're going one level deeper to the handle Punk transfer how was this actually defined well that was in the Manifest let's go back to the root and we'll go to the subgraph.yaml which is the Manifest and this is really how your subgraph is defined it's really important information we have everything we put in the CLI the address the crypto bunks Mark and the star block important information I'm not going to go over mapping it's a little bit right there but let's look specifically for the event handlers right there and then there it is the handle Punk transfer and it's looking at the event from the smart contract indexed uint and in uh we have a uint right there we have a few different uh informations right there the address um that's really all the way to the front of this diagram that I had a second ago let's go back here and here it is so we just went you can look at this from back to front or front to back and you'll need to do that regularly when you're developing on subgraphs because you'll be interacting back and forth so let's keep on going and let's look to see are there any immutable events we're running low on time where I keep on going this is something that you'll see regularly immutable events are important for just keeping logs of straight up blockchain information if it's in blockchain information you should keep it immutable because you're simply just keeping a log that's just something you should look into aggregation entities entities that have a lot of information from various sources on the blockchain you want to have on a single entity like a contract might have information that is being aggregated definitely have that in there and linking entities to have a little bit more power on the front end you could link entities using one to one or even one to many or many many relationships and also reverse lookups you also want get or create helpers because assembly script needs null checks just check on those and there's a few other tips and tricks so document your schema well let's go into that really quickly as well so we could see how important that is when you document your schema well that allows people to really understand what they're querying and let's go ahead and go into a sub graph that is now published on the graph Network that is well designed because remember the graph network is not just indexers the graph network has subgraphs those subgraphs are also part of a Marketplace let's go find a subgraph that is also indexing the crypto punks smart contract in a more built out way so we're going to go to the graph.com and explore and you can see all of the subgraphs that are currently active we just spun up a subgraph and we haven't published it yet but now these are all published and what we want is a crypto punks sub graph that is currently gathering information there we go so we can see a few different crypto Punk subgraphs which one do you want to choose generally speaking the one with the most signal generally speaking that is being incentivized by the marketplace itself that is the GRT utility token that is incentivizing this right here so we're going to choose that one and you can see it's gone through a few different few different versions We can go into all of this but what we really want to do is go to the playground and also from here we want to go let me go back we want to go into where is it there it is the actual GitHub repo all right so we have the GitHub repo we're seeing how this is being built and we can also query it let's go ahead and take a look at the GitHub repo and see here the schema.graphql which is really what you're querying you're sending your queries to this these are the buckets remember that okay so remember we had the punk transfer entity before we were tracing that well this developer here he had the kindness to actually put in the actual things you're querying so if you're looking at a let's not look at interfaces let's look at an entity here we have an aggregation interface right here where we have an account and there's lots of different properties on this account it's not add that there and he's done such a nice job with the mappings and also with Gathering blockchain data and organize it so well that when you type in a query a graphql query to the account entity you know exactly what you're going to be querying do you want to know Punk transfers by account and you put in the account number you can get that right there it's very nicely well documented from this which is of course you know we don't have documentation on our actual API Json we don't have it on our schema.graphql because this was just boilerplate but when you're building out have some nice documentation tell developers what you are actually doing and that helps them from there let's go ahead and go to the next part of this presentation where there we go we can compare queries with two different subgraphs we have our boilerplate subgraph that we spun up and we have a more advanced subgraph that we spun up and then from there well what type of questions can we ask to this subgraph what can we ask some questions for this base of sub graph sure we could or could we ask some more advanced questions and I if you guys want we have this presentation right here with all of these questions we could practice and see okay well if I could look at the schema and trace it all the way back to the blockchain that would be helpful on your subgraph or if you would like to use a more advanced subgraph some of these questions can be answered by the more advanced subgraph let's just go ahead and do the first one just to do that and then you guys on your own are free to go through this on your own tell me about crypto Punk number ones transfer history so what we're going to do is go to subgraph studio so we have this we're setting up right now and we'll go here on the right and this is our little Explorer right here we can send some test queries we're not of course going to the API endpoint and remember that punk transfers event that we're looking through from the very beginning let's go ahead and look at that we have the pump transfers event and what we want to do is we want to go ahead and let's go ahead and look at Punk transfer ID blah blah we want the children transfers that there we go index and then first ten where the punk index is equal to one okay and we want to see the punk index which is the first Punk zero zero zero zero one right there the first ten right there and we want the punk index we want the ID we want two we want from let's see where it came from the time stamp and why not let's get the transaction hash two demo Gods there we go oh dang it let's go ahead we've been having that happen there we go so I pulled this up earlier we're having internet connection issues sometimes and here it is we have this Punk transfers Punk index number one where I have the ID and then two and from and the block time stamp all the way through all the way from the chain and then you're able to organize that at the front end there we go okay that's the end of the presentation everybody