Stanford Seminar: IPFS and the Future of the Permanent Web

[00:00:11]thank you very much uh so first of all thank you for having me here this is a huge honor because a few years ago I was usually sitting on that side of the room um and I used to come to e380 to watch basically like the heroes of computing come over and like talk and like I don't belong here I really don't uh I think maybe in like a few decades like four or five perhaps I might earn like a SL a small mention but really thank you for having me here I'll at least use the uh the venue to talk about uh the feature of the web because it is really important to get this right um we've come to depend on the web more than almost any other uh technology at least when it comes to International Communications and infrastructure um and it could be uh it could be in danger uh at least uh We've encount we're encountering some problems uh that we should fix all right so this the way I'm going to structure this talk is that I will be giving a a pretty comprehensive overview of the problems that brought about the um or the problems that ipfs tries to solve uh then I'll discuss the protocol in detail I'll actually talk about ipfs in general uh and then I'll talk about the project uh from a meta perspective of how you go about building open source uh protocols to try and Patch the living system that we have that is the internet and uh I I'll also uh at the end kind of uh so I have this presentation with a whole bunch of slides but at the end I'll I'll go through a set of principles uh in terms of like Lessons Learned as you go about Building Systems and uh give give way to like a a discussion around how protocol development should perhaps happen um what are some critical pieces of uh some critical principles when you go about building protocols to make sure that the thing gets adopted so switch and cost is one of them uh there's a bunch of others that we've uh had to struggle with and had to Think Through carefully uh to make sure that what we make is actually used instead of you know become part of the pile of cool amazing technology that goes absolutely nowhere um so yeah uh that's kind of the whole thing and at the end I kind of will open up for questions and can really dive into whatever people uh want to talk about all right so ipfs this talk is is called you know the distributed permanent web or how I learned to stop worrying and love the Merkel web uh because it it really like when you think about Merl links and I'll describe how those work they kind of solve almost everything uh not exactly but but it's a really important property that I just wish uh had been uh put into the web from the beginning and uh funnily enough uh I think the timing was such that they could have been put from into the web from the beginning they just weren't and uh if only um that had happened the the world today would be pretty different so ipfs uh we like to calling it The distributed web or the permanent web or the Merkel web in honor of Ralph Merkel who came up with merkl links and it's really a protocol to upgrade how the web works and the goal is to make something that doesn't change the interface meaning that if if you do anything that will cause people to have to mutate or to change how they think about their application system it's never going to be deployed uh because people just won't uh switch to it as we were discussing so we're matching the interface that people already expect uh there will be some things you can do uh with some more advanced features but the goal is to get uh web developers and web application creators to just be able to layer whatever it is they're building today on top of ipfs and just if nobody knows that there's been a switch then we've won because the internet should be upgraded that way uh where the end user should never have to think about uh programs changing or having to install anything else it should just get better uh and ipfs stands for the interplanetary file system and uh this is a specific choice uh as an homage to jcr lick lier who when he created the arpanet or he had the ideas for the arpanet what eventually turned into the arpet um called it The Intergalactic Network and that just sort of sets the uh the ambition of the original internet project right like we're talking about a network that should operate at an Intergalactic scale and that was the idea let's take all of these small pieces um all of these networks that are use case specific um and bind them together with this overlay network uh that we're going to call the The Intergalactic Network and now everything is going to be able to talk to everything else and that's pretty much been true I mean today all the devices are connecting through the internet and are able to talk to each other from the these very simple design principles now what does it mean to think about file distribution in that kind of scale uh one of the first things that you interplanetary sort of brings up is we're now moving data across planets uh there you run into serious issues when you think about latency uh and it turns out that that's the same kind of problem that emerges when you think about moving data from the data center uh to users in say like developing countries or um really slow networks and you end up with different constant factors of course but the problem is the same when in the data center you're dealing with things at the microsc or millisecond scale uh when you move out to the edges of the network you're dealing with seconds and minutes and that is that kind of a difference is the same as communicating over short you know distances uh here on Earth versus communicating over distances to something like Mars so the claim is that we should be building uh the internet and the file distribution system for the internet I.E the web in such a way that it could should work uh when we consider cases like interplanetary travel um right so that's that's a so the name ipfs uh tries to encapsulate all of this is on a m ligher and puts it into people's minds that we really mean business when we say this thing should work across these massively long distances uh cool so all right so this is uh uh Paul Byron's uh first categorization of networks and I like using this image because it shows how um the the differences in in a the structure of a network can have vast implications in the in the uses of the protocols right so when we think about centralized architectures it's pretty easy to change you just go and change one thing and you can upgrade things very quickly uh but it centralizes power in one location and so the end users may not be able to have the same kind of capabilities when we talk about decentralized uh networks these uh We've now kind of sharded the responsibilities of the center node uh and we could potentially have a network that's more resilient uh that perhaps could deal with some failures but still uh it you don't have full um the same level of resiliency that you could get if everything spoke the same protocol uh and so in the distributed case when everything is peer-to-peer when everything can talk to everything else when every single node is running the exact same code uh or at least is able to speak the same protocol now you're talking about an extremely resilient fabric that you can cut in any any kind of shape uh and the thing should still work uh for the most part the web started in the distributed World um you know the the original idea was that you would have your own HTTP server and your own HTP client and you would both serve files and browse the web but today it's not really the case it's kind of centralized completely um you have browsers that talk to servers and you maybe create content that way uh but all your interactions or most interactions are mediated through this uh Central Central Point right and this is kind of distributed across or rather Federated across different organizations or different web websites but for the most part when you deal with one kind of system you're completely centralized that's an issue uh so why does this matter right like why should we try and make the web more distributed at the end of the day the internet today is kind of like this nervous system that we have right so Humanity uh invented this technology and we've now become so dependent on it and use it for everything to the point where it really is the nervous system that we're evolving together and the web um I I'll go into that in a second in the internet the amazing thing is that it's just a collection of protocols right it's just a whole bunch of really good ideas implemented and deployed uh that made the whole thing work in scale and it's actually a remarkable feat of engineering and design from the GetGo to sort of construct an architecture that would scale to the use today and I I always uh I actually love the fact that we're still running ipv4 of course I would love to switch IPv6 but I actually love the fact that ipv4 has has gotten us here and is still in use today uh that's pretty amazing you can't say the same thing about most other protocols and again like the the great idea was create this this thin waste uh to try and and allow the lower layers of the network to evolve and also allow the upper layers of the network to evolve separately and only have this very small protocol in between that will mediate how the whole network will grow kind of Basics but the cool thing about the internet is that you can just you can change it and you can make it better by coming up with good ideas you take these ideas and if they're good enough you then write specs you turn those specs into code you deploy them into computers and you enhance humans right and this is very real think about your daily life think about how much you use applications that uh augment your abilities through software that's connected to other software elsewhere and think about how many of those applications were built by people that you know didn't have the capabilities to just build this massive infrastructure they just wrote some code and deployed it to you and now you have it and now you have a superpower uh this is a a remarkable uh kind of Technology right like this it didn't necessarily have to be this way it could have been completely mediated by powers that be but for whatever reason or rather by clever design and Engineering we today have this internet that is upgradable in a fantastic way right where any kind of person you know like the the quintessential college kids in their dorm room uh can come up with the best idea or like some great idea deploy into the network and suddenly build um something massive and create tons of value for the world and then the cycle kind of looks like this right you have some research you then uh develop it into code you deploy it and then people use it and it's it's this um it it's in this in this uh sequence that sometimes problems happen uh many people uh so there's a ton of research that happens and when you think about uh what Academia knows we're basically like 15 or 20 years ahead of what's deployed on in Ed today and that to me is pretty sad because what it suggests is that it's really kind of like a like a lazy deplo development and lazy development um process in that we we know what's right to build just nobody has done it and in a big way ipfs is really just an integration of old ideas it's taking good ideas that have been known about for a long time and trying to upgrade the current Internet by developing them uh making sure that the interfaces make sense deploying them to the whole network and making them easy to use for developers programmers and end users uh so why does a web matter so we know about the internet it's this amazing um piece of technology but what what is the web really how is it different from the internet the web is this application platform that allows you to write these pieces of software and put them on this amazing uh interconnected nervous system and it's what actually gives you the capabilities it's these applications that give you the superpowers it's not necessarily a communication sure you could potentially write some bits but the end user is not going to you know sit and write a protocol and try to communicate with somebody else the end user is just going to use some application deployed through uh something like the web and just think about in your day-to-day life how much of what you do across all verticals of your life from learning to communicating with your colleagues to communicating with your loved ones to maintaining your personal relationships how much of that work is now done through internet protocols in general and specifically the web so the the point is that the properties of the web have vast implications in your capabilities as a human being which means that if there's some problems with the web we better fix them uh because it might mean trouble for people and lo and behold there are some problems with the web as I discussed earlier uh it's kind of centralized now and that's you know a source of problems but it all kind of boils down to some design decisions that mostly have to do with location addressing and this made a lot of sense at the time location addressing is just the idea of saying when you have a some resource placed on the web you address it with the IP address of the host that contains the resource this was great at the time it made Act made the web actually work finally uh there have been many hypertext systems in the past for decades before and it was actually this idea that made it scale very quickly uh so this was a great idea at the time but location addressing has a problem which is that in a network where the same file might exist in many places only a specific host or you know virtual host it could be multiple computers pretending to be one uh only one specific host can actually serve you the file uh that's because that is the authority of the content uh if you tried to get it from somewhere else you couldn't know that it's the same thing uh so you specifically have to go and talk to that host to retrieve the data uh to sort of illustrate this imagine that we're kind of in this room uh and I were to share a picture to Facebook uh a web application and I sent you a link and now all of you would have to go and talk to Facebook to pull down the image and so that is when you kind of look at it in the network uh I upload some image over to Facebook I hand you a link now 30 people go and make a request and pull down the image which is slow and a huge waste of bandwidth perhaps the web could be structuring a different way to make use of the fact that perhaps the image is sitting right here in this computer and you could get it and images are not that big of a deal we have a lot of bandwidth these days but think about video if uh if there's a a 200 megabyte video which is actually pretty common these days we could be wasting you know something like to the tune of 48 gigabytes just by sharing one video across these links uh we took a a uh a number of views from from Gangam Style which is I think the most viewed video uh yeah are you including overhead fits packets with that or no no no this is just raw data right like of course there's all sorts of other stuff that go into yeah but arguably you could uh squeeze out like these are kind of constant factors I get add it it's more about like the order um to deal with here this is just kind of illustrating how much bandwidth we're really losing uh something like Gangam Style has been viewed to the tune of like almost or over two billion times and when you just when you count just the data coming out of Google servers let alone all the links we're dealing with something close to 500 pedabytes of data that's a lot for a video right I mean this is clearly an issue there's no reason we should be moving around all of this data constantly through the network and I wonder how many of those people actually sell a same video multiple times in the same day through different tabs that might have caused the the content to move again caching is not perfect HTTP caching was sort of invented as a way to deal with this problem but in reality it doesn't work most of the time and the security model kind of precludes caching most of the time right so you can't actually use in network in between caching because uh if you want privacy and Security in the internet you have to armor the wire which means you're back to the same problem again where you have to move the data all the way uh problem so another issue around bandwidth is that it's not increasing very fast so this is a graph showing the average connection speed uh of the G7 countries and this is from 2007 to 2012 and the average connection speed hasn't increased as much as other things like processing power or or storage uh in fact the prices when you when you graph the prices over uh decreasing over time bandwidth is decreasing in price slower than storage what this means is that in a sense we get this impression that the capacity uh of the the storage capacity is outpacing the speed at which we can move around data in the network giving us the impression that the network is getting slower so bandwidth could be getting better our speeds could be getting better but because our discs are increasing in size faster and we're using media that's ever and ever larger we get the feeling that the network is actually getting slower this gets gets worse when you think about developing countries and the number of people that are coming online all the time and you're now in like these Ed networks at the edge of the of the of the internet that have really high latencies slow B uh small bandwidth and then you have websites that try to give them you know tens or or or dozens of megabytes uh just to load a web page and they're completely locked out of the internet and so these amazing capabilities that we were discussing about be discussing before all the software that you can deploy suddenly can't reach the user in the network um and that's a big issue when you think about the people out there that need the Internet most the people that we kind of discuss and say let's go and and create all this great software to deploy it and kind of equalize the the the disparity of wealth across the world and ends up being locked out for by stupid issues like latency and bandwidth like that should not be that should not prevent uh people from maiting the network and uh recently uh there was so there's this huge uh immigrant crisis right now in the the huge Refugee crisis in in Europe right now there's all these camps being set up and there's reports that people have food they have clothes and so on but what they don't have is bandwidth uh they can't actually talk to to their loved ones they can't find each other because they don't have the capability to be able to communicate with each other and so there's this kind of like insane situation where something so simple uh is blocking people uh latency of course is like the the bane of computer networks which is that you can't go faster than the speed of light problems and so the solution is try and spread the content everywhere so here's kind of like the Amazon data centers and the Google data centers spread all over the world to try and get you content faster but that's only those companies what about everybody else everybody who everybody else in the network should be able to have capabilities like this but the designed structure of the internet precludes it and sure you could hire them to to serve it but it is possible that you may end up not being able to use these these systems uh it turns out by the way that this picture kind of elucidates all the problems here uh there's another set of use cases that the web is mostly um mostly has ignored so far which is what happens when you deal with disconnected or offline operation and so again in this case if I send you a Google doc and we start all collaborating in the same same thing and it's amazing we're sharing all this data um it's kind of silly that we have to move the updates through the backb to some server out there and shipping them back here when we now have really sophisticated algorithms that can do um you know smart Conflict Free resolution that allows us to collaborate in real time and yet we're still moving all the updates to the backbone right this is very silly and it gets worse when you think about the network falling apart so if our Uplink were to fall uh to go offline we lose the capability of working together completely let alone um you know we should be able to continue working because we're in the same room room we are in the same portion of the network we should continue having this capability and yet we don't and uh there's actually a lot of important uh sort of infrastructure tools uh that are out there or infrastructure applications that basically fall apart in this kind kind of scenario or you have to work really hard to make sure that you as a user are protected against this this problem there are kind of caches of Wikipedia that you could download but for the most part most link will fall apart this is something that the web can and should fix uh and we I actually think this is completely unacceptable right like uh I want to live in a world where the applications that I use will continue working um whether or not I'm in the best place in the network um I recently took a trip through Europe and I was surprised to find that in the trains between cities the latency got so bad on mobile that I just got locked out of many websites just the the latency was so high that the roundtrip times for the the crypto handshake would just time out and suddenly it couldn't do anything or like turn off https right and this is really really silly uh so so I sort of see it is this like you have this this Mothership uh that is kind of controlling everything and like you you lose access to the Mothership and everything falls apart and we need to as engineers and as designers of applications and as builders get out of this huge problem um and uh the thing is like it you don't have to even think about full uh disconnection sometimes low bandwidth blocks you out interference between wireless networks congestion of use travel outages you know that is a picture of a data center uh on fire or like after a fire so things do happen and the web should be able to uh have an infrastructure that uh makes it easy for the average web developer to build tools um to build applications in a way that will be resilient uh basically today only the major corporations are able to build resiliency against this then there's also other kinds of problems like human problems around you know surprise suppression uh Egypt kind of woke up one morning to the fact that the government had shut off access to the internet and now how could they organize they had no way of contacting each other all of their communication infrastructure was gone uh then there's of course there's also censorship and and so on and so this this kind of stuff really really matters and when you think about the messaging tools there's really no reason that these shouldn't work in disconnected networks uh we already know from a technical standpoint everything we need to do to make these really critical applications work in the disconnected or offline case uh it gets so those are human problems what about natural disasters right this is a big deal all right so if you don't care about you know the eventualities maybe you I can convince you through pointing out that the web is getting kicked out of ubiquitous Computing um so we have like all of these devices uh that we're now using from you know tablets and phones and pH uh watches and soon enough we'll have like earrings and bracelets and so on that are all connected to the internet and none of them basically except the regular desktops or laptops uh really use the web most applications that we see in Mobile are now shifting entirely to to using the web only as a sync system and running entire application platforms that are locked down and closed and this is basically taking that amazingly malleable system where somebody could just build some code and continue to update it very easily into a lock down platform that only a few people decide whether or not something will be installed uh so this is a huge step back uh from from where we were right like the web gave was this amazingly beautiful platform where you just ship some HTML and some JavaScript and CSS and now you have a wonderful new capability that you've granted to the whole world and it's getting shut out uh and this is because the you know mobile browsers are kind of slow how would you do a web browser on the on the watch uh and so on but there's actually a deeper problem here which is that the web model doesn't work in the disconnected case like how do you deal with a website that you can no longer talk to and so they suggest that the real problem is what we're layering the web on we shouldn't be doing the web over this kind of just simple file distribution system that works over TCP and you have to work really hard to put over anything else we should be putting the web over a distribution system that can deal with the distributed case that is offline first and uh this is are kind of like stats showing the usage of mobile apps versus uh the web and so on so this is a very real real thing there's of course data control we have all of these huge applications that are gathering all this data uh and it's out there and theoretically is our data sort of but in reality they control it and you can't actually link the data to each other you only link to accessing the data through their application so if you make a post on Facebook that is some important uh piece of information that you're declaring to the world or Twitter or whatever the link that you can give out to people is specifically a link that accesses data through their website so if they choose to censor it or they go away or you close your account suddenly all of that data is gone you can't link to it this breaks the whole idea of the Web the whole point was to be able to build small pieces of information that you can Interlink to each other not to like rely on Gatekeepers all the way and applications that design how we access things all right security is another problem which is of course uh how do you deal with the fact that most of the traffic is in the clear of course we now have TLS and so on but still people break into these systems all the time and uh you know steal data and so on there's of also this like tragic situation where most web developers don't understand security and so and the tools are not really there to help them design secure systems so people make all sorts of mistakes all over the place and are data gets uh leaked stolen and so on and this can be a really big problem when you think about all of the important pieces of information that you are storing yourself and you know as an individual organization Corporation whatever government uh think about all the information you're storing on the web and how easily it could be stolen corrupted changed under your feed you you might discover someday that all of your communication are slightly different somebody could actually sneak into your email servers and do this um and and this is kind of the fundamental problem uh this is an image uh from the uh Snowden leaks uh that shows how the model of the web is such that we armor the wire and this is um a really good description from Van Jacobs and actually who's working on a project similar to ipfs uh and which is really really I I'll describe it more later but uh he talks about how in the web today we are armoring the wire only and not really armoring the data and so what this means is that we're obsessed constantly with protecting the communication between your browser and the server but but anybody could sneak into either your browser or the or the server and change everything or steal it or whatever and in reality we should be protecting and armoring the data itself uh so we need authenticated and encrypted data at rest all right um the last the last problem hopefully I haven't like filled you with fear yet um because this one's a big one so permanence uh throughout history we've kind of seen these societies that burn books as the worst right we've sort of seen like the the whole idea of destroying knowledge which is kind of what makes us human um the whole idea of destroying knowledge is just abhor to what we think of civilization um and yet uh on the web today we burn books all the time it just doesn't happen in Mass right like we break links all over the place somebody could just be redesigning a website break some link and now the whole thing is gone and every every application that depended on that link is now broken so though of course you could be searching for the content again uh all of the software that you've built isn't doing those searches so they're now broken uh so the idealized web of documents is really a web of documents on computers specifically on specific hosts in the network and if you go and take those out the whole thing Falls like the the all the links break uh thankfully of course the internet archive has been trying to uh deal with this problem by by ingesting the as much of the web as they can possibly get and backing it up right and this is a really critical component but this suggests that perhaps we should be rethinking how the web is structured so that this kind of work is easier and that the web itself doesn't um disappear uh on its own or accidentally uh then surf has also been talking about this which is yeah K the person in your illustration there is speaking here in two weeks weeks so so I think he's actually gonna talk about a whole bunch of the same kinds of problems so you'll get uh to see this from different perspectives uh yeah yeah so that's true sir uh if you don't know about the archive and bruer work please look it up it's hugely vital and important to how the the network is today funny story actually one of the systems that we used as inspiration for ipfs uh the source code for it was found through the way back machine like I couldn't actually found it anywhere else uh so that's there you go like this is this is important stuff Vince surf also describes that this as a problem of digital Vellum of like what happens when uh we lose the ability to understand the data that we store so we forget about programs we uh or or we lose some some program that knows how to read some format and now we have a whole bunch of data that we can't decipher or is a lot of work to decipher so we should be thinking about uh constructing applications in a way that we could um resuscitate all the whole thing like we should be able to emulate every single machine that we ever build uh by virtualizing the whole thing and being able to to make sure that we back up all of this stuff in multiple different places in the world to prevent I guess some some catastrophic scenario um all right so that's that's it with the problems and that was a lot but it it's it's core to what this project is about and it's important to set it as a as a jumping off point for trying to discuss why it is critical to just upgrade the infrastructure of the network of the web specifically and why it is warranted to do this whole work which is to design a new protocol think about developing it and deploying it um to upgrade the web right this isn't uh easy so why are we spending the time it's because of these very very important problems so we're trying to make the web distributed work offline be permanent be safer move around the content smarter and actually surprisingly most importantly make it faster because if you don't make the thing faster no one's going to use it technology gets adopted because you make some performance Improvement and that's why you make some important uh change to the bottom line of major corporations and they say great this is faster let's use it and so that if if we don't focus on that and make it really good uh this whole change isn't going to happen uh finnally this whole thing started because I wanted to make things faster anyway uh so it's important to start there all right so just like HTTP ipfs is what we call a hyper media transport protocol it's just an algorithm and a program that moves around data with links that's it uh but it's the the cool thing about it is that it's it's the synthesis of a whole bunch of really good ideas that have come out since the web emerged and this is just a small selection there's actually a lot of other good ideas that we've picked up along the way uh but these kind of are perhaps the most important ones and they kind of translate directly to a stack of protocols that we use and so now we go into like the the detail of what exactly ipfs is is this stack of protocols uh to try and rebase the the entire web and the application stack as it is today and move it to a smarter transport protocol and we need some way to do naming we need a way to represent the data that is offline and distributed first we need a way to move the data efficiently and and smartly and we need a way to find the data and uh we the the core of ipfs is what we calling sort of the Merkel dag or it's kind of like a Merkle tree describe it a little bit more but that's really the Central Point uh of this whole work it's a data structure that changes the web from completely always mutable links to making a distinction between mutable and immutable links if you are able to understand when a link is mutable meaning that the content that you're pointing to has not and will never change then you have a much better uh possibility to be able to Route the content quickly make caches work and and so on and that actual realization um was made many times over in many different kinds of systems that adopted Merle linking uh along the way I'll describe some in a in a moment so that's the heart of ipfs on top of that we lay your naming that's based on on David M work I'll describe how how it works later and underneath that we just learn a lot from all the peer-to-peer protocols out there to try and build a very sophisticated way to move the content as effectively as possible but that's that's able to take in policies from the user in terms of what uh where the user what capabilities the user wants meaning that sometimes you want things to go fast and sometimes you want privacy and fundamentally these two are with each other because the way you get privacy uh in Internet protocols is by doing a whole bunch of wasted work so oblivious routing protocols oblivious Ram they all work through doing a whole bunch of expens expensive computations and expensive moving around of data um and so you need to be able to dial um the the transport uh to understand when you want to how you want to move things this is kind of similar as the work of the tour project which is to rebase uh or or to to insert T underneath htttp to move around uh the web is today through a privacy preserving protocol turns out that ipfs layers very cleanly over tour so you can use it over tour directly uh but in that case it has to be done carefully to make sure that you don't leak important information like IP addresses and so on uh so this is this is the network stack uh as we see it which is that there's there's on top of Ip and so on uh on top of the transports uh we have to deal with finding and routing content so we can use dhds and a whole bunch of other protocols we have to exchange the content but the central piece is the merel de meaning we have to change how we we think about data and we have to think through making it possible to do immutable linking on that we can layer naming and so on but in reality though we have made a whole bunch of protocols that you can use the whole thing is designed so you don't have to use anything that you don't want to use the the only piece that is critical is the heart like the core uh it's that description of the data structure that allows merco linking that is really the only piece that you really need it's kind of like IP when you think about tcpip the whole protocol got developed the same time but the core contribution was the IP network and similarly here we have a whole bunch of protocols and a big stack but they're all um cleaned up in terms of interfaces so you can really layer them on top of anything else um and uh like I described uh it should layer over Tor and TP uh cleanly and we sort of see this as the new as a new thin layer um th thin waste of the internet or rather of the web so if IP is the thin waste of the of the internet we see the Merkel dag as the Thin waste of distributed protocols uh and it turns out that it already is the the thin waist it's just that right now everybody's doing it in different ways with different formats and we're just sort of integrating them all into one system so that you can link between them uh so we see this as an internet of data or an internet of data structures um where you should be able to build applications link them with this mutable or immutable linking and then have them write over whatever transport makes sense for the user or the or the developer and so on uh and have that be a separate discussion in term instead of today which is like just the same thing you you only do HTTP over you mostly do only HTTP over TCP all right Merkel trees how do they work why why is this data structure so important uh the basic idea is that if you have some piece of data that is linking to another uh in HTTP land you would do um just look you you would have an IP address that identifies some location but the idea of a Merk link is that you link things together with a cryptographic hash so you you use the content itself to determine the link um and when you do that uh cryptographic hashes have uh it the whole point of crypto of a cryptographic hash is that you cannot come up with some other pre-image that gives you the same hash or if you do if you're able to come up with a pre-image um on demand then the whole cryptographic hash is broken and you should be able to uh attack all sorts of systems in the network right so there's this huge Bounty right now for whoever wants to break shot 256 they could run away with the whatever you know five or six billion dollars which is the uh the the whole worth of the Bitcoin Network it all rests on whether or not you can break shot 256 so if you can do that go ahead and then we can move to sha three and over time yeah we have to upgrade uh we have to upgrade hash functions and and this is an important detail that uh we're still working on but the whole point is let's use cryptographic hashing to address the content uh and that that's what gives you the immutability because if you change anything the cry cryptographic hash changes and now the link is different uh so a let's take the web and add links to them to it this is the Merkel tree which is like the first kind of meriz data structure uh this is where the idea of a mer link came out uh git uses them this is why git works as well as it does uh data within the Git Version Control System is merco link uh Bitcoin uses it this is how the whole blockchain operates the whole thing is is merco links uh and there's a whole bunch of protocols right but the point is that all of them have these different Merkel links what if we built a thing oh and and yeah Bitcoin is like this massive this massive Merkel tree uh whoever said that uh money doesn't grow on trees right so let's take the uh let's take the idealized web and add some mer linking and that's what ipfs is so you can see it as a sort of massive Forest of Merkel trees where any one tree can point to any other tree and you do this by having a common format around how you do the linking and for those specific systems that are pre-ips we'll do some for like the big and important ones we'll we'll try and make them work natively um and we'll have some mapping to be able to address things like git and Bitcoin and so on uh so we can import all of that data uh but for things going forward it would be great to just use this one format so everything can interoperate and that's kind of like the the hard and core of what ipfs is about uh here are sort of like the landscape as we see it there's multiple blockchains I don't know if how many people here are into dialed into the Bitcoin and blockchain world but there's these massive merco trees emerging that are now running Financial systems and smart contract systems uh meaning like legal pseudo legal code that's executing on the internet kind of on its own um that is mediating the transfer of property and it all relies on the this merco linking so uh ethereum which is one of these important uh blockchains that is emerging uh which is the one kind of focus on Smart contracts is going to be using ipfs already so that's that's exciting and and to explain why this Merkel linking is is so Val valuable it's good to think about the CVS or SVN transition into git so if you remember the good old days when people had to use CVS uh you had the centralized server that had all of the data or or all of the version history of of some code and you and you had a system where if you got disconnected from the central server you couldn't commit any code so you couldn't really work when you were offline uh also if the server fell apart you couldn't talk nobody could do anything at all uh and so came in and said okay look let's not do this at all this is pretty stupid what if we put the entire uh version control system in each of the nodes and you have an offline first system where everyone is working independently contributing and adding immutable objects and then over time you sync it and this is an extremely valuable idea if you get disconnected from the rest of the network you can still work and operate you can still communicate with each other uh if the centralized servers fall apart doesn't matter can still work and uh this is the same thing that ipfs does but it does it to the entire web and this so the whole idea is let's let's take this this distributed offline first data structure and use it to make the entire web uh and this is what gives us the distributed web this is why we're able to make websites and applications that link to each other that work disconnected that work offline and that have no origin server so in a sense if you make a website on IP or if you put a website on ipfs you might need a server to have the data to seed it out or at least one node but it is not uh the origin server uh any server that has the data or can distribute it can do so and the website operates wherever it executes so if you're running the website locally the whole rest of the network could disappear the data could be deleted from every other computer and you still have the code and you still have the data and you still operate that is the uh the transition the model this is what I like to call hypers speed because uh you're kind of beating the speed of light right like you're going a uh you're going faster than the speed of light because you're able to move content sometimes preemptively or know that content hasn't changed at all so you don't have to connect in any way and you can reason about the content you have and not have to um you know waste uh time or or rely on links that may be off uh so we call this the merel dag it's not a great name it's kind of hard to say we might rename this to the Merle web or something like it but the idea is you have this huge graph everything is this directed a cyclic graph where content points to each other uh with these merco links um and you can represent Unix files and directories over this of course uh you can get as an example of this uh you can do Bitcoin and blockchains and so on and you can even do arbitrary key value stores and there's even some people that are building a SQL database on top of ipfs so you could have like full SQL semantics on top of the Merle dag uh which is kind of cool right uh so how would you go about doing this well you represent a file as a dag node uh big files would be split into multiple nodes and so you can do chunking and duplication right so if you have a massive file or or two different files that that share a ton of of data you just duplicate it it's the same thing that file systems have been doing for decades let's just put that on the web itself um directory or do nodes and so on uh and so to show kind of how it looks so in in the regular web you would have this uh domain name mapping to a location um and so that again would be like you talk to a specific node and you pull it back in the ipfs world let's use the hash and address things by hash uh so you now have a a mapping from the name to the content and you now can retrieve things uh so any node that has the content can serve it to you uh the way this works or the reason that we have the uh the the reason the format is tricky to get right is because you want to preserve two really important properties of of the web uh you want to be able to resolve paths the same way that Unix brought it up right you want to be able to do Unix pathing on top of this distributed web so you want to be able to represent directory say dag nodes and resolve links through it the by finding the hashes along the way this is effectively what G does uh and so the question is if you now have this massive immutable uh log where you're adding content all the time and whenever you make any change you just create more content you still need mutability right so so we've we've gone from having pure immutable we we've gone from a web that is completely mutable to a web that is completely immutable where changing content means creating new content because changing the bits would change the value of the hash which means the link has to change you still need mutability to be able to do Dynamic content um but turns out that people uh the folks over in in uh that made plan n uh came up with the right way to do it which is that you use mer linking but you have pointers on top of the graph so the old fossil and ventti file systems work this way so you had this every time you made any change you would just create more data um and by the way ZFS works the same way you just constantly create more data and when you want to do Dynamic content you just move the pointer to the latest version so you need some way of having these pointers work across the whole web and git again works the same way in git you have immutable objects and you have mutable branches that you keep pointing so the master branch is really just a file in your reposter that has a hash in it and whenever you commit you change that value to point to something else uh so right so we can do mut ility really trivially with DNS right so all we have to do is take a DNS text record and put in a hash there and whenever you want to change things you change it but that's really expensive right like we don't want to be doing mutability across DNS um we want submillisecond mutability uh this wouldn't work on its own so we introduce a new naming system in between uh and that's what we call ipns so the interplanetary name system and this is based on David M's work uh the first file system that came out up with it was SFS and that's as far back as I've traced the idea uh it might actually be even older who knows uh but the idea is very simple if you want to point to some immutable content you generate a public private key pair and you take the hash of the public key and hand that out as a reference so when people are going to look up something they it includes the hash of the public key uh which means that they can retrieve the public key and they can check that that is correct the next thing you do is you create a pointer which has the value that you want to point to and you sign it with your private key and you take that record and you put it somewhere in the network that people can resolve it could be over DNS but that's slow again we want to do it faster um and this gives you a mutable link that you can update whenever you want by just signing the new pointer and distributing through the network does this kind of make sense this explanation is not very good and it's a very subtle idea that has wide reaching implication um I can explain it again how many people got that or were completely faced sort of uh so so again like the uh you take the content you take the hash you create a pointer you sign it with your private key and you put that out there which means that somebody can verify that it was signed by you and nobody else um and then what you do is hand out a reference to it which has the hash of your public key so if I receive the the link at the top with a Blue Link I take that blue uh hash I can resolve uh I can find the public key and retrieve that with it and I can then find the pointers and I can verify that the pointer was really signed by you so I have full mutability back uh and so these are these mutable pointers that are later on top of course nobody wants to look look at hashes so yeah how do I ensure that I have a recent copy of the pointer and not an old one uh yeah so you can do that in multiple ways so uh it turns out that you don't actually want to choose just one way of doing it uh because there are different application models that want to support different kind of updates so one trivial way would be to just update a counter right and so you you look for whatever pointers you can see and you take the highest value that's one trivial way but there are other cases where you can do ancestry chains so one pointer can point to the last pointer so you you scan for a while fetch a bunch of pointers and you take the one that is the the ancestor of all the other ones um uh yeah so we are building this as to be entirely uh pluggable with whatever pki you want to use so we don't we don't enforce some pki the idea is that you can you bring your own keys and you do whatever Keys you want to use and ipfs will work with whatever system you have and that's an important thing because many different organizations trust different kind of cryptographic Primitives and you can't tell everyone hey just use these uh because that that's not going work um so the point here is that uh we have a whole set of formats that that give us interoperability across different crypto systems so the hashes here are not shot 256 they're not shot 3 they're actually a thing we call multi has which encodes in the hash itself which function was used to generate the hash so that you can upgrade it over time and you can do the same thing with the keys you can describe what key it was and so on and uh the the hash there so this is this gets more into like a kind of like more more esic stuff but one cool result out of this is that you can take the keys themselves uh the public Keys put them back into ipfs as immutable content fetch them and then link have the entire pki as objects here that you can find so the you can have an object that that that includes the data of the public key and a pointer to its parent key and a signature within it so you can with having only that object you can check the key you can fetch the parent and you can verify that the signature is correct and you can crawl you can do this all the way with a whole pki so you can use the immutable part of ipfs to distribute the entire pki uh and then from there you can do revocation however you want um that's a whole other like how to do revocation and key rotation are really important pieces of this whole thing um but there's no one answer nobody has come up with like the one right solution and it's mostly different camps want to use different things so we just work with all of them that's kind of our principle around all of this um you won't find one solution that's right uh forever so you should make the thing be able to work with different people by introducing these small simple formats that can Multiplex over the other ones it's kind of how we do uh we work over multiple transports that way so ipfs can work over quick can work over TCP can work over uh web RTC can work over any single transport you want uh because we're we've made the whole thing self-describing cool any other questions on on this mutability stuff no all right cool I'm gonna go ahead because uh running um late so what this whole thing gives you is this huge mesh of content um that's separate from the hosts that host it and this huge mesh of content is either linked with immutable links which you can verify by hashing it or linked with authenticated links so key links which you can verify by checking signatures and so the the what it means to publish to this network is that you just create content add it and share the hash or you create content sign it and you share the your key and that gives you full power to publish into the network without necessarily having to host it yourself and without having to trust any of the people that are hosting the content for you which buys you those elusive properties which is that I should be able to create the content and store it all over the network and it should be able to be served all over the network and you should still be able to check that it was truly me who published that content and that you aren't being attacked by some man in the middle and so on uh I hear a dog quacking it's probably uh tell me to rush uh so this this web of data so so this huge mesh is is kind of like a web of data in the same way that uh the link data world and the semantic web World wanted except that instead of relying on the links being these mutable references and relying on just constant queries to all of these online servers you can package the content up and move it to wherever it it makes sense to be and everyone can check the the check the whole content Integrity on their own and so you can you can model web application and so you you can translate the standard websites of today directly onto this by just adding them as files directly onto ipfs and it it all works and that's kind of like a critical comp component to do to interface with all the databases that people use today those would still work the same way and that people still will have some restful apis and so on and those will still operate the same way but it gets really interesting when you start thinking about how to do Dynamic content or how to build applications that really have no centralized databases we're talking about being able to create applications where a client would generate the data locally so for example a Twitter client could generate the data locally I would have a set of keys I create a tweet or or an email or whatever I sign it locally and I distribute it through the network and people can check that it was truly me who wrote that um people can move it around however they want and if you add encryption around the data itself then you don't have to worry about uh encrypting the wire all the time you should also encrypt the wire but the point is you don't have to worry about the data leaking necessarily and that the data is encrypted on its own big asterisk there in that depending on how you encpt things depending on how you move it around people could break encryption and so on but it's a very different model for computation right it's a model that goes back to the distributed notion of the web where the publisher is anybody where the Hoster is anybody and where it's completely distributed and PE to um and so we can have websites that operate entirely on the browser and have no no brow no server necessarily of course servers are a really good idea it makes things faster but it should only make things faster or maybe do some critical processing but you shouldn't have to trust a server all the time with all your data constantly to the point where you no longer have the ability to publish yourself uh and that's the sort of shift in model um and you can do this so this graph could be used for anything right like if you really sync down you can start creating like legal records and contracts and and Link all these pieces together and have just this web of data that you're checking not necessarily worrying about the representation of the files themselves uh all right that's kind of how ipfs Works uh I wanted to say a little bit about how the project is going because there a whole bunch of ideas and we've I've kind of just given an overview there's really a whole bunch of pieces that we're touching um and and so where why is this different from any other good idea out there why is this different from the tons of good ideas that have um been written up and maybe implemented but haven't really uh you know been uh deployed so we're designing the whole thing to again interface with the web of today and we have like this whole sophisticated adoption plan to make sure that we can migrate to ipfs without having to um without the user uh telling right so the first step for example is you run some HTTP Gateway where you can give people regular HTP links that will resolve through ipfs and they can pull content and it all kind of works seamlessly of course that still relies in some gateways the next step is you ship them some JavaScript code which runs a full implementation of the node in their browser tab so now you have the full ipfs node in the browser tab capable of doing the entire uh protocol and moving around the content but the user never had to install anything that's not as fast so you ideally want it to be as part of the browser itself and you can do this through um either browser extensions or you can do this by implementing directly in the browser and we are actually working with the browser vendor uh a major browser vendor on implementing ipfs in the browser which is actually a show of traction like this is actually to the point where browser vendors are going oh this is really cool let's actually use it um and uh the last case and this is where we stop where we actually kind of extending the web into Unix uh we see ipfs more as an OS service that should be running as part of the OS itself and if you notice the links that we had are they don't have the colon in it right so if you look at an HTP link it looks kind of like man where's an HTP link there we go so an HTP link has this colon SL slash thing and we just have slashes trivial detail but in reality this is what allows you to mount the entire thing in your Unix file system and now you can write applications that are running in the regular standard process model that are actually part of the web so you can ship code this way you can have entire package managers this way you can model entire file systems this way you can model entire machines and virtual machines as just entities in this in this graph and so you could have your name pointing to a VM and all you have to do is remember your one name and you pull down the entire virtual machine you emulate it you first decrypt it you emulate it you run it you have all of your personal files in there and you can now compute uh and all of that's all that what that's doing is just adding more immutable content that you're now spreading back into the network so this is where where it where it goes from parody with a web to really giving a whole bunch of new capabilities that have been known and studied in distributed systems and file systems but haven't really made its way all over uh to the web uh cool so there's a whole bunch of stuff uh there that that is really exciting and interesting um but uh but that kind of is an example of how you can go about making sure that that this thing isn't just some random idea it's really the whole set of Protocols are designed to plug into all of the other systems that are out there so that's actually the majority of the work the majority of work was not coming up with the ideas was not making the core implementations it's making the whole thing interface with the rest of the ecosystem it's a whole bunch of tools that have to be made it's a whole bunch of careful design decisions about whether or not to include a colon there right that could so so that colon the HTTP colon slash it's like this reminder that a trivial decision could make it impossible for the web to be layered on Unix which created a rift between the browsers and the web and the file system and so maybe by removing that colon we can bring back the web into Unix and make the whole thing work together um that's just an example of a of a of kind of like this principle that we have in the whole project which is to make sure things integrate well and choose make decisions based on what will cause zero friction of adoption uh and we're really trying to get to like that zero friction of of of use um and we we seem to have done a good job because a lot of people are are now using ipfs so this this project is completely open source anybody can use it um we have a two implementations one in written and go one in JavaScript uh people are actively running ipfs there's between 50 to 100,000 websites now running on ipfs uh they can be viewed through regular web browsers through our gateways uh or they can be viewed natively with ipfs if you happen to be running it locally uh people are using it not just for the web but for uh file systems so people are using it to move around uh containers so there's this huge effort around uh application containers with Docker and cor and so on which is kind of virtualizing in a very thin way and people are using ipfs to move around these large images they tend to be anywhere between 5 megabytes to a gigabyte and and people are using ipfs to move them around the data center uh because you have these usually the typical construction there is you have some seed of data in the in the broader internet uh where you kind of publish these images and then you want to ship them out to a whole bunch of machines in your data center and you want that to be really fast you don't want to pay the cost of of talking to that seat outside you want to minimize the cost to move it into your data center and then use your own machines and your own local network to distribute the whole thing really quickly um so that's another use case that people are doing we recently found out that uh free Nas which is the FreeBSD distribution for network attached storage devices is now bundling ipfs uh and they have like a pretty huge install base between 100,000 to I think maybe a million users I'm not entirely sure um but they're now going to be shipping ipfs uh by default and people are going to be using it another set of interesting use cases is around package management so Linux made this amazing contribution or like Linux Community had this amazing contribution which is let's ship code through this very well um cleaned up ecosystem of of carefully designed packages and signed and do the whole verification and make it really trivial to install code um turns out that you can put all of those artifacts directly in ipfs and if you are in a Data Center and you need to download a package there's no reason you should be having to talk to the whole package manager and wasting all this bandwidth you could be getting it from whatever other computer is near you um that you happen to be connected to there's a lot of careful policies that go into which computers should be able to connect to each other but that's where the routing system comes in uh it turns out that there are some elegant ways to construct those those policies um yeah and and perhaps like for me the most interesting piece of this whole thing is that people are now building uh web applications that really have no origin they run completely on the browser um and they have they generate Dynamic content and they communicate with each other and it's all host on this huge web of content encrypted by default uh and you're now mutating data without an origin and the whole thing works uh in the distributor case so completely offline first uh it's a huge project now there's tons of contributors there's uh last time I counted there was upwards from 200 people that have submitted code um the core team is very small there's about six of us but the reason this massive project is actually doable is that we're doing the whole thing open source from the beginning everything is online uh all of our design discussions are up there anyone can come in and talk and we've had a number of very interesting developments there where suddenly we had some question we didn't really know what what the right answer was and someone happened to know that the right person was uh you know out there and had worked on this kind of thing they just tagged them on GitHub they showed up solved the problem for us and then we moved on right and so this is kind of like the next um of course this is kind of like simple for most people that do open source like GitHub is this great resource where there's this huge social network around just contributing and making code um but we are living this amazing dream of being able to just pull in the right expert at the right time whenever we want um and the whole design discussions everything is there so people can contribute any any kind of work there's a whole bunch of uh interesting algorithm questions around crdts crdts are these Conflict Free replicated data types uh where they're a perfect fit for PFS because these are immutable data structures by Design um or or rather they're immutable and mutable in the same way that ipfs is immutable immutable uh and they map very cleanly to ipfs and the world experts on it suddenly started having a discussion in our forums of discussion around how best to layer this in ipfs how to use ipfs to move around their content how to build applications on it and that is uh what is making this whole gargantuan effort possible um right so so uh yeah any so again in in short uh we are trying to upgrade the entire internet and like our our life is really about uh this whole process of research development deployment uh of protocols and if I were to say one last thing here in this venue at Stanford where people are doing so much research uh uh for the network is that truly when you look at uh the papers we the idealized systems are 20 years ahead of what we have deployed uh or what's in use today and there's this filter there's a set of filters around how much of that research gets developed uh how much of that research gets developed cleanly in a good usable way uh to the point where deployment actually makes sense how many of those deployments are done correctly uh to the point where they're actually widely used so many times the reason great ideas don't make it to be used is that these filters uh which are mostly around not coming up with ideas but rather how do you take this idea and ship it to the real world um these filters uh kind of like prevent great ideas from from Distributing and this is what we we really care about uh so we're building this organization called protocol labs around making sure that those filters uh are better uh in that many times you can just go in and realize that some implementation if you tweak it change the language or just try it again you can now make a really good system that can actually be deployed uh so this is where where I would encourage a lot of people to spend time on uh thinking about what old ideas are really good and just maybe it wasn't the right time maybe the implementation wasn't very good maybe it just didn't get deployed the right way but in in reality there's nothing else holding it back and how can we upgrade the internet on the web as a whole by just uh putting some work there uh great so that's kind of everything sorry this has been pretty long