The Mess We're In | Complexity, Entropy, and the Future of Computing

[00:00:01]about two and a half weeks ago on it was a Friday I had a bad day a really bad day um I had to prepare a lecture that I was going to give on Sunday uh it was to the commercial users of functional programming and I'd very foolishly put up a title how to make money from functional programming um um people thought that because I had invented Aang and because WhatsApp was written in Aang that and whats that was sold for $19 billion I knew how to make money but so here I was with this silly title and I'd taken Friday off um to be at home cuz I can't get any work done at work cuz you get interrupted all the time and I started open office and uh a little prompter came up and said there's a new version of open Office oh great so so I oh that's good it'll be better so I clicked on this little thing and uh yeah what happened yes I down I installed open office and I started writing my side I put some images in it and and suddenly all the images vanished oh goodness and I wasn't really pleased about that and um so so I Googled a bit and and and and it said oh open Office can lose images under some strange circumstances this is reported several years ago um I'm still there and uh so I thought oh golly what do I do now well Apple had very kindly um broken keynote as well for me because they had my workflow so I have a I have a um a a MacBook Pro big one 17in whatever it is heavy and I do all my slides there and I and I put them into Dropbox and then I have a small 11in air air machine there and and then when I finished my lecture I close the lid and I open the lid on this thing and it SNS up so Apple had very kindly put different versions of keynote on one machine and the other so that the things I saved on one couldn't be read on the other they completely destroyed my workflow so so I wasn't very happy about this and so I thought I know what I'll do I I'll do my slides in HTML you've seen these slideshows that in HTML yeah they're really nice but there's one problem with them that people like to get PDF copies of the slides afterwards and and so most of these slideshows that people have done in HTML don't produce decent PDF so if there's anybody out there who's WR written one of these you know please make it produce decent PDF um and so I Googled a bit and and I found a project that said you can make nice slideshows in HTML and they can produce decent PDF and so I I downloaded this program and and I followed the instructions and uh it said I didn't have grunt installed and now now I'm an old guy so I didn't really know grunt was and my grunt files weren't right or something so so I so I Googled a bit and and I found out what grunt was grunts I still don't really know what it is but um so I I downloaded this um thing and I installed Grunt and it said grunt was installed and then I ran the script that was going to make my slides and it said unable to find local grunt but I just installed grunt so I I turned to Twitter and tweeted and I'm having a really bad day here it's unable to find the local Grunt and and and Twitter is very helpful because people said well your grunt path is incorrect you know set your grunt path and and I I set my so so I gave up so that was it right so you'll get to the title of the talk in in a in a while um I I've been programming since uh golly a long time ago 1965 something like that that's quite a long time and uh program go actually goes back longer than that um the first computer program that is remotely reminiscent of the the way we write programs today ran on June the 21st 1948 and uh it was written by Tom kilber that's Tom Kiln he is the world's first I they say Adra love list was the first pro programmer but she didn't program the sort of machine we program today this is Tom Kilburn uh holding a Williams memory that's that was the thing that enabled uh programmable machines it's it can store uh what is it 64 32 bit words right that was the total storage and it's a cathod ray tube so as the as the store is changing you can see on on the display uh what was going and this is the first program that was ever written it was uh that was it and it ran on in I forgotten the date in June 1948 and the nice thing about that program it was probably the last program that was totally correct I mean there's just one program in the entire world this is the first program that you put the program into memory and you can change the instructions and then they're executed and it and it computed I think the first I think it checked the first 38 prime numbers or something something like that and and they're all very pleased at it work of course they didn't realize what they had done and uh and I don't think kilber thought that you know I'm just imagine he was programmer number one that's kind of cool right um why am I talking about that well let's let's go roughly halfway between 1948 and today 1985 this is this is when I started work on airling and at the time a computer work looked like this this was a a really powerful fantastic ftic super duper machine uh typical PC it's got 254 kilobytes of memory isn't that a lot and uh an amazingly powerful 8 mahz clock speed that's pretty cool you can do amazing things with that um yeah it was great so if we look at today's computers this is this is a typical laptop um it's got 8 GB of memory so it's 30 2,000 times much much more memory than that little thing had in in the mid 80s it's got four cores running at 2 and a half GHz that makes it about thousand times faster it's got a big solid state disc and and so on now in in 1985 that machine would boot in you know 60 seconds and this machine is uh 10,000 times faster or a thousand times faster it should boot in about 60 milliseconds how many of your machines boot in 60 milliseconds right so something's gone wrong where did it go wrong okay so so that's the title of my lecture the mess we're in I'm going to look at the things that I think are going wrong and and I talk a little bit about why they're going wrong and uh then I'm going to talk about the physical limits of computation so so I'm going to relate the speed of computers to some physical quantities so so we can just see kind of ballp part where we're in and then I'm going to um suggest some ways that we might be able to clean up this mess okay I I think software is actually getting worse and worse and worse with time um it's not that we can't do amazingly good things with computers of course we can but when they don't work we don't understand why they don't work okay in the past you see go back about 30 years ago if you had a program and you didn't understand why why it didn't work you could look at it and now when you've got a program that doesn't work you don't look at it you you you ask Google you know why does my program work and and and then of course it says well you know try this and it'll make it work you try it and it doesn't make it work and I'm going to explain why that is because I I was thinking about that a bit um right so um by generational programmers I think we we should get big medals you know we should get the Congressional Medal for creating employment because we we we have created billions of manh hours of Maintenance work for people in the future we we are the job creators of the future all that stuff we wrote years ago um you know these poor sods will be scratching their heads and going what the hell does this stuff do it's terrible right so um so what went wrong I'm going talk about that what are the laws of physics because I used to be a physicist so so um I'm going to tell you about black hole computers and and things like that we can if we can make them they would be quite good um okay so I just When I Was preparing this I thought ah I know I was going to show pictures of Gustaf dores Dante's Inferno and you know the levels of hell and all that kind of stuff and I thought well I'd write down seven deadly sins and I I was doing this on the Underground and uh I tweeted I I got by the time I had to get off the underground I'd written down 25 and and and um I had some trouble in sort of ordering them um if you look at these C I mean one thing I noticed is code code that I write now which I can't understand in a week's time do you do that am I unique in that you know I write some code I really understand it and a week later I can't understand it how how many people do that oh good I'm not alone I just thought it was only me um I think that's because your brain works in two different ways it sort of when you're working on a program you're like cashed into the program and you see no need to explain it to any because it's bloody obvious so you don't write any documentation and and when you've cashed it out you wish there was some documentation because you can't understand it at all right so this is very difficult so the answer to this is comments and and if you look at these sins no comments in the code you can't understand it no specification very obscure it's not beautiful that's all about shifting your brain from the mode you're in when you're writing code to the mode you need to be in when you're explaining how the code works right so has anybody heard of comments does anybody put comments in their code right does anybody put no comments at all in their code in modules right yes yes I've done that um a lot Robert Robert Ving who developed airline with me um was famed for his comment I think he only wrote singular the entire stuff he wrote had one comment in in the middle of the patent matching compiler there was a single line that said and now for the tricky bit which I thought was quite good now comments are really good so I would I would advise you to write comments now um and write big comments right really big really exclusive comments now how many people write really big comments right well I going to say there's another word for a really big comment what's that no a book how many how many authors have we got here well done thank you very much because if you haven't got a book you don't know how the bloody program works right so a a book's just a big comment right and it's difficult at first just write your comments they're bigger and bigger and bigger and pull them out and stick them in a book and once you publish the book you'll be rich and famous and they'll invite you to conferences and you can talk about your book um we'll forget about the Rich and Famous bit you can talk about your book anyway right okay so so we've done all these deadly Thins and things and and there's a load more um and then today we we've got this stuff Legacy code to deal with um what's Legacy code well that's that's dead that's dead programmer stuff I mean not only are there no comments in it um you can't ask the people who wrote it cuz they're dead um and it's written in languages that nobody understands it's written in Cobalt and and and there's no specification and yet it works beautifully possibly and and and management thinks that modifying Legacy code is cheaper than a total rewrite I can tell you I've looked at some Legacy code sometimes you know changing one line of Legacy code is equally difficult to totally rewriting the entire stuff um but management doesn't think that and so what they think you should do is bung it in a virtual machine and don't mess with it right I'm now going to just just talk a little bit about complexity um because I used to be a physicist and um here are some sort of numbers that you can have in the back of your head um the mass of the earth is 6 * 10 27 G um the number of atoms on the earth is you know 10 50 atoms and the number of atoms in the universe well the observable universe is about somewhere in the range 10 78 to 10 82 that's the total number of atoms in the universe ah excuse me it's a very big number okay so let's um solve a little equation here I'm going to solve the equation 2 the K = 10 50 right we don't need Mathematica for that we can do it in aing sorry stevenh K is 50 uh log 10 divid by log 2 so K is 166 right so 166 bits 2 to the 166 is equal to the number of atoms on the planet and if you divide that by 32 you get 5.18 and round that up you've got six so what's that telling you six 32 bit integers in C the number of states they can possibly be in is the same as the number of atoms on the Earth right so if you wanted to test your program by Computing all combinations it's going to take a long time right and and we can work out how how long it would take don't ask me about JavaScript because it's a it's only three variables in JavaScript that's about the number of states that three variables in JavaScript could have is greater than the total number of atoms on the planet right just think about that right so a computer is a finite State machine right it's got state but how many states has it got well it's got an awful lot of States okay so finite State machine is state cross event and you get a new state so the number of states my my little laptop here has got 250 gab of flash memory on it so the number of states it can be in is 2 to the 200 50 gtimes 8 right and the number of atoms in the universe is say about 2 to the 260 and that means you need two to the whatever that number is it's unreadable universes to find two computers that have the same state right so this explains this this gives you a very good explanation to why your programs don't work and why Google because what happens is you you you download some thing onto your machine and you do something and it doesn't work so you Google and you find something and it says uh oh I had exactly the same problem as this do this and then there's like 10 males after say gee thanks that's really great and so you think I found it and you do the magic spell it doesn't work so you Google again and you find another mail says I had exactly the same you know it's the same story and you do it again and again and again and then finally it works and you don't understand why okay so why did it work for this guy but not for me well it's because our machines were in a different state when we performed those operations and the number of possible states of the machine is a whacking great big number and I'm just not going to find somebody who's got a machine that's in the same state as mine it's going to be in a different state and that's why it's not going to work right so what are we going to do about this well there's all this math stuff functional programming scary stuff we could try and prove programs to be correct but that is way way that that can deal with programs that that that are very small not with the size of programs that we have today um another thing we have to deal with we have to deal with failures computer system fail they're just going to fail we have to deal with it we can't we can't ignore it and um this is to to handle failures we need to go into territory that that is unusual um to handle failures you need two computers you might need 10 computers or 50 computers you can't handle failures on one computer if if You' got a program executing on one computer and it fails you're screwed you need two computers or 100 computers if you replicate something over 100 computers and the chance of one of them failing is one in a thousand then the chance of all hundred failing at the same time is one in a th000 raised to the power of 100 so you can make systems that are almost incredibly reliable provided you can replicate things and keep them independent but in order to do that you need to understand distributed programming and parallel programming and concurrent programming you see you can't this leads you into this territory if if you're running programs on two machines you are riding distributed programs they run at the same time that means you're riding concurrent programs so you need to understand this this leads you into territory this is places you don't want to go so it's quite easy to make things that are not fault tolerant and not scalable but if you want to make things that are fault on are scalable you need to go into this strange territory and uh there's some very good books actually that can help you yeah Hope they've got it outside it tells you how to do this so systems we build should self- prepare self-configure and evolve with time they should be more like biological systems um languages yeah this is another problem um people say notation doesn't matter you know they don't like Airlines notation it's got curly brackets and funny symbols and I it's not uh it's not like Java it looks it looks different but but notation does matter because the Romans they weren't very nasty if you had to do arithmetic I was actually thinking writing a program do prime number of arithmetic RSA in U um Roman numerals I thought be quite fun have to do that as an exercise um languages do matter but the problem is in 1985 um I think all programmers knew shell scripts and make and see so there was a a sort of lingua Franco we could talk to each other all programmers could talk to each other actually they couldn't that's a lie because there was another lot who knew cobal we didn't talk to them they were already the sort of clicks we took and there was the other lot Unix and c and stuff make shell scripts make see well now um we don't have these common languages which we can talk to each other you know there's Dooby dooby-doo and and futran and Ruby Dooby and grunt files and it's funny when I when I tweeted and I say my grandfather doesn't work another tweet came out Jo it's gone over to the to to the dark side so I had to tweet uninstalling grunt later after I'd removed the nasty little thing from my machine so so right so um when I learned to program you could choose between these three languages and uh now um yeah I well I don't know how many there are I when I first made the slide I I found somewhere there a 676 program then I found another side 2,500 and haven't a clue what most of them do um that's very difficult if you're a beginner which language do you choose to start with there are so many um right so we don't have this lingua franer to talk with and and we don't have the build tools we used to have um make everybody had make FS that were great and now we've got ant grunt make rake malvin jake break bit bake fabric paper shovel and I I checked on the net and I found found this um woo well um is there a there was a lovely question is there a rake equivalent in Python well you could use pava invoke shovel oh I don't know who cares another thing these build systems I was writing aning program at work for for Ericson in anger and I'd written three or four Aang modules and and had to go into an embedded system guy says oh I've got a there's a script that does that was actually a make file that invoked a bake file or a bit bake file and it was taking rather a long time and then I I stopped it after well it it had downloaded 46,000 files hang on I've got three files what what's happening do you really need to recompile the entire Linux kernel and GCC and everything and build it oh yeah we do and I I know I should have used grunt or something and it would have been much easier so without without Google and stack Overflow programing would be impossible I how many of you can program without an internet connection for more than 5 minutes well the rest can't it's terrible it's terrible right and then we've got this sort of dichotomy between efficiency and Clarity you know to make something clearer you add a layer of abstraction and to make it more efficient you remove a layer of abstraction so it's kind of what should we do um so go go for the clarity bit you don't worry about efficiency wait 10 years and it'll be a thousand times faster if you want it a million times faster wait 20 years it's quite easy doubles in speed every year is that right um yeah just wait a bit names oh I don't like names I I'll talk about names names are very imprecise um we don't unless we can agree on the meaning of names um we we get in a mess because we we can't talk to each other I I'll say much more about that later I'll get on to this bit now how I do for time yes we're doing all right um so what do the laws of physics have to say about computation um I I sort of got into this when I I was looking at the manual P for earling and it said um make generates a unique reference um and then it says will reoccur after approximately 2 to the 82 calls 2 to the 82 calls is uh about 10 25 you remember that number 10 to 25 remember the number of atoms on the earth is 10 to 50 so it's quite a big number right so here's some bits just I next physic I'll just remind you of some laws of physics causality a cause must always precede it it's event right something happens and something happens later because of it and how things happen are because we propagate rays of light or sound or something we we don't know that Something's Happened until we get a a ray of light or some thing that's conveying information to us okay now a lot of a lot of um systems actually breaks the laws of physics so this notion that you can have consistent data in two different places breaks the laws of physics it's a bantine general problem um if I know something suppose Suppose there two computers and I say um hello the value of x is five and then does it know that can I assume that it knows that the value of x is five well no because I don't know if that message got there so I want him to send a confirmation back he'll send me a message yeah I know that X he'll send me a message back yeah I know that X is five can this computer now assume that I know that X is five well no he can't because he doesn't know that message has got there so he won't know it's got there until I send him another message to say that it's got there and um that's the bantin General's problem so you can't actually replicate data because you have different knowledge of the system and different points in the system despite this fact we build systems with two- phase commit and forget about that you've got two- phase commit yeah it works well two-e commits breaks the laws of physics so it's not very good and then physics is all about things like Concepts like simultaneity I mean if two stars explode in the universe somewhere um the guy sitting halfway between them will say that A and B exploded at the same time and if the guy to the far left he'll say that a exploded before B and the guy on the far right will say B exploded before a because it's it depends upon the time that light takes to to reach these things if we if we forget that fact especially with M building distributed system who get into big problems you shouldn't right systems that that violate laws of physics uh yeah that's a slide that just says that um entropy so another law of physics the law of physics says entropy always increases um what's that mean it means you know if you got a load of dice Chuck them all up in the air they're not all going to land with six upwards or one or something like that they're going to get more and more disorganized as as time goes on that's what happens to software the entropy of the software increases and some fundamental limits to the speed of computation um here's a couple of physicists who who who knows who the guy on the right is no left well left if you look at it who's that guy Max plank yes yes um so he this is Plank's law it relates energy to to Plank's constant and and a frequency and the the hairy guy that's um Mr Einstein he said he he's so plank said e is H mu and and Einstein said e is MC s and so H bre put these two numbers together said well okay so H mu is MC s so mu is just MC s over H that's called the brem the Breman limit and it's 1.36 * 10 50 Herz per kilogram and that says that's the quantum limit to how fast a kilogram of matter can can change State how fast it can vibrate so if You' got a uh it can vibrate that quickly so if you look to Quantum Mechanics you'll find all these fun numbers you find the Breman limit the margolus lentine theorum the birkenstein bound and all the so so let's look at some of these see the the Breman limit that's the the fastest clock rate that you can get out of matter it comes from quantum mechanics and and it's 1.36 * 10 50 hertz per kilogram um then you've got the amount of energy that you can get you can you can you can do 10 33 operations per second out of a jewel of energy um you can store two times 43 um bits per Mass a radius kilogram whatever that was and the land a limit the minimum amount of energy to change one bit of information these come from quantum mechanics right oh and a just because of that they you could work out better jewel is is actually a 10us 21 of a jewel and a watt is a jewel per second and a Facebook data center is 28 megawatts which is 28 * 10 6 per second which is kind of let me see it's 27 orders of magnitude bigger than the smallest amount of energy you could use which is not very good so now we want to so so now we want to build a really fast computer so how do you build a really fast computer you you squash the components into the you know you put more and more components into the box and we limit the weight to 1 kilogram we squash more and more stuff into it so ultimate just becomes a 1 kilogram black hole that's the ultimate computer um that will actually do 10 it will operate at 10 51 operations per second right and it's got a size of 10- 27 of a meter but there's a problem with it it lasts for 10 to minus 21 of a second right and it emits data through Hawkin radiation and quantum entanglement right so we don't actually so it's kind of fun thing there's a picture from a Scientific American article you drop you drop a pair of you've got your quantum computer you drop a pair of particles into it and one of the and the computation takes place inside the black hole and then through quantum entanglement the particle that's on the outside flips its spin or whatever you're measuring and and you've got the information out so so you've got to set up this thing that can measure what did I say let's go back it's got to measure you know' got to get this uh 10 to the 51 operations per second going and then we've got to store all this stuff with quantum entanglement and uh yeah I don't think it's going to work for a while I think I think we've got to learn a lot more before we can we can make this thing work so if if you're interested in that kind of very nice Scientific American article um on the physical limits to computation so why do we want to know these numbers well one group of people oh wait a let's add a summary of these um so just summarize this a 1 kilogram computer can do 10 to 51 operations per second of s 10 to the 31 bits and a conventional computer can do 10 to n operations and store 10 to 12 bits of information so you see there's a big gap between what's physically possible and what we can do today of course perhaps the ultimate computer isn't the 1 kilogram black hole it's the entire universe condensed into a black hole behaving as a supercomputer and actually in in this article uh you'll see that the Universe the entire known universe has done 10 to the 123 operations since it was booted so when the universe was booted a few years ago it's now performed 10 to 123 operations so who's interested in this number cryptographers so if you want to make a crypto system that is uncrackable if it takes more than 10 to the 123 operations to crack the code even with a quantum computer because this is a quantum computer you're going to need several universes to to crack it right okay so that that was the physical limits of computation so let me see yeah was on time so what can we do how how can we sort out all this mess so now I'm going to go into Uncharted Territory well I want to build the entropy reverser this is a device that we can you imagine I I try to find a big sausage machine where you put sausage you know you turn the handle so we put all programs into it and we turn the handle and a smaller number of programs come out and we throw away all the other programs um and that breaks the second law of Thermodynamics um trouble with software you see it it complexity increases with time we start with one program and it splits and become two programs and four we want to reverse that process and and this is a problem I've been thinking about for many many years uh I I I I show you some of the conclusions I came to um okay so there are all sorts of problems with with things files and systems they mutate all the time they they they grow in entropy discs are absolutely huge and there's all these problems with naming naming is horrible if you've got a file or something what what file name should it have um what directory should I put it in can I can I find it later I I've been Pro I I programing a lot I WR about three mod say about three modules a day um so in a year I write about a th Airline modules and I've been doing that for 25 years or 30 years or something 25 years so so I got like 25,000 airling files that I've written on my machine and then I've downloaded so when I checked I heard my 85,000 airling modules of my machine is that 85,000 different things no probably not it might be 5,000 different things I would like to take this 85,000 things and put them into something and turn a big handle out would come the irreducible kernel of that the 15,000 things or 5,000 and things so so I need to figure out how to do that so the first thing I want to do is abolish names and places right so to talk about things you you've you've you've got to have names or you have to references to them or some kind of name so so You' got just a paragraph like that cup of tea he sat down and a butter slice of toast it's difficult to refer to that okay it's paragraph number 46,000 no parag paragraph number I don't know 42 from James Ro's ulyses but that's a rather opaque reference rather difficult to follow it up but it's quite easy we could we can just compute an sh1 check some of it so if you were talking to your friend and you said ah I was reading this really great book said yeah what was it called well it's called Uh 78519 ad1 1438 you know and then you would know exactly what he talking about there be no problem at all if if you believe in sh1 that is we all that's do we all believe in S1 yes you know it's like a religion or md5 or something like that and then somebody say no no no no no no no no it's been broken well I don't care I believe in I believe in it well it's good enough it's good enough for this right okay so so we've got this number thing how do we find it if we've got this number well you eyes that they're sort of bad things um that's an address you know ww. fu. well why is it bad well it needs DNS and you can spoof DNS and the host might be unavailable at the time when you want to get the thing and if you change the thing being pointed to the reference is wrong and the reference is somewhere else and you can't cash well you can cash it but you don't know how long to cash it for um and and if you request that a man in the middle might change it before it gets to you so you really don't want something like that so what do you want well if we went away from Uris to hashes um you just say okay uh get me that thing that's a URL okay you you're not going to that that'll be embedded in some other document they got in the file you'd just be a link you click on it in your browser or something um the ice thing about that this is a Content addressable store is you haven't said where it is okay and you don't need any form of security I mean a man in the middle could change the content but you can validate when you get there you just you say go get that thing you get this blob back you compute it to S1 check something and it's the same as what you wanted and therefore a man in the middle cannot have changed it and you don't need secure sockets or anything like that for that reason you you can't damage that content um you have no problem choosing a name you just take it and you run the sh one check some on it and there you are and you can cash it forever right that's very nice so then the only question is how do you find that thing well um there are algorithms like like cord and and uh cadamia and this is well studied okay so for those of you who don't know it's one of my favorite and most beautiful algorithms is you you just take IP addresses of machines that's the IP address in the uh right hand column and you compute sh1 check sum of that that's in the leftand column you sort the sh1 check sums these are now in a sorted order and you say okay so which par which machine am I going to find that paragraph whose sh1 check sum well suppose the sh1 check sum was uh you know 4 six something or other I just look in that list and now I I find two machines that bracket that one IP address is lower and the other is higher so I just send a message to those machines and say um excuse me they're they're also going to have a table like that for the machines they know about okay and so they send back to this machine an updated list of that and we can refine that address down and down and down till we narrow it down to a small number of machines who the hash of their IP address is very close to the hash of the thing you're looking for okay that's called cadamia and and uh it's a basis of a peer-to-peer system so this is actually combining sort of the ideas from git and and um bit turrent the big torrent trackers and the the dhts that are used for that um are based on things like Camelia and cord and these are self-organizing distributed hash tables um the idea of of using hashes to identify things is is widely used in git and and and it's so when you combine the two you get you get um you get git torrent which which is uh I I I got very excited a few months ago I thought oh if we combined git and bit torrent we'd have git torrent so I immediately went in oh I'll register the domain name get torrent found two other people there were two other projects already doing it so you know there's no such thing as a new idea okay so then let's start using these let's start making the condenser well uh the first bit's easy find all identical files right so how do we do that um for f in all files on the planet do c is the content key is the sh1 check some store the key content in a distributed Global file store that will condense all identical copies of things to a small number of copies that that will reduce the total number of files right that will run uh because it's a distributed computation everybody does it on their own machine it will run very quickly if we were to set up an infrastructure like this we could collapse the total number of files on the planet to a small fraction of those that they actually are at the moment by condensing them down into this store that that's easy to do the next bit is much more difficult and that is um to find files that are similar to a given file so this is this is a problem that that I've been thinking about for 20 years or so and I I made embarrassingly small progress on it um I have in my back of my head this idea of a of a thing that helps me and and I've built one or two of them and they some of them work and some of them don't um so the idea is it's rather like Twitter you have a little box when you have an idea you have a little box and you type something into the box and and I've done this I've implemented it you have a little box and and then there's a little icon Sherlock Holmes at the bottom you type the stuff into the box you press the Sherlock Holmes button and the idea is that will find among all my files that I'm interested in the most similar thing to what I've just put in this box so I want it to find the most similar thing to this new thing and then I want to know is it different so once it's found them it it it makes a list of them in order and then I can look at them then I can make a decision is this actually a new idea that would be great if I had a new idea that would be fantastic you know or is it an idea I've had before and just forgotten about or is it an idea that somebody else has had which I don't know about so once I've once I've found these other things I can then make a decision if it's a new idea or if it's similar to them and if it's of course a what do I want to do now if it's a new idea that's fine if it's similar to an old idea I might want to edit the old idea and put the I want to merge the two together so slowly we can start to condense um the the amount of uh information so there are various ways of doing this one one which I'm currently playing with is is called least compression difference it's a very sort of it's kind of nice how how do you know if two things are similar if a is similar to B um then if you if you compress a and work out the size of it and then you concatenate A and B and you compress a concatenated to B and look at the size of that and if a and b are very similar the compression algorithm should if it's a decent one be produce something that's only slightly larger than a compressed if a if B is identical to a and when you compress a plus b the compressed a plus b should only be a tiny bit bigger than a because the compression algorithm will find all these sort of similarities and that's true for small data structures but um it's not true for big data structures and for programs it's worth because two things could be similar but they're sort of isomorphic but you've changed all the variable names in the function you still want to find the similarities there are other methods there's um latent der allocation there's basan inference there are all sorts of techniques the trouble is you know the whoops the algorithm was where I had the algorithm here uh yeah oh no find all identical files it was for it's for all files on the planet compute this thing using least compression difference it even takes like takes minutes to go through a few hundred files doing this because it's got to compress everything in then and do it so if anybody out here you know wants to work on something for the next 40 years and figure out how to do this um please go ahead because I we really need to reduce the complexity of everything we've been doing right so um we've made all this mess dyra said that that um Computing was about controlling complexity and we have failed miserably okay I think one way to do that that by I think about 128 kilobytes of stuff is about the limit to what the human brain can can can do the only way to make reliable systems in modular composable systems is to make them out of small units which we can validate and then and then connect them together I think we know how to do it in a way um we we actually need to reverse entropy we need to take we need to do the opposite of you see GitHub is s of cloning off things and it's getting bigger and bigger we need mechanisms to make things smaller and smaller um quantum mechanics does set these kind of upper bounds on what is possible so when we're figuring out the complexity of algorithms um we need to be aware of that we also need to be aware of the state space of what we're doing is is enormous and that's why I mean if we relate the complexity of the program to the number of atoms on the planet we see the very very small program has as many possible States as the number of atoms on the planet you know it's just enormously complex we need to abolish names and places and replace them with hashes and we need to set up Global distributed hash tables and things like that and then we need to make lower power computers that don't do environmental damage um we don't really want to fry the planet in in answering all our questions we need to make these carbon neutral powered by solar panels and things like that we need to get down the energy of computation computers are becoming a big environmental threat they they're using more energy than than air traffic and things like that and this is something while we can probably do without um well no we need both air traffic and we need computers one is capable of going down to very low energy the other is not and computers can be made to operate extremely low power we need to do so with with a degree of urgency um so that's what we've got to do we've got to clean up the mess we've made thank you