Erlang Multi-Agent Programming: Supervisors & Demonware

[00:00:04]of course I'm happy to see so many of you many of you will be video on the video today I want to continue finish talking about multi-agent programming entire life I will start by giving you Refresher how Airline works the basic important Concepts especially into concepts of linking and also very quickly the gym server and some patterns to get you back into the way of thinking and then I want to talk about supervisors and I will show you the code the complete code of a simple supervisor this is really the the magic of Airlines how the supervisors will help make the behaviors reliable and then in the second part I will give you an industrial talk this is the slides were made by an engineer from a company called demonware which does a lot of support for gaming systems and this is a very comprehensive industrial talk that shows how Airline is really used in a very complex and large industrial setting so this is very a nice this gives you some experience how things work in the real world okay so that will be the second part so all of this stuff is not just Theory but it's really useful in practice okay so first compressor but before that a few seconds of respect for The ukrainians Who are fighting for their freedom and for our freedom okay let me first refresh you very quickly I'm not going to go out through everything but I will refresh human process linking and it's the Gen Surfer so let's go to the linking this is the basic mechanism that's used to build what I called the Deep magic of Airline which is the supervisors and the behaviors that are supervised the basic mechanism is the linking where two processes are linked together and when one process terminates a signal is sent to the other one and this is transitive if you link many processes it makes a set of linked processes and we distinguish between normal exit everything's fine it just terminates no problem the exit reason is normal and so it uh it could be ignored eventually but if there isn't any kind of an error a runtime error hardware error software error anything that causes the process to fill the exit reason is given and the sent to all the linked processes so I said it's transitive and a process can also not terminate when it gets the exit signal but it can receive it as a message which means it can observe the failed process and this is a really important basic mechanism and it works because the processes are independent remember I said there's no sharing there's no pointers there's no shared memory each process is an independent little island and one process can then observe the other and this is the mechanism this is all you need this and the message passing out is all you need to build the magic the real strong magic of early okay that's one thing that I want to refresh you on to go for a little bit farther I want to so everything has this has a certain philosophy you know that failure should be very simple feel fast the number of failure modes should be very simple let it crash and Let me refresh your memory on the concept of a behavior a behavior is like a programming pattern but it's not just a disciplined by the programmer there's a software library that implements the behavior and this is a concurrent fault tolerant pattern okay and the airline when you write an airline program you don't use linking yourself you don't actually send even send messages yourself you've used the behaviors and the airline platform OTP provides five behaviors that are standard four that are part of the application and the fifth one is the supervisor that's what I want to talk about today this one is connected to all the others so there's a lot of very clever codes behind this facade all of these other behaviors are supervised okay and the three important ones and it's basically honest is the generic server the generic event handler because events can travel in the system and the finite State machine to their finite State machine so this is actually very general any computation is modeled as a finite State machine huh so a state and an event gives you some actions and a new state okay so this is the core of the computation and then there's the two others are for how things are communicating and then there is a bundling of everything into an application that can be started and stopped together and this is the application Behavior so these five behaviors are really the core and they're built on top of the simple message passing linking stuff okay that's the second thing now I refresh your memory on what we saw last time the generic server I showed a very simple generic server so let me just give you this picture first this is generic module so this is part of the system okay this is part of OTP uh the generic server and you can use it and this one the specific model module is what the programmer writes okay so the programmer calls the server start stop call but the server has callbacks so these two interfaces are part of the server so the server interface you start to stop and you call it and the Callback interface and that's all you have to write and the rest the external interface is defined by the specific module so basically all you have to do here is Define what is the external interface and what is the actual computation how do you initialize something how you'd actually do a computation handle and be terminate so it's very easy and you start you're working and you stop okay so this is the generic server I showed you the code of a simple server this is very simple one okay just to show you how you can separate the specific stuff from the genetic stuff but this this one here it does not use a supervisor the real gen serve gen server is supervised okay is doing much more than this even though from the outside it still looks very simple so this guy is doing a lot of stuff and uh is supervised okay for bad fall tolerance okay so that's the second thing I want to refresh now I want to talk about two things two little things programming patterns little programming things that I didn't talk about last time to refresh also your memory on how people you think you have programming is done in our life so here's a simple example so message passing you send a message from one process to another and sometimes you wait for reply but sometimes It's Tricky says hidden difficulties you can have race conditions sometimes and I'll show you one of the problems of message passing when you get up when you want to do uh call and get a reply this is done internally by gen server right so the programmer doesn't actually have to do this the Gen server will do this but it's important to know how it works okay so here's the problem so I have a client and I have a server the client sends request to the server and gets a reply back okay the client here sends a request to the server so this is a tuple and you have to reply back so this is a very usual kind of pattern in in Ireland but what if there's another guy sending a message that has exactly the same form as this reply how do we make the difference so this is kind of a race condition huh you see this one is doing request reply but what if somebody else just sends a reply there how can we be sure how can we guaranteed that this won't happen you see that because this reply might have might arrive first and then it is a problem okay so here's another one this is sending reply but this reply is maybe not the reply coming from the server how can we be confident that the reply is actually coming from the server and not from somebody else but with this code we cannot do that so this bug is actually bad code but we can solve this problem we can make sure that the the clients that the reply we know it's coming from the store can you imagine how we do something like this if you didn't see how would you fix that problem all right how would you make sure it's actually very easy to fix it okay do you have can you think what would be the way of solving that you see the problem huh so you somehow have to distinguish those and the idea is kind of to make this one unique yeah another content with the state that is only known to the client yeah something like that so you put something in here which is only known to the client and that's very easy to do you have to put some kind of a unique thing and this is sometimes called the knowledge or something a unique value so here's the fix we airline has a function called make ref this creates a unique concept okay what they call a unique reference which cannot be made forged by anyone else notice this is very similar to what we used the name of the concept that's basically the same idea and the fix then is you create it you create a unique constant here ref when you send the message you send a tuple instead of just your own PID you also put this unique constant in it and the reply this Tuple has an extra field which is going to be the unique constant that means the receive remember how they receive works with pattern matching it can take messages out of order it will pick the message that has this constant so if there's another message here's the fake message huh reply and ref this is just an atom it doesn't do anything reply it will not take this one okay so the receipt will take the first message that matches notice it's very important that the receive takes any message in the mailbox out of order okay that lets you take it doesn't matter which order these two messages are in there the receive will always pick the good one because of this unique concept okay so you see this is one way of making the messages unique and this works also because of the receive they can remove messages in a different order okay this technique would not work for Port objects which receive messages in the order there you would have to do something else okay but this is a simple pattern that works for uh for Erlanger and this is used all the time to guarantee that the request reply is never corrupt enough it's a secure request so that's a simple pattern so you see the kind of thing when you really do multi-agent that you have to worry about another thing I want to talk about is synchronous and asynchronous calls so this is very simple and when you do when you call or you send a message from one process to another sometimes you need to reply and sometimes you don't okay sometimes you have to wait for a reply and that's called the synchronous column sometimes you don't and so the Gen server will give you two patterns to interfaces which are called call and cast so call here is actually a a round trip it waits for a return value if you remember the the frequency allocate example we did in the Gen server we use call okay so notice Let me just go quickly back to the code for that so notice we we when we call this this is the specific module what we call the generic module we use column which means we need round trip but actually for the dallocate we don't actually need to get a return it's just enough to send the message so go back going back here if you want to do an asynchronous call sometimes you send a message but you don't expect to reply so the gem server has another part of the interface called cast so the idea is cast when you cast this you just throw something you don't wait for a reply okay so the idea is you're just sending and you're not waiting for a reply here and the deallocate in our example it could have used a cast actually I showed a call but actually if you want to make it more reasonable you don't have to wait for a result so it's useful okay so that gives you back so this kind of stuff is automatically done not into behaviors now now let's go to building a supervisor so first I refresh you on the supervisor concept so the supervisor is really a key Concept in paragraph it is the concept really that makes Airline a complete and very nice system for building robust software okay so a little bit of uh motivation so we're building concurrent distributed systems with failures most faults and errors are transients that means if you retry the system will work not all of them let's say 80 to 90 but the others are not but that means if you can already handle these you will already have a good start so the simple retry is actually a very good strategy as the first strategy and supervisors started out like this and then it was elaborated you can do much more of course but the basic design lets you very easily do retry okay so you have a set of workers they are supervised so supervising means that there are processes that observe the workers but only the failure of the workers they don't care what the workers are doing you know they only worry about failure and the supervisors have authority that means they can stop and restart they they can control the the working of the supervised processes and it's recursive okay the supervisors can themselves be observed okay so it's a treat and the supervisor trees that are implemented actually it's kind of tricky to implement them if you really want to do it well because when you start up the supervisor you can imagine you have to start it up exactly right or what if there's a failure during the startup of the supervisor so you can see it's not so simple but no problem don't worry this has already been done this is implemented in the OTP so you don't have to worry about that this prop this is already solved okay okay so the supervisors make a tree huh you haven't supervisor knows so the root note that is the top level there is a top level supervisor and the internal nodes so this Rook node of course is very important uh if that one fails then you're really dead okay so this is a typically an extremely small and very well tested piece of code so that's a very important thing but it's usually extremely simple this root number and so a supervisor has child processes and all behaviors the four other ones are designed from the start that they can all be supervised so this is built into the system okay and when a supervisor wants to restart the workers it has a strategy and basic their basic strategies there are four basic built-in strategies one for one one for all rest for one is simple and for one so which I will explain and you the basic strategy is just to restart when there's an error many errors are transiented surprisingly effective this is something well known by people who use Windows huh if you had a window system has a problem we just reboot this is kind of a running joke for a Windows system but it's a good strategy to restart but you have to uh you have to carefully design it you don't want to reach time forever the number of restarts is limited so you don't restart too fast and the number is if the limit is exceeded then the supervisor will terminate and there's a higher level supervisor it says oh there's another it's another problem okay so the basic retry strategy is only the first level okay and maybe for example there's a disk and each time you reboot it it crashes again and this might happen once because it is a piece of dust in there and it works or you have to reformat or something but then at the higher level if something goes wrong you might have to reconfigure the system to remove that disk so that's something that's done at a higher level okay so the supervisor has this hierarchy so here's some example so these the rep the squares are supervisor processes and the circles are the workers the workers are standard Behavior this could be gen server and the supervisor so here one means it's a one for one and a means it's a one for all I'll explain what it what it means okay so one for one basically means the children are independent and one for all means they're collaborating so if one of them fails you have to restart all of them so you have this hierarchy and each supervisor observes the children beloved to other workers okay so the two basic strategies are one for one and one for all if any child dies on one for what it is restart automatically by the supervisor within a consistent state and the others are not affected this happens when the children are doing independent things you do that and one for all means if any child dies all the children are restarted because they are working together in a consistent state so this uses the stable storage when you restart to put everything in consistent State and there's actually two more strategies so the one for one and one for all are shown in the picture okay there's another one rest for one is if there's a sequential dependency and if the chair if the child fails crashes all the children that are started after are restarted because there's a sequential dependency maybe the the this guy this worker has created like someone contracted things to others so if this one dies all the subcontractors have to be restarted but not the previous ones okay so let's rest for one and simple one for one is a dynamic version of one for one where where you can create new children and stop new children dynamically during execution so really you have a bunch of independent processes that you could always start in stuff so this is a more Dynamic version of the one for one okay so that's the idea so you get the idea but I'm going to show you the code of a supervisor now if we go through the codes in detail this is a very simple supervisor but it supervises it doesn't follow so you have a supervisor with children If the child terminates abnormally it is just restarted so it's like one for one if it terminates normally it's removed from the supervision tree so nothing else happens so there's no further action nothing happens to the others and if you stop the supervisor you say I want to stop the supervisor it basically stops the whole application all the children are terminated okay so that's a very simple supervisor and of course you can do much more sophisticated stuff this supervisor assumes that nothing abnormal happens when the supervisor is starting up the children and so the real supervisor has to do that a little bit more carefully if the supervisor cannot Start Up Normally the whole startup is aborted so that means that the supervisor could not start so it only works if the supervisor can actually start with the children okay and each child is started by calling the function an airline function called apply where you have a the module a function and the arguments so this function is the one that actually creates and links the child to the parent okay okay and we use linking for this remember all the linked processes are when the supervisor terminates the linking processes will receive an exit signal okay so this example is taken from a book called designing for scalability with Airline OTP by Francesco cesarini and Steve vanowski this is a really good book and I'm only given a small piece it has many many code examples and a lot of very good explanation of how everything works the philosophy how everything fits together so if you want one book on really understanding how erlang does everything this is probably the best book to take for that okay so I know Francesco and he will be very happy that I'm complimenting his book so here's the code this is the complete code and I'll explain how it works this is a module I call it my supervisor it has three and uh calls three functions that are exported start which starts up all the children initialize in it which initializes the loop normally you won't you don't need to call this one but it's also exported just in case and this one is the stock okay so how does it work you start up the supervisor you give it a name the name is registered I didn't refresh but remember register as a way to even name to a process like an atom and I will now create a new process and this question module is a macro it basically is just the name of the the module so it would be my supervisor and it will start a process and call in it with the argument is the list of child specifications okay so when you start it up it starts this and then this one returns okay and PID is basically the actual uh supervisor process okay now in it first of all it sets that trap exit Factor true remember this make means that the supervisor will not fail but receive messages and then it starts this Loop so it's like a server Loop this is the loop here I'll explain in a minute and it first it calls the function start children so start children here what it does is so this is a very compact kind of airline code this is a missed comprehension this will take this list here child spec list it will create a new list for each element of child specialist it will do something so it will child specialist is a list of tuples that are triples module function and arguments argument and this one will actually create this function will create a child and here I have a tuple so the brace is the top of one so I'm creating a list so the bracket is a list I'm creating a list with two element tuples the first element first I start the child by calling apply this one will return okay PID but I only need the PID so I'm taking the second element of this double element two and then I also have MFA in this Tuple here because I need this if I want to restart the channel the children okay so I have a list of tuples which have two elements the first element is the PID of the running child and the second element is the original child specification okay so that's all the children are now started and this function when I will actually spawn and Link so this is child specification huh all these children will be linked to the the process that's doing this so the supervisor okay and then this is the loop so a supervisor it's kind of like a server um it handles messages coming in what are the messages well the messages are the exit messages from the children okay so there's two exit messages exit PID normal and exit pad with some reason that's not normal so this normal is an atom so this one will match first so if the if a child terminates normally then the supervisor will automatically receive this because of the linking and all it does is it removes that child from Child list so the loop has an argument which is actually the list of pids well actually these tuples of the children and supervising so if the child terminates normally just removes it it uses this erlang function he delete which I'm not going to explain you can imagine how it's worked so it will basically remove the the element from the list okay so China just gets smaller that's this one second one if the child crashes any kind of failure it does runtime error whatever then get this message and reason will be different from normal what do you do then so the child has crashed remember erling has this letting crash philosophy any failure you crash do we know the child is crashed so we restart the child here we have the PID in the child this so here's the function restart child so basically you looked up so PID is a constant term it's the ID of the child so we looked up the child in here from this one and we create a new child this one will have a new PID yeah whenever you create a process it's new and then we call P replace which replaces PID entry by this one so remember childist has these tuples then the PID and the MFA which is for restarting it so it replaces the PID by new PID so this has basically restarted the child and it has put the PID of the new child in the list okay and no one now I'm calling the repressively loot with the new child list okay which is returned by research child so that's very easy yeah and the final message is stop that this message comes from the outside others should be lowercase s none of this comes from the outside this will terminate so I can stop the supervisor all of the children at once from the outside so basically the determination it will do a for each so we'll go through all elements all children and it will basically kill them actually that's very that's very murderous huh you're killing children here actually this is kind of funny because the the book the book that I just mentioned it has a box saying be careful people don't like when you say you're going to kill children so don't say that okay but that was the original terminology early was done like in 1988 and they were not so politically correct in those days okay so this is the full code of the supervisor so you see how it works though and so you have you I expect you to understand how this code works huh so this code is actually not so complicated right note that you have linking you have higher order and you have uh this notion of separate processes with message passing you can do this fairly simply huh you see that done so I've added a slide and I'm going to put this new slide on the Moodle with some explanation because I realize that maybe maybe this is a little bit complicated to understand like that it has things it is so I added a slide so the the question module is just a macro so erling has this macro system value is the name of the current module to list comprehension is something that's used here so airline has this concept of a list of comprehension where you can create lists from other lists okay and I'm explaining here what apply does but element 2 does and the function f this is the child specification it spawns the child and immediately links it to the parent so despondently here's what it does okay and the supervisor Loops so this Loop function here so it kind of looks like a server it has a loop like a server Loop because it's always doing something and waiting for things and the argument is a list of tuples so these are pairs the running child PID and the module function argument which you need to restart the channel okay so you see how that works so this seems very straightforward right doesn't it simple but it's very powerful this is what I'm just showing you here is the foundation of why Airline is so powerful for building robust systems okay to let it crash no sharing and supervisors okay so just to summarize a little bit uh This what I showed you is the generic part I didn't actually show you what the children are doing the code inside the children I didn't actually show you anything but in the book by uh by cesarini they show they give examples of what the children could be doing you might for example it might be doing the frequency allocation okay so he respond supervisor to jam clock to start children you monitor restart clean up so this is all generic stuff huh and the specific child to child dependencies maybe the children depend on each other that might actually affect the supervisor what children okay so there's a generic part in a specific part okay so that's how the supervisor works and this gives you a simple example of a supervisor and the real supervisor just Builds on this and it does much more than this but it's just as easy to use okay now I mentioned the second part of the Supervision in Airline which is the stable storage so this is done here you restart the children [Music] you have to restart it here is nothing special you have arguments a but really in real life the children might actually be doing something in the real world they might have some State maybe it's doing like phone connection spending money or whatever so there's some stable storage so it's not enough just to restart with the initial arguments and so then the second part of Supervisors is the stable storage an airline has a really good support for that the basic simple support is called ETS erlang term storage this allows this is the very fast stable storage when a process fails the ETS does not disappear the ETS is like it's like the global in some sense it's kind of like Global variables that are stable it's in memory so it's not in disk so it's not as reliable as disk but it's extremely fast so this is the fastest version and that airline has more uh slower and more reliable functions so the ETS and then you have disk ETS dedex and then you have magnesia which is a full-fledged transactional database but here I just mentioned ETS because it's the one that that it's used most of the time first okay so the ETS is in memory when the process crash is ETS values are unaffected and you can actually the ETS tables are also managed okay so it's typically managed by the supervisor not by the child process because when the supervisor stops then the ETS table is deleted so there's no garbage collection of ETS tables it's managed by the supervisor okay not by the child's process okay and so this is the second part of the supervision done supervisor and stable storage okay okay now let's take a step back and compare this so later I'll talk about industrial use of early but right now let me just let's just think about it a little bit so supervisors handle the errors when a child process has an error it is designed to immediately crash it does not try to fix the problem or remember the example of the surgeon who did the appendix I removed his appendix on himself the child process does not do that that's very bad you don't want to do that this child process immediately crashes and the supervisor does the rest so the child will not try to fix the errors so all error conditions are mapped to one failure action which is depression okay so no defensive programming uh it used to be that when you write programs it was kind of recommended to do defensive programming which is putting checks the children will be checking a lot of things and there's actually some studies okay so for example here's one done where you compare a piece of a system which is done in C plus plus and in Airline so the same functionality so this is an example uh done at Motorola okay two-way radio communication so if you look at the airlift implementation it has 85 percent less code which is means it's like 1 6 of the size twenty percent of the C plus plus codes was defense it was error handling whereas only one percent of the airline code huh nothing okay so typical studies they show like the same results so for the same kind of fault tolerance the airline code tends to be a lot smaller you can say so Erickson did a study here which is Plex which is a kind of imperative language and there are like local kind of Ericsson language and they also saw enormous increase tenfolds decrease in the code size so you can say wow that's amazing is it really true why isn't everybody using airliner this is true actually heirloom is used a lot and I will show in a minute that games gaming is an area where airlift it's used a lot when you really need the thumb toggles so it is that the answer is exactly is used a lot okay so that is the conclusion for the first part so from last week and this week I showed you the airline plus OTP the behaviors huh The Primitives so it's two levels huh there's The Primitives and then there's the behaviors and I showed you the Primitives because this is a course where I want you to understand the magic but a programmer will only see the behaviors number okay so a lot of effort goes into the design and and the Improvement of these behaviors so all of the work of the OTP team most of it was in building the behaviors okay server Finance State machine vent Handler supervisor trees and the stable storage is a very important part the ETS dets and Venetian okay so this is the the conclusion so you have here a system with The Primitives and the high level concepts for building concurrent distributed systems okay so that's the first part of the today's lecture the second part will be explaining and giving I will give you an industrial talk so this is important this is a really [Music] um a real industrial experience okay so there's many talks on airline so there's a lot of industrial conferences things there's a conference called code beam which is like twice a year there's one in Europe and one in the United States and so this is a talk I chose this was in a conference called erlang Factory held in London in 2011.

[00:45:15]and it's called erling and first-person shooters okay and so this is by a guy from a company Malcolm that was from a company called demonware Ireland okay but maybe maybe and this gives a lot of industrial experience maybe I can break now can impose and after the break I'll show you really get into the nitty-gritty of uh industrial application let me start again this is a talk this was what I'm going to present now is was a keynote talk done by an engineer Malcolm daus at a company called demonware which is based in Dublin Ireland in 2011 in one of the main Airline conferences and those days it was called Airline Factory and this was in London okay tens of millions of Call of Duty Black Ops fans load tests early so this guy's giving his industrial experience and uh in several parts first the history of demon Blair how did this company start how did they get into Airline what are they doing so that's really interesting if any of you want to start a company to see the experience of these guys the second part is how they use erlang now now this is 2011. it's still true and today and what we have learned so the Lessons Learned mistakes they made and fixed and things they would like to add and what are the nice things and what do they especially like about Irma okay so both the good and the bad okay demonware is a company that supports games and they do a number of things middleware for client to client game State transport so you have two Gamers exchanging data you have encryption you have Nat traversal these slides are on the Moodle and I've added some text in red on the bottom to explain some of the terminology that maybe you might be always familiar with so and that is a network address translator is basically changes the IP address from inside the company or or the house to outside because they maybe don't use the same IP space and so you have to manage that also connection management peer-to-peer peer-to-peer is that the clients are talking directly without contacting a server but you also have star and topology a star where you had one server in the middle and all the clients are talking to the server even when they send messages to each other so you have both of these things so deep and mirror supports that that's one thing they do they also have they basically support everything that is done that is outside the actual game engine so you know a game has a kind of a physics engine inside that models the real world to some degree of accuracy and all has all the game logic but it's very highly optimized so this part is not written in Airline this is a special this is a very specialized high performance kind of engine but everything around that is what the middleware does and this is called the lobby servers so before getting in you go to the lobby server so things called matchmaking leaderboards statistics messaging audio video websites teams and also things against cheating so the library servers is what allows players to meet and arrange the games as opposed to the game servers that actually run the game so you basically have Lobby servers and in the center you have the game service okay so all the stuff around the game servers is what even aware does okay so how does this company started it was founded in 2003 development middleware for games so basically a bunch of uh developer students actually 2005 they started hosting Lottery servers in 2007 they switched to using Airline and which I will explain how that happened they were then acquired by Activision Activision Blizzard and in 2011 they're one of the world's largest online game service providers that doesn't mean much in numbers of employees they have 60 employees offices in Dublin and Vancouver okay this is some games that use them this may have used some of these games I don't know Call of Duty you've heard of Call of Duty I guess huh so there's most of the Call of Duty games and also other ones um are using software from demonware so there's a lot of support for games and many more so demonware is really a big company for support all the things that go around the the game engine okay so that's deep in where so what do they support the full online infrastructure for Call of Duty Black Ops in 2011 this was the world's best-selling game four of the top 10 games on Xbox Live over 2 million different services not bad for a piece of software 150 million registered users at its cross-platform so a bunch of Hardware Xbox PS3 those days the Wii and those days okay coming soon 3DS today but that is still going on today yeah how did they get into Irvine okay so that's an interesting story so here it's very interesting maybe for you guys in 2003 it was founded by students from Trinity College Dublin okay so you can see that this was a bunch of students they wanted to sell client-side networking middleware to game studios they thought how do we could build support for games one year later it didn't so far didn't work out so well lots of polite interest polite interest means oh yeah looks very interesting goodbye found a few customers okay but the game studios wanted online servers they didn't want middleware so they started creating this Lobby Services platform okay uh Xbox and Playstation okay so Xbox Live was kind of the standard in those days so they started by using C plus okay and my SQL MySQL so MySQL is a relational database management system open source very popular okay so they made a C plus plus server kind of regular single threaded they created some process for a service and the application logic is in C plus plus and use MySQL okay now this was their first thing so this was actually not very good one OS process per connected user was not very good huh so you have like 80 concurrent users maximum so but the first game that they connected into did not have so many users so also C plus plus crashes a lot if the code is immature so you need to do a lot of testing and debugging and testing in debugging so it crashes if the code is immature and the code was immature and you crashed a lot okay so this was the first thing but then they they did better they used put python in the middle they regrow it all the C plus plus logic in Python they maintain a pool of os processes a core server and C plus plus now they're able to handle thousands of concurrent connections they encrypt encrypt this patch they have messages asynchronous messages they even have some cheats detection licenses duplicate login connection but the code was very complicated so please C plus plus is the wrong language for concurrency and their experience it was very hard the code was very complicated impossible to maintain poor error handling debugging was very hard metric scalability was very hard whenever they had to change the configuration they had to disconnect all the users so note that Airline wouldn't have to do that okay so problems problems okay so this is now in 2007 one of the former developers just by some kind of an idea rewrote the server in Airline just to see what it was okay as the server was in a C plus plus and after a few weeks he got this prototype and it was running oh and it looked nice and then amen and then they said hey let's turn this into something real and they took them four months of development to turn it into something that was able to go live so in mid-2007 it was used for the first time so you see how it went it's just somebody said well we're having so many problems in C plus plus let's just try erlab seems to be maybe better we'll try it okay so they tried and in a couple of months a guy actually got something nice okay so some of the improvements so it didn't crash that was amazing the configuration was easier they could reconfigure everything without affecting the clients amazing up hot code you can run it reconfiguring the clients don't see anything the logging Administration was better faster developed far fewer lines of code so this is the industrial guy who's talking here okay now 2007 okay still very small company they have now several bunch of customers so they have this nice Lobby server now it's written in Airlines working nicely they have some customers nice customers Activision Ubisoft Etc Activision like it liked it then acquired them they had a few big games you may know some of these but still no monster Blockbuster okay so in 2007 they had games with twenty thousand characters it's not bad but it's not the huge Blockbuster yet and it's still kind of a tiny company 11 developers three persons handling the operations and three managers that's kind of small um so this is and but still they were doing stuff and then they get a real challenge in 2007 a blockbuster arrives this was Call of Duty 4 okay 2007 you may remember it I don't know this was the most popular game on the PS3 in those days but for them it was lots of pain and suffering so they have the system of 20 000 and it was working and then this huge thing happens and the number of users was always growing growing growing five months every always growing for five months new bottlenecks every weekend things were going wrong they had bottlenecks to solve lots of outages and late nights so this was like growing up for the company okay they had to grow up so before that it was a small company 20 000. and now all of a sudden ooh they're in the big leagues okay and so erlang actually caused a few issues they were able to scale up and they kept all the garlic out so without the switch to her like the crisis could have been a real disaster but they actually survived this crisis and actually got bigger and bigger okay you can take a little growth from twenty thousand to two and a half million compared users and Slots okay the number of requests per second of their system from 500 to 50 000 the number of servers so you have lots you have millions of users but you have a bunch of servers okay the original they have 50 servers now they have 1850 servers talking to each other that's big system okay and then the company grew in staff from 17 people to 60 people still not that much out for something like this spread evenly between Vancouver and Dublin so Canada North America and Europe also they grew in competence that really matured they added many new features services in 2010 there was the Black Ops launch this was an enormous day so it was a lot of work for them okay so the core of this thing is airline with python and my sequel but many other pieces so not just as that is a lot of them things okay there so how do they use erlife so you now you see kind of the growth you see this is these are a real company yeah and you see kind of how they grew from just a few students doing something to a real company playing in the big big piece okay so how do they use Airline okay foreign is their core server so it controls python so it's lots of Python scripts it manages 100 hundreds of thousands of concurrent TCP connections it schedules tasks for the python code it gathers metrics so statistics are maybe there's bottlenecks so you have the system has to measure itself it has a presence server so this is a kind of database this is using magnesia a member that mentioned this in erlab a fragmented version of Manisha which means it has some different servers and all everything is done using message passing okay they also did other things with Airline they made other kinds of servers for lots of extra kinds of things transient in-game data through servers testing bandwidth leaderboards racking in real time so basically the error room used for was used for all the concurrency and gluing the sequential code together and sequential code could be written in Python for example or for other languages okay so this was basically the core server controlling the concurrency of okay so so how do they do some of the things so the TCP connections the test scheduling for each connected using user there are two earlier processes so remember we have the supervisor with other workers so you have two processes that are handling each connected user and they use a simple one for one which means you can start and stop them individually and they delicate to work these processes delegate work to python actual operating system processes and there's so there's there's a large supervision tree here they have dedicated task queues so you have lots of things going on and you can restart or update the python code without affecting users so this this is pretty nice and it's all controlled using error okay they also have periodic tasks using timer so this is the kind of structure per user how it's done they did something called a presence server this is something that you need to ensure that a user could not be logged in twice and that there's no duplicate license Keys like on PCs and who is connected so this provides a snapshot a concurrent distributed snapshot of who is connected it also supports all the in-game messaging okay this is using an erlang Library called fragmented Manisha or the Malaysia is existing on multiple nodes so this one actually scales very nicely and robust and the the single so they have clusters clusters which are collections of computing nodes that are tightly coupled so their biggest cluster was a 60 [Music] 16 core Devils so this thing more than six cubes this thing has more than a thousand cores okay that's pretty big with the present server they also do metrics in a big system like this the system is always measuring itself they're seeing whether there's bottlenecks or you have to keep track of everything and erling has this SNMP so network monitoring protocol libraries so they're used so when they monitor the users or best per second request times queuing times logins logouts per second why the users get disconnected or thrown out so and then they make these real-time graphs and the most used thing is an ETS remember the ETS is like Global variables that you can update very fast stable in memory ETS update counter so they're counting so they're calling ETS update encounter a lot so this basic little heirloom function is really called a lot millions of times okay and also between all the Clusters they're combining the metrics to generate overall metrics so cross cluster metrics okay so here at the bottom you can see I had uh sometimes a little red text stuff simple network monitoring protocol a certainly okay each game has a different configuration can be very complicated a lot of all different pieces have to be connected together in the right way and so the airline code handles all this game configuration with all the different options the defaults instantiation inheritance if you want to connect one configuration depending on the other one upgrades between clusters roll back on failure so this is nice so wrote record failure when there's a failure the system goes back to a previous consistent State okay and something called puppet integration puppet integration so there's a lot of kind of jargon buzzwords in this training puppet integration is an approach for delivering and running the software the same no matter where it runs okay so it's kind of independent of where it's running okay and a lot of effort to make this as simple as possible usually configuration is one of the hardest things okay okay so this was something they worked a lot on okay there was also web interfaces lots of web interfaces and airline has this uh generic web server called Yaz yars yet another web server which has a very high concurrency you can use to have many tens of thousands when you saw the graph in the beginning last week for the web console this is for the Developers okay and live debugging but while the game is running you can also do development web services you can control the game message of the day you can remotely see how popular game features are you can control the customer's remote so there's lots of web interfaces for controlling for debugging for developing okay and they have a lot of services related to the game one of the things people like is the leaderboards so giving the ranking of the users so these are huge these are Community boards or 15 million users ranked in real time so this is using the ETS stable storage and a a kind of a tree a special kind of tree management module okay so that basically a data structure that lets you kind of write things and find out which is the best dynamically okay also they have key Value Store a key value store is like a database it's like a hash table where you put things in there it's a key and any kind of value so you can store it and update this they built it on top of ETS so this has a very simple database huh and this lets you do things like grouping the users keeping little groups managing chat channels presence information so it's like a little a simple very fast database built on ETS okay so key value store is like a hash table a table of values indexed by key GP trees are General balanced tree modules so this is a way of managing a large amount of data organized as a tree that is dynamically balanced keeping so you can always know which are the best ones okay so this is a module of erlang other things they did was bandwidth testing UDP packet blast oh it's a nice way of calling it you said huge number of packets throw them at an erlang server and see what happens and the clients can get the estimates of the bad group okay so that's Services they did okay now what lessons they learn so you can see kind of what they're doing with uh erlang so what lessons they learn about our life okay so this kind of reviews the stuff we saw in about earlier the core data types records that propose binaries bit streams we didn't use them so much the refs and unique values the Lo list special kind of list data type second think functionally think concurrently you have to think concurrently so forget all the sequential stuff Terror person functional data structure higher order so they use lots of high order new processes really are that cheap so that's something people who are used to programming genre don't really believe that a process can be extremely cheap and they really are it says okay so simple things can go a long way okay that you have a couple of erlike features here so I O list so this is i o list is a list whose elements of integers binaries or other eyelid IO lists a kernel poll is the kernel Poland file descriptors so this increases i o performance with many descriptors so it's an operating system technique okay so don't forget to read the red stuff okay now OTP this is the behaviors what they find is use it okay religiously means only it and use gen servers use the supervisors avoid touching receive and send don't actually send messages yourself do everything with behaviors don't do spawn yourself spawn link which spawns and makes the link at the same time don't call trap exit all of that is done inside to behave so really use the high level stuff okay also split reused components into their own OTP applications if you have pieces of a system that is reused turn it into an application and OTP supports applications okay also the modules erlang is organized with modules keep them small they don't make up they don't have side effects they don't update Global state hello keep them sequential the concurrency is in Behavior okay so the modules can be OTP Behavior modules under the specific modules or simple non-side affecting sequential modules so all the concurrency of the default tolerance sustainable behaviors yeah can you remind me what OTP stands for oh well it's the name that Erickson gave to the platform it's called open Telecom telecommunications platform because it's basically open source so this is the name they gave when they made a public release so this is all the libraries not just the language but all the libraries and systems but they call it the open Telecom platform because the original was for Telecom and then there's another lesson very general lesson which a kiss principle this is a very important principle keep it simple stupid a slogan for system designers don't do things in a complicated way keep things simple for example avoid internode dependencies dependencies are bad let's generally say again not just our life of course Airline makes it easy to have connections between nodes you just have PID of another one or the node here is the computers huh a single computer in a network of computers they shouldn't depend on each other so the no sharing principle you have to use it also not just the airlines you use it okay avoid notes with special responsibilities everything should be kind of decentralized okay and you should expect that sometimes you have my legacy don't expect things always to run fast expect problems okay internal network issues expect problems because of dependencies complex inter-process dependencies very bad be very afraid of processes which rely on each other you shouldn't have a bunch of processes all connected that's very very bad okay and casts instead of calls remember I mentioned cast and call call is round trip huh you send and you receive but a call is kind of a dependency yeah you call and you wait for the result so you're dependent on the other guy so it's actually a dependency call up so what they find these guys from demon Warriors who want to avoid that even a simple call can be nasty because you wait for a result what if no result comes what if the other guy fails what or something happens so if there's a delay SO waiting for something is very bad tests they don't wait cast you just send send it and forget so casts actually are the way to do it so your system should be asynchronous don't wait okay the calls are waiting this is very simple you think why is this a problem it's so simple it's a call even something so simple is a problem okay because here I mean these guys have millions of users huh even small problems can go into big problems huh okay another lesson bottlenecked processes now we're talking millions of users concurrently and there's lots of processes handling this what if one process is like a bottleneck it receives many many many messages huge number so it is slowing everything down okay that you want to avoid it avoid the bottlenecks so you create a pool instead of one you have several and each one handles part of the some of the messages also make sure these processes are not actually doing so much work because if they do work and they have many messages huh not good okay and manually purging the message queue so if there's a lot of messages coming in sometimes you can get rid of some of them okay so that's one thing the second one some processes do real work they actually have to do work well when it's doing work leave it alone don't send messages to processes that are doing work seems very simple okay and also it decides when it wants to do more don't overload somebody with work but when that person is finished that person the process is finished it asks for more work so you have a pull not a push approach but pull approach if you build big system then each process will ask for a word and you don't actually push work don't send work unto processes but wait for the processes to ask for work and that will let the system kind of balance itself that really is important for a big system okay okay and some bottleneck processes are logging the log process so logging is important because you want to keep it keep track of what happens so the log puts all the things that are happening the trace in a stable storage but there's if there's one process doing that that process will get a lot of messages so you don't want one process doing the login or metrics you want to keep track of performance also if there's one process doing that it will be overloaded so these guys are important but they tend to get overlooked okay not important lesson okay the stable storage so ETS not our long-term storage the lesson is ETS is great use ETS it's the standard solution for most of your in-memory storage it's seriously fast they let industrial people love to use the word blister okay blisteringly fast so fast that you get blisters I don't know okay it's also automatically cleaned up it's remember I said it's connected to the supervisor so it's actually deleted when you don't need it okay it avoids single process bottlenecks because the ETS can be accessed by many processes okay okay no mods depression so a crushed up is something that happens when something precious and you have a huge uh dump of what's everything what's in memory if you use ETS then you don't have to do that so much all the stuff is in the ETS okay so use ETS but ETS has limitations it's just variables it update on ETS it's not a transactional database is a transactional database in their life so don't reinvent Benicia if you need transactions use Venetia otherwise use ETS also uh what if you have a distributed system so ETS is on a single node what if I have multiple nodes multiple computers huh that have to store things I have to make copies of the ETS tables so you have to worry a little bit about that but so that's a little bit harder but the lesson is use BTS second lesson is Malaysia so I said there's like layers of database in Airline starting with pts and then D this DTS and then when Asia so the top one UTS is fast but very primitive the other one is slow but very sophisticated Tunisia is a very powerful database it's distributed it handles fragmentation atomicities transactional surreal strong database and it stores Airlines data and this is one of the main reason they moved to Airline they really liked Tunisia but Manisha is a complex so this is a really complicated thing big databases tend to be complicated a lot of subtle custom codes it handles partitioning it handles no death I want to know the crashes it handles fragment distribution how do you spread the database over multiple nodes so Manisha is kind of like a traditional relational database management system it's powerful it's fully featured but it's complicated and this is the industry guide you will swear and pull your hair out so I don't know if he was bald or something but pulls his hair out but ETS is simple but it's maybe sometimes too simple so you have this kind of dilemma do you build something on top of BTS or do you use medisha okay so the idea is use Malaysia but carefully okay okay testing very important automated tests so you see that already here so this is the industry guide saying it they use e unit which is the airline unit testing the testing full clusters a system for stepping out modules basically making modules turn into uh I'm not sure exactly what definitely stubbing out stubs fault injection killing random Airline processes this is a very nicer you have this big writing system with lots of early processes just randomly kill one or more of them just start killing them and see what happens that's a really nice way of testing out randomly killed the processes ingesting and this is called fault injection because something else almost certainly will if you do that and if it keeps on working then you're really good huh okay also there's pay attention to dialyzer and as Prof so dialyzer and f-prof are tools for airline for debugging profiling type finding bugs and finally end to end load test you simulate two million users so the test is just huge same where they have two million simulated users pounding on the system okay that's very important okay so these guys you see they're very serious fault injection heavy duty and time blood testing okay and a few other lessons keep your clusters apart this is part of no dependencies huh different local area networks different cookies keep your customers apart beware of shared cores with other OS processes I have five multi-core at this core and there's many os processes on there problem don't do that okay off sub process priorities ten thousand relatively unimportant processes slightly inefficient will clobber will kill one vital process so a large number of unimportant low priority can actually affect an important process so be very careful about that also they talk about the hot swaps and the code replacement so we saw that The Primitives for that huh amazing but often more effort than it's worth don't do it all the time huh do it rarely so don't be too uh enthusiastic about it so don't use it so much and then things like kill switches metrics graphs okay so a kill switch is a mechanism that's used to shut something down in case of an emergency so if something is going wrong just shut it down okay lots of helper tools remote shells for looking for debugging so this is all of the industrial stuff so here you're really getting some real industrial lessons huh final finally kind of a political lesson be polite so Airline is actually a small part of dealing where it's all the concurrency is done so just three or four earning developers or do it another concurrency stuff all the other things other person doing Python and C plus plus and other things so your co-workers are not all Airline Fanboys they don't care about or like you do don't force the user of your software to use erlang syntax to read early Impressions has to understand Airline codes so this is very general I don't force people to do all your internal stuff okay either you convert them all or you say well it's a niche language so erling is kind of a niche language it's used for small parts but very important part the concurrency the fault tolerance but that's actually in terms of lines of code it's only small parts even though it's important so be polite because it's only one piece okay so that's uh the lessons now they also have things they want to see in her life okay some things we'd love to see so these guys are really industrial users and they once things they want things that didn't exist maybe some of them have been added since 2011 huh they want improvements in Malaysia Network outages no crashes maybe combined with each other with Cassandra so Cassandra is a another database not Berlin one but has eventual consistency and as good support for known crashes so they want eventual consistency automatic rebalancing and then they understand there's something called the cat theorem which maybe you have not heard of has anyone heard of the cap theorem no no the capture is a very important result in distributed systems that there are three properties that you want consistency that all your data is consistent with each other in a big distributed system availability which means that whenever you make your request the system is working and answers your request in a reasonable time and also partition tolerance if your system breaks into two if the network breaks you want your system to keep working now these three properties are all 3D very important but you can prove that it is impossible to have all three of them at the same time if you want a system that is consistent available and the partition problems it's impossible that's known as the cap theorem that was proven in 2002 you could have two of them if I have partition tolerance and availability but then maybe the system will be inconsistent or I can have partition tolerance and consistency but maybe sometimes the system will not be able to reply when I make a requests so it won't be available all right so available as a formal concept availability means that if I make a request the answer will come within a reasonable time okay so of course they say we want Venetia to be better but we know there's no magic bullet okay the capture was a general theory that handles works for any distributed system okay what else did they want automatic cleanup budget magnesia data right now is not connected to the process responsible so if that process dies diminisha data should be cleaned up okay so they they kind of want that one kind of extensions or old and new incarnations of a node it's the old enough for note crashes and you have a new version the two have different data okay and you have to clean up the data from the old one which might be spread around maybe you want that to be more automatic okay okay here is a very nice slide they want improvements in OTP here is this is actually what I I I like very much receive send link spawn is like Assembly Language it's very primitive okay it's The Primitives uh is this so you don't want to use it but you still have to know how it works when you don't use it so what they want is more flexible supervision trees hand crafted dependency they want better ways one for one rest for one is is kind of too primitive sometimes they want better restart remember I said the thing restarts certain number of times they want improvements like exponential backup so exponential backup means whenever you restart you wait a little bit longer or you multiply the time and this can reduce congestion okay huh they want more ways to provide process monitoring can also be supervised okay they want system messages to be handled quicker writing well-behaved blocking or busy process is messy so they they want some a new but they want a new Behavior gen background Scripts okay so they want more from OTP okay also what they want is connecting earlier to other languages erlang for them is not really general purpose it's great for concurrency and hard cry concurrency but they would not use it for business logic they would use something else the ease of concurrency does not make up for the difficulty in interfacing with other languages okay they want better interface to other language it's just too easy to muddle through without Airline it means you can do all kinds of weird stuff without Airline it's hard because using our line is hard because of the interface it's not the airline code okay so scripting they would like more scripting to be an early in the process okay so and then there's a couple of things interfaces so 2tp and J interface okay which I'm not going to say but they want more connections with scripting kinds of things okay ah types static types Dynamic attacks maybe they want static types uh erlang is dynamically typed so it's strong but Dynamic which means that the variables are not types are not declared but there's a tool called dialyzer which is able to do kind of a static analysis okay so they maybe want more of that they also want more Dynamic features so they want something called monkey patching so monkey patching means you can test the code that depends on global settings so you change so the code is there and it you can tell you have to modify these global settings to make this code to test it but you want to do that only for the test you don't want actually to permanently modify those Google settings so that is a technique called monkey patching okay so they want stuff like that okay so they basically want things like that of course gender learning curve well they said in Airline very hard things are possible but very easy things are not easy moving to Ireland is a big commitment so they want more better guides better documentation improved syntax actually this must may be solved since 2011.

[01:35:56]because now there is another language of elixir so then we gotta make a picture here so airline has a virtual machine called the B and Airline runs on top of that but since a couple of years there is another language called Elixir which was invented running on the same virtual machine so these two are interoperable but Elixir is more like standard uh python Ruby kind of syntax and actually there's a large Elixir community so the learning curve problem is kind of solved the airline a lot of Airline people using the beam using fault tolerance are using elixir okay that in addition to Airlines okay so that's something that they want okay what do they love about our lives it's a little nice cute cat pretty much everything else effortless concurrency it is the complete solution for hard concurrent problems this is the industry Gap it says it it's also open source everything is visible we can look under the hood play around you can do remote shells easily lifesaver robust and reliable it's incredibly robust in fact no other system is so robust they have these Game servers running for months without crashing that's very normal part for the course means it's normal so they like that a lot of course they complain also Black Ops 24-hour stats okay who said gamers are lazy for the past 24 hours all the players in Black Ops have collectively killed more than the entire population of the USA shot 28 million enemies in the hand so this kind of Statistics is maintained in in uh they shelled out 16 billion euros in weapons they stabbed two million enemies in the back okay 24 hours of pass okay so the conclusion Airline helps make tens of millions of levers happier across the world and if gamers are happy then we are happy so demonware love Airline and then they're hiring okay so this was a this was a keynote talk at an airline conference so you have a big Auditorium and there's like 500 Airline developers there all doing Airline stuff so of course they are and just companies are always interested successful companies are always interested in hiring okay so there you have a Industrial an industrial view on Paramount so the guy is very honest now the good and the bad but it lets them make very complex concurrent robust system okay so hopefully when you guys later go into industry maybe you take a look at something like Airline but not just stick with a C plus plus or Java okay okay let me let me stop there