Modern Linux C++ Debugging Tools: Internals & Usage | ACCU Talk

[00:00:11]okay i think we can get started so uh welcome welcome everyone uh we're going to talk about uh debugging tools and like how they really work so i'm just going to start by setting a little bit of context why why i think this is important and really i mean i think debugging is central to programming right in fact programming really is debug there are two there's programming is made of two parts there's bugging and debugging right there's typing it in and making it work like you cannot be a good programmer without being good at debugging and i would say also you cannot be good at debugging without being able to take advantage of the tools at your disposal right now of course that is not to say well obviously we don't need to use all of the tools all of the time but there are certain times when certain tools come in really really useful and all else being equal one programmer uh you know isn't able to make use of the tools available another one is well as being equal the one who is able is going to be the better programmer right programming is also models we got i mean it is you know what ultimately comes down to you know the the transistors on the chip you know it is impossible for a human to uh to think uh in in sufficient detail you know to be to be able to cope uh with what's really going on in the computer right so it's all it's a so if you're if you're a java programmer you know you've got this model of the jvm and you don't have to worry about you know memory management right but you kind of do if you want to be a good java programmer you need to know that this thing called garbage collection happens you need to know that there's this mark and sweep algorithm and sometimes a thing will pause well that goes up and does its thing and you know in fact like it doesn't completely take care of it if you've got uh references to objects you know that are still live then those objects won't get tidied away by the garbage collector so you can totally still leak in java right um if you're a c programmer and mostly i think this is you know and see we're talking cnc plus plus programming in in this talk um but you know that's a level um but you know even doesn't matter you you have this model um uh that that you kind of you the c model that you're programming to but but sometimes you need to know what's going on underneath in fact even if you're writing an assembly code you know underneath the x86 if you're writing x86 assembly code the the it's micro ops right those instructions are made from micro ups and if you want to make it you know super fast or whatever you might find that these three instructions actually you know run a lot slower than these other set these other a lot of 10 instructions to do the same thing because of the micro ups it boils down to so 90 whatever percent of the time you don't need to think about this you live in the model right but sometimes things go weird and you need to the model breaks down they say or no no model is perfect some models are useful right and you need to know where the boundaries are and you need to know so at least at a conceptual level kind of what's going on uh underneath that that model that you have and so that you combine the two with the tools right the tools give you a kind of model when i'm stepping through the code in gdb you know i have this kind of conceptual model and i'm stepping a line at a time but what's really going on 90 plus percent of the time i don't need to worry about that i just like accept the model that gdp gives me but sometimes like which it will happen and uh in fact then you can find yourself really led astray if you don't understand conceptually what's going on with these tools um and uh yeah and that's i think one of the reasons why some uh programmers kind of assume all these tools right and just like print just printf the whole time because you know one too many times in their life the tools kind of led them down the garden path and actually took them away from the right answer so that's why i think this talk is important probably preaching to the choir a little bit because you're here listening to me but hopefully that will uh convince you to stay so so really well debugging what's another way of saying debugging is thinking what happened right i had again my mental model of what my program was going to do and reality is diverged from my expectations and i need to figure out where that reality diverge that's what we mean by debugging obviously so let's look at the tools available uh to help us answer these questions this list of course is not exhaustive but i kind of categorized things into um four well sort of two buckets of two right so at the top we've got the kind of the debuggers um what we think of as a traditional i mean all of these tools are debuggers in the sense that they're used to remove bugs but there's what we think of as the debugger right typically gdb maybe lldb they're kind of the same for the purposes of this talk um you know there's there's also s trace which i've classed um alongside with those debuggers now so it's really just what in my program uh you know what's happened what what's what how can i see what my program is doing as it goes um time travel debuggers we'll look at those a little bit um you know for those of you who know me that obviously that's the subject kind of close to my heart um it gives us the the the the control over time and then the other the second bucket of two is these checkers so the debuggers really are just generic program comprehension tools i've got this bunch of code it's doing something let me understand what it's really doing right um it's not actually looking for the presence of any particular bug or other right in fact sometimes they're useful just to get your head around how some complex piece of code works you might not even be actually debugging to get used to these things the other but the other the other two are the checkers right and these are looking for instances of certain classes of bug i mean a buffer overrun is the classic right but of course there are all kinds you know race conditions deadlocks memory leaks all kinds of things right and and and um uh you know it's kind of like there's only what is it there's only seven different stories or whatever they say and it's probably only you know most bugs kind of fall into one of these categories so these checkers can just look for instances of those those anti-patterns um and uh and and tell you when you're done i mean there's two kinds it's dynamic so it's a running program and static just looks over the source code and finds the you know where they might be and again pros and cons right so the dynamic ones um you've got to actually trigger the bug if the buffer overflow is kind of hypothetical then most of these dynamic checkers aren't going to find it they're only going to spot the buffer overflow when it actually happens um so maybe you've got a vulnerability but if you're you know not under test while you're running in these in these in these checkers you're not going to see it whereas the static checkers will analyze the source code and kind of tell you you know what what might happen so i'm delighted in this talk to be joined by dewang from coverity from from synopsis and the coverity people who's going to talk about the static checkers because he knows way more about that than i do but that's kind of how i would categorize these different things so to recap you know the debugger is what is my program doing one way you might think about that is you know there was this there was this cinematography technique that was made popular it really made popular thing was the matrix and then got used all the time in that kind of early 2000s period where they were trying to freeze frame you know that and and then the camera would like pan around while time is frozen right so the bullets in mid-air and then the camera would would come around and you'd see it from all kinds of angles and that's kind of what the debugger does right it stops the program and you can see everything and you can maybe move it on one frame or you can just continue the action and pause again and have a look around so just what my program i'm doing the time travel debuggers tell you what your program did right gives you control over time so i go back to that metaphor it's like i'm in the editing studio and i can wind the tape back and forth to like when the bullet was fired but i can then stop and look all around at any angle from that point right so complete visibility of any point in the program's history so incredibly powerful um and then we've got the we're going to checkers what are we saying with the dynamic checker we're really saying did this thing happen did a buffer overflow happen at any point and if it did i want to want you to tell me exactly where it did right because of trouble with you know one of the things that makes bugs hard is how long elapses between the bug itself and you noticing the assertion failure or you know the segful or the bad result being produced and the longer that that gap is the harder the bug is to fix right um and so the sanitizers and valgrind and these dynamic checkers will just go right there it is that's where your buffer overflow happened straight down to that line of code very useful static checker could this thing happen right could the program possibly do this thing now one of the problems with that is you've got this real problem with with you know i mean many programs could do all kinds of things in theory right but in practice will they really and that's kind of what really you know separates the different kind of static checkers and and kind of gives you the value on on those on those checkers okay so that's kind of the context we're going to look at some of all of this stuff today and uh let's start though with that with with the debugger with good old gdp let's start to look down now how how does that actually work well um so here's my you know super fancy uh diagram i'm not very good at drawing these things so you just have to put up with it anyway so gdb controls the process being debugged sometimes sometimes in the gdp documentation they call it the inferior um controls the inferior through this thing called ptrace right which is a kernel api specifically for debugging it allows one process to go and you know monkey around with another and and inspect its state and actually even change a state as well um and then and then gdp will find out when the program has done something interesting by receiving a signal um so uh let's uh let's just go through this just before i go into that so let's let's have a look let's let's look at what we're talking about here so uh i think i've got hello world program here some where yeah there we go so i'm going to do g3 always use dash g3 we'll talk about why in a bit but i think that's one that's a very useful thing to be doing unless you're super short on space or something i'm going to compile my program i'm going to run it in in gdb and probably know all of this right so um here i am debugging uh my my program if i control z from that from ps i can see my processor right so here's my gtp process and here's the process that i'm debugging now if i uh uh if we go cat proc that pid i liked it when in the old days when pits were shorter and you could reasonably type them in um status we'll see this line right look at that that's the tracer pit that's the period that is p tracing a dot out and sure enough one six one seven one thirty that's my gdp right so gdp has connected to my process over p trace and also note this process is in this state t tracing stop right so what does that mean well let's come come back to this little diagram so uh oh sorry i've got claire and i out of steps let's talk about we'll talk we'll come back to the tracing stop in in a moment so um just let's just just back up a little bit and talk about signals right so i said that tracer gets notification of interesting things that the tracee has done by signals so think about linux and unix signals here so when a process receives a signal the kernel will do kind of the following a little bit of a simplification it's a model but it's it'll do so it might um well most you kind of think about this terminate the process most most signals by default will terminate the process not all but most will some of them so sig in for example just terminate the process some of them zigzag v will cause it to dump core maybe like if it's so configured um some signal might be ignored the program i've decided to ignore signals it might go into a stop so if you send six stop um on some other t-stop and which is the terminal stop and then t10 and ct doubt that will also stop suspend uh the process that receives at least by default that they can accept for six stops they can all be overridden uh maybe this program's installed a signal handle and signal handler might run or if it's being p-traced you could go into this tracing stop that we just saw right so if the process receives a signal and it's being retraced it will go into a tracing stop and the process that is tracing it will be notified of that signal signals with the exception of sig kill which is a bit special will never actually like directly kind of hit well they might they never hit the the the debuggie they are the tray c they are intercepted by the colonel doesn't send them to the tracy it puts the tracy in a tracing stop and and notifies the tracer so in our little geb thing we just looked at right so we got a hello world program being debugged by gdb uh in this little diagram doing red uh boxes mean that it's stopped it's in a tracing stop so i'm at the gdp prompt and then when i type continue and the gdp prompt it's going to issue a p-trace cont call and that will wake up the hello world process and up it will go it will run um and uh until um and it'll just keep running until at some point it's gonna receive the signal um or or or terminate um and in either of those cases uh what happens is uh the the signal notification will go back to the tracer and via a weight pit called gtp will be notified that something interesting has happened usually i've received a signal or possibly the process has terminated um and it's all signals right so uh we've you know the only way that so when you type continue at gdb and it runs uh the the the it runs the inferior contin resumes the inferior um the only way you're going to get back to that gdp prompt is when the inferior receives a signal or if it terminates um so when uh when when it hits a break point actually that's that that's a sick trap um when it and and and the process will go to the trap except the process doesn't get the trigger the tracer sees the signal when you do control c at least if you run the like i did just in that little example if i run the process from within gdb um it doesn't do anything special actually it's just uh they happen to be uh uh well gdp does make the process when you type run gtp will make the process the uh controlling uh the foreground process of the terminal so that when you do control c all of the processes in that process group which you adjust the db they'll all get sigint right except they don't actually get sticky because it's being detraced so it goes to uh gdb and then gdp will by default at least not send that um that signal in um so so here's essentially what happens right so if this is what the kernel does so if the sig if the signal is sick kill then that doesn't matter right doesn't matter if you're being p-traced it doesn't matter if you try to install a signal handler i mean you can't it doesn't matter if you tried to mask it you can't sit kill protest there that's it right um almost as kind of uncontrollable as six stop and that will suspend the process again you can't must you know this the process cannot mask it you cannot install a signal handler for it these two these two signals are special and they will always do the same thing um otherwise if the signal is blocked masked then it just gets marked the kernel just marks it pending and continues this is true even um uh uh even in the case that you uh you're debugging it right even if you um have a tracer um uh the block signals just and gdp will not be aware if the program receives a signal that it's masked gdp doesn't get to know about it until such time as that signal is unmasked um there are some exceptions to do with hardware signals so if you masked actually i should that maybe is a little interesting thing that we could talk about if you if you've masked uh say six segment for example and then you said v maybe you've got a seg or you know maybe you've got to say v handler and the handler runs with cpmask and then the handler segments um then uh the kernel will uh in that case the kernel will replace the handler with the default handler and unmask the signal and then and then gtp will be aware of it and that in that instance but generally if you mask signal the the debugger doesn't get to see it um and then and then if it's being traced right if that tracer pit is non-zero then the process goes into a tracing stop and there's a weight pit notification that goes to the tracer so gdp can know that something interesting has happened um or if the signal's ignored then we can just ignore it if there's a handler run the handler but if you're being p-traced like you don't get to these second two else's right um and finally none of them then terminated process and you don't get these last three elses and if you if you're being traced it just i think the process goes into a tracing stop and the debugger gets uh gets to see that's happened so let's look at like some of whoops i didn't actually need to do that uh let's do that again uh uh so if we look at some of um uh let's look something like what that means so come back here so uh i've got some example i've got an example code here with um the masks um masks uh sigit right um so uh this this this basically you know it's this program very simple uh it does sig prop mask to uh mask uh sigint um and then it just sort of goes into a loop feather just just just just sleeping right so if i run that so dash g3 always dash g3 and then i run this in gdb run so normally i would be i would expect to be able to control c and interrupt it right but that doesn't happen i've got no way to get control of that that's a bit of a bummer um that's uh or if i could do i did ctrl z there and it got a different signal it got it got the sig terminal stop and now i got back um to the prompt another way i could do it is uh actually to uh i've probably got two gdps i don't think i've killed um the unknown if i want to get good so uh a dot out so i can kill so here it is look it's still running right i can kill six stop uh to that um actually i can probably do this contact with this max stop out and voila look it's received it's received the six stop right um okay uh so um yeah so if your program is masking signals and you know sometimes programs do right they're quite common to mask signals and do something you're trying to run that inside the debugger you're gonna find you know damn it uh you know my control c doesn't work um but the good news is they can't mask six stop right programs cannot mask six top so i can always fire stick stop at it and get control back in fact internally gdp will do that use that as well so you know if i've got a multi-threaded program one of the programs receives a signal it's a break point or whatever but what one of the threads hits a break point or whatever then gdp will actually go and fire sig stop to all of the other threads in that process all under the hood you don't see this hoover them all up and then return to the prompt safe in the knowledge there's no way those signals can be masked or have a signal handler or anything okay so we went through this sorry we have to go through so again so i'm not going to go through the p trace interface because it's quite broad and it's quite horrible as well actually i i it's a really crummy uh interface it's a single call and then it has this op code as the first argument i always think that's an anti-pattern if you have an op code as the first argument you probably mean you have a whole bunch of different uh uh functions and then you get like some kind of type safety and and the rest of it anyway petraeus doesn't um i think it's inherited certainly inherited from at least solaris maybe probably i think it's probably i know i think it's even the original unix um and and so sort of a legacy api obviously not many people need to use it and if you're fortunate enough to be writing a debugger but um but it's good to know kind of what it's doing um there are loads of them right so i'm certainly not going to go through all of them and this isn't even a complete list that's quite quite an extensive uh interface but i am going to go through these three just very briefly petraeus can't pre-straight single step and p-trace cisco uh um so p-trace can't just gonna resume the the you can you issue it when the tray c is in a uh is in a tracing stop and you issue a p-trace contact and off it will go right and it's run until the next time it encounters a signal if it's already got signals pending it won't make any progress it will stop immediately um or uh i can do petra single set not all architectures support this this does require that you have um architectural support in the hardware most modern cpus do x86 certainly does modern arm cpus do older ones maybe don't um but anyway what it does is to it'll run exactly one instruction one cpu instruction and then and then it'll stop with a sick trap um and so that's that's you know you can see you can imagine how that gets used sometimes for you know next or step type uh commands although actually often what gtp will do when you do it next is um um put a break point um and uh the next line and continue um um at least it will when it will actually do a repeated single step i think unless it leaves the function in which case it will continue until it returns and then there's ptrace syscall which it doesn't have any architectural support it's entirely a software thing and and that will run until the next time that the tray c issues assist a cisco and then the kernel will fabricate a sig trap and it will stop there so that's quite useful right particularly for tools like s trace so you actually get two sig traps um persist called if you use p tray cisco you get one on cisco entry and another one on school exit is actually configurable but that's usually what you get uh you can if you happen to be writing one of these you can actually distinguish between cisco entry and cisco exit as in the way in on the way out by looking at aax bit of a hack but if it contains minus 38 which is you know cis then it means it's a um at least most of the time doesn't quite happen all the time you are just expected to keep control keep keep track of that yourself so here's a really kind of i'm not even trying to compile this but this is basically how uh s trace works right it's a really really simple airstream not going to have you can know of course you can attach to running programs the rest of it uh or straits or however you like to pronounce it um so it could go gonna fork the tray c the program is being traced and then the tracy issues this trace me um petraeus call which sort of causes its parent to be to trace it like and strangely and that's quite quite surprising actually sometimes in some ways but yeah the tracer doesn't get any choice it's just like the the child can decide that the parent is going to trace it and then do x exec and exec will actually rather you whatever is happening exactly cause um if it's being traced will cause a sick trap um to be received and then in the parent we just like go forever uh weight pit um on uh on the pit that stopped and then in that status just sort of crummy wait interface but you get to see what's happened so if it's not we've stopped which could be a signaled or or exited but basically that means the proto the the process is terminated so we're done um otherwise i haven't shown the implementation of this but you can imagine how it might work right i'm going to go and show the cisco um and uh you know print the register get the registers to see what the arguments for the cisco were et cetera and then off i go again be traced so the first justice will be exec and then we go again with the p trace cisco and uh and this will actually end up with two lines being emitted per cisco um one on the way in and one on the way out um but that's fine um notice this thing at the bottom right so you pass in the signal um from the p-trace call and that's the only way that the only way that a traced process will actually receive a signal is if the tracer feeds it back in from its p-trace con p-tray cisco single step right so the tracer in s trace or gdb has complete control over what signals the trace c will actually get um with the exception sig kill um so that's kind of how i'm gonna that's uh hopefully that's kind of all reasonably clear i'm going to let's just um i've got a feeling i um uh let's just do one more thing on signals because it's kind of this is kind of um kind of screwy but um kind of an example of what i'm talking about where you can get led astray if you don't really have a good understanding of what's going on underneath so um we looked at my uh mousekent uh um uh little program just now i've got another one very similar that mask trap right that's pretty much what you think so i'm explicitly raising a sig trap now this is kind of interesting because if you raise a stick trap while you've masked sig crap actually uh you you're still going to receive it and then gdp tries i think what's going on i've got to check the source code i think what goes on is gcc tries to be a bit clever here so uh ggb sorry tries to be a bit clever here so um so if i i'm gonna start the program i'm gonna go into two mode um so if i um sort of do it in the normal way uh if i just uh continue my program um it just runs through to the end right it's masked to trap it raised to crap it didn't see it and gdp kind of works out what was supposed to happen but things were supposed to happen if i start again um and if i start doing uh next and if i now get this right if i put a break point i think i'll put a break point on line 18 and continue pro gdb gets very confused because it kind of sees the sig trap uh it kind of knows that there's a break point roughly in the area but you might if you're eagle-eyed you know just the gdp would normally say i've hit a break point i've hit break point number two um because that's what that's what uh sorry i've hit break point number three because that's what i said there but it kind of got very confused so it got the sig trap it wasn't quite sure whether there's a break point or not um and i think if i continue from there um oh then i get a seg fee so poor old ggb's got really confused now um i think it's not quite managed to step off the break point or something correctly so this is something that um uh is perhaps worth just going into a tiny bit i think we've got people doing okay for time um yeah i'll just go into this tiny bit while we're here so as you may or may not know when when you set a breakpoint um gdb writes a breakpoint instruction into into the text right into the executable um memory um a break point instruction on x86 is oxcc um that's the op code single byte op code because that's kind of important right because x86 is a variable lens instruction set some instructions are only one byte um if the breakpoint instruction were more than one byte long and i had to set breakpoint on two adjacent instructions one byte apart that would be uh problematic right so um actually x86 designers kind of thought of this right right back in the original x86 days i made sure that there was a special uh uh trap instruction lxcc um written in three um but it's especially if you write int four into five that would generate interrupts on those um uh of those interrupts and they're two but it's two bytes the int op code and the and the operand and three is special for debuggers so gb will write oxcc into like whatever address the uh the the um the where the break point resides you can't see it right if i um let's restart this program um and so if i like uh so if i put a break point um let's put a breakpoint in line 13 and run to it now if i if i look at the disassembly i don't see that in the in three instruction there that's the original code right because what gdp will do is write in all of the break points if only before it continues the program and then when the program stops it'll put it all back to what it was so it's kind of transparent difficult to kind of see what's going on but what happened just now when i got that seg fee i haven't looked into this detail but i'm pretty confident that what's going on is right so so when when you when you've hit a break point when you continue from that break point you don't want to just hit the break point again right it's kind of a special case right so if i continue from a breakpoint what gdp will do is single step to repeated a single step without setting the break points and it'll step over over that instruction then put all the break points back then continue right um when it hits the break point in um in x86 at least it does depend on the architecture it will be it will stop with the program counter at the instruction one after the oxcc instructions so gdb also when it hits a break one has to unwind the instruction and what i think we saw when we got that segfie was that gdp was getting that wrong it got confused about whether it hit a breakpoint or not it got it either subtracted one from the program counter when it shouldn't have or didn't when it should have and then you're in all sorts of trouble right because x86 is a very variable instruction set and you can start executing from the middle of instructions and it gets very confusing um so what's the what's the moral of this story i i you know using gdb on programs that you sig trap for something is um is a bit dodgy basically uh best avoided if you can um and uh and and best avoid putting any break points where you may also hit the sick trap and if you do just be aware that like everything might not be quite as you wanted it to be okay um let's talk a little bit about cisco restart so um uh here's a really simple program right it's got a an empty null signal handler um and it's going to touch that thing to the single arm this alarm function if you don't know alarm um it just uh uh means that in five seconds after doing alarm the process is going to receive a signal then it does this read that's gonna block and um i mean if you type anything at it and then five seconds later the sig alarm's going to come in the handle is going to run and read will return the intro right that's how signals uh in unix land work so if i so i've got that pro almost that program it's slightly uh i'll show you that here um it is uh the integer so i've actually done the sig action rather than the signal with a little bit more control so always dash g3 the intro so if i run that and i've actually changed it to alarm 2 because uh waiting five seconds is a surprisingly long time in a demo so uh run the program on to finished good but if i type something at it then it aborts because read didn't return the intro it returned what i've read right um nothing sort of too surprising uh if i run it in in g d b uh i'm gonna now one thing i need to do here is um i should have said this before actually let me show you now so um the remember we said that um no okay so uh remember i remember we said that that um with the exception sig kill uh the only way signals actually hit the the program being debunked for tracy is if the debugger so decides if it sends them in and and this setting in gdp controls like what it will do for each of the signals most of them you see this yes yes yes right so you do what you kind of would expect so it stops the program it prints a little message to say that you received let's say you know this is unfortunately common one we all know and love it i'm sure so it'll print a message program receive six segment uh after it stops the program and then if you continue uh it'll pass that segment back in right and if there's a sexy handler that will run if there isn't then it's the end of uh you know that your program is dead um uh but no they're not all yes yes yes uh so sigint like for the reasons we discussed earlier on right it'll stop the program it'll tell you it's receivers again when you continue it won't pass that sign into the program at least by default so if you're debugging a program that has a sigint handler and depends on that sigint handler running you're going to get confused and surprised and disappointed when that doesn't happen right so i can change that i can go uh handle sigint stop print pass it will warn me you'll say this is kind of a bit of a special one are you sure you want to change that i think actually probably i don't want to change that um and just like because i don't know why i guess the gdp designers at some point decided that um sig alarm was kind of common enough that you probably didn't want to know about it uh i think maybe actually early versions of the sleep lipsy sleep call then maybe even modern versions too i don't know but certainly early versions of liberty's sleep would use alarm or implemented in terms of alarm and um you know perhaps it'd be confusing if you're debugging and you've got all these cigars kind of popping up all over the place so it kind of hides them from you unless you ask to see them again you know if you don't understand what's going on then then you know this can confuse you so i do care about the single alarms in my little program so i'm going to handle sig alarm stop print pass and it's already started my program didn't know yeah here i am just going to two mode uh so uh i'm just going to continue the program um and after two seconds there we go got my signal on now um and i can continue and it's going to run the handler and then it's going to return and everything's fine right so at this point i didn't my model of what's going on kind of all makes sense like if i just run that again you'll remember um how i've set the to handle the signals so it stops when i'm kind of inside the reed cisco right um the kernel is in the read when it receives the sig alarm it kind of looks like i'm in the read and it's right after the cisco instruction look right um and uh if i continue it just actually continue that read um although um and the read will return uh the intro because that's what it does right so actually just go once again and um so here we are we've got the sig alarm and uh i'm gonna go out of uh two mode just might get us wrong so i'm right after the system call if i s step i step instruction see what happens next i'm up into that signal handler right signal handler doesn't do very much but i can run that and if i step off the end of that and get this little and here i am at the end of my read and i can see um it's returned uh actually it's returned something that is surprising why that's not what we expected right why did it return two seven six um well it hasn't actually completed that line yet i guess that's just where it was there run it again and okay yes return minus one and and it was um four which i happen to know is the intra all good right it's the model's kind of holding up at this point but we'll start we're going to go um we're going to eat the so the blue pill or the red pill in the matrix i can't remember whichever pill lets you know what's really going on um so um uh i think it's a red pill anyway um uh so when the read happens okay they're gonna be gonna cool down this little lower block is the kernel the upper block is my program so it cools down into the kernel the kernel itself will have a whole bunch of function calls that happen right and end up down in the depths of the file system although this is a read from the term and also in the terminal handler and and then it blocks right now when my signal comes in what happens is this then starts to unwind so my sig alarm happens right and and the kernel will start to unwind back up but it doesn't return back to where it would have it returns not the not straight after the read but into the signal handler right and when the signal handler returns actually it returns via the kernel there's a little system called we'll see this in just a minute um it does a sig return so it can unwind any kernel state and everything and it goes back to there and it returns the intro right so we can start to see uh because i spent what but but when when sorry when it's being traced actually you know there's this essay restart if i program my signal handler with essay restart it will cause the read to resume when the signal handler returns what the kernel actually will do there is um this is transparent the user space will never see this but it'll actually return from that read after the signal handler's run it will return e restart cis not e intra the intro is four he restarts this is 512 it's not a regular um actually the kernel returns negative of these negative he restarts this rather than negatively intro and then um the kernel is kind of clever and when it's resuming from the from a system called where he restarts this will actually restart the system call it in fact what it does on x86 is subtract two from the program counter to reissue this is called instruction and we can see this uh so uh i'm going to um i'm going to run it again and there's a see that takes a long time doesn't make a good job by changing it from five to two um so here we are remember we've uh received the signal the right after that's his call instruction if i look at rax which is the register that contains the return result it doesn't contain -4 it contains um minus 512. um and because in fact if i change my i'm going to change my uh program so that um [Music] what was it which were ian intra let's make a bit longer so i get a bit more kind of um [Music] i continue um it get ill and i can control c same deal i can continue it goes back into that this is called right because back into the read cisco and when i continue uh again it's because it's got that magic 512 and so the continual calls that to school to restart now you should never the user space should never see that negative 512 actually there's a few of them as these wrestlers that's this you restart block you restart no hand and so some of them things like um plot nano sleep some of this and some of the other sleep functions will uh adjust the timeout so that when you restart them you know otherwise you might have a sleep with a timeout of five seconds or a poll or something with a time out of five seconds and um an alarm that's going off every second and if it restarted uh properly restarted that poll with a five second sleep every time after seeing the sig alarm then the pole would never never time out right because it would always go back to five second time out so and actually the kernel is quite clever it will restart the poll or then actually with an adjusted uh timeout and um uh and so you could that's another it knows whether to do that with another sort of code so what is this why does this matter well if i'm looking at return results and cisco's um i might get kind of confused right i have no idea what -12 is that looks wrong also there are bugs um in in in gdb and in some versions of p trace where these leak so user space should never see that when you continue the kernel is supposed to transparently just restart the syscall so the user space never gets a chance to see that return code um doesn't always happen that way um and uh in increasingly rare circumstances you that that can leak particularly though likely to happen on red hat enterprise six um so um that kernel they back ported this thing called utrace so remember i said that p trace is kind of a sucky interface and it is and somebody some you know decent people said right enough of this let's let's design it properly they made this kind of modular thing called utracing the kernel with the ptrace kind of compatibility layer onto it uh this was being used a lot by the d-trace stuff um system tap and so um and red hat wanted that in so back ported it all into rail six um and it never really worked properly it's always been a bit unreliable and uh in the end it got dropped and they did things differently and so it's all just regular petraeus so debug so using you know um using strace or or gdb or anything that uses p trace on rail six it's a little long in the tooth now but it's still a lot out there you can see this is quite there's quite a few bugs actually um very you know things are signal heavy and very threaded you can see this he restarts this will leak sometimes there are other problems with deadlocks and things not terminating than they should um so you know if you can't always avoid it of course you might be investigating an issue that only happens on rail six or that's the machine you happen to be on but if you can avoid it um with with gdb i would say avoid um kind of covered all this stuff actually um yeah okay there's also debug registers on x86 which you use for watch points as well um so you can actually do it without writing into the code and that could be useful um but there's a limited number of those right only four hardware watchpoint uh uh registers um okay let's get on to um let's let's get on to uh just this dwarf info right which is so when we've talked about peach race and how that can the process goes just to to tracing stop and pitch around and you can do pictures get regs and pictures peak to see the memory in the registers but of course gdp um and the same by the way it's going to apply to valgrind and address sanitizers and things they don't tend to talk about program counselors because that's not very useful right they give us lines of code and we can look at you know we can look at we don't have to look at memory addresses we can say print the value of foo um and uh and and this is all done by the dwarf info which essentially will take a program it'll say for every program counter what it maps to a source uh source file line and even column actually and it'll give information about what the types are and the variable names and everything and so the debugger can pass this board information to see kind of what's going on um and we're gonna have a little bit of a look at that some of that as well so um uh uh and we're gonna talk about something that we've probably um all seen uh only too many times so if i go gcc das g3 all those three and so this is the dash g3 it actually produces more better late and later version of dwarf um and puts more stuff in there and it's much better like inline um even um it can even do macros um so if you've got p princess and macros that debugger can now understand them if you do dash g3 and inlines work a lot better and generally it's all better um and but i'm going to give it some optimization and um okay so now here we are uh now prince who probably all seen that message right how annoying is that message gdp can't tell me what the value of foo is because it's an optimizing compiler in fact if we look at the source code um what's this the source code the assembly what's this doing um well it's uh okay got some sort of preamble stuff here calls random okay fine um put some stuff into registers like at no point does it load the foo it's memory foo is entirely held in a register she would kind of hope right with some optimization so for old gdp how's it going to know well actually because i partly just does this anyway but particularly because i did it with dash g3 it does know when it says foo is optimized now what i really ought to say i think it'd be much better if it said it's not live yet i'm at the beginning of line seven right so food doesn't exist um it's not been optimized away it just doesn't like exist at this point if i stepped forward um uh wait that's not what i expected i don't know what happened there and next next weird okay um that's not what i expected to happen 3.098 uh and now print food now it's got it right now what even though it doesn't exist in memory right it's not if we look at this remember the the the disassembly um i'm sure i tried this earlier today it worked really good anyway okay all right so um and i can have a look i can look at that um with uh redelf uh debug dump of a dot out and we can start to see what it tells me um so here's and there's a variable called foo right and uh it's in file number one which is the only file in this source code and that's the line where it's declared and the column and it's got this locations list and it says it's location uh a lxa and uh if i do uh debug damp equals lock we can start to see um some of the details of what it tells me so in fact let me so i can go side by side with gtp and we can take a look at the disassembly so it's telling me that uh this is that remember this location a foo and um it gives me some offsets in the file so it says these will be offset by page numbers but but it ends basically 108 109.8 in the file in the text file which is um which is here so it says from here it's kind of live and uh it's kind of clever it says it's in uh it's in the rx and then actually if i look at my disassembly um it's in uh it's in rx and uh and and and edx as well and it tells me yeah from from this offset 109c it also exists in in rdx but also look it doesn't exist in eax anymore from here onwards right because because the value of foo is just in edx um and and and but now it's saying from uh from this offset when i say 1086 it is now uh i can see it back in in in rax right and there i can see it's uh this lea is actually just adding one uh to eax because that's what the program does so um yeah so nicely you know even though even though um i don't know why it's behaving differently today it's a demo effect doesn't it but um look at that i i didn't do anything different that's mad all right okay for some reason now it's working um and so now foo is live and i can see the value and notice my print um increments foo and it uh it it it's a post increment and i can see it's it's still great there so it's kind of almost got not quite got it right because it said it was uh 384 so it kind of done the increment before um at that point but um we can actually see a bit clearer if we step through it but they're concerned about time because i'm going to give to wang a chance to talk so i think we'll move on um from there but anyway the point is that you know there are bits there are times in the variable is live um and modern modern versions of gcc with dash g3 in particular uh will will do um a better job than you might think [Music] um uh with with that stuff right so that's all um that i want to get in i'm gonna oh gosh we are running out of time right um it's very quickly on stacks um and frame pointers and things right so um optimized code is often going to emit frame pointers but but we can still find out um where the stack frames and things are very quickly what i mean by frame pointers i just think just just in case we're not familiar so this is a here's an x86 stack which grows down my program does some stuff uh the stack pointer moves down because it's like done some stuff on the stack it does a bit more stuff the platform was done does a bit more stuff then i'm about to call a function so the return address is pushed onto the stack and the function is called and that function does more stuff and then it's going to fall a function so it's return address is pushed onto stack and it does some stuff right now uh if i'm compiling with uh a a very low optimization um my compiler might use uh frame pointers and right uh below every return address on the stack um it cause sort of point of the previous frame and it also spill burns a register frame point to register the points to stop so at all points that that frame pointer uh points to the the top of the frame and it can just a linked list right it can follow that and that is a pointer to the previous frame so by some pretty simple walking of the linked list and a recent check we can unwind the stack and uh so when i'm in gdp i can type back trace um or you know some of the things like um some of the tools that we're not going to talk about today l-trace and things like that we'll we'll use this to be able to walk the stack but uh it burns a register right and registers are very valuable things so optimizing compilers um don't like to do that it also wastes a bit of stack space which probably doesn't matter if it might and it takes a bit of time just to mess about with the frame point on each function for an exit so optimizing polishing this is a bad idea and actually you don't need it um because if i know where my stack point like i can know that every program counter what the offset is from the stack pointer at that point in time to the stack frame so if we do this example again it says well when when it's here i know that the stack frame is whatever 12 bytes below where the current um stack pointer points and then as it goes we can just we can just keep knowing that right we can know for any line of code how far the top of the stack is top of that stack frame is from the stack pointer and uh this is this is this uh this is what they call um beautiful animation so cfa call uh what is it called frame addresses right and again this is in the dwarf information it allows us to walk the stack without having frame pointers and actually is used by exception handling uh and the exception handling stack unwinding code will will use this cfa and cfi information um uh uh so it actually gets used by runtime uh as well uh oh my gosh i need to go through this again right uh skip over that uh very quickly threads and so the debugger knows about threads for db is a library that you can link against debugger which is able then to figure out basic sort of companion library with the thread library so um most threads most of the programs on linux these days are using nptl um but not all right and you can have your own thread library and you know obviously programs like you know programming languages like go do different things and so you can have your own lib thread um which is kind of companion library to the uh to the threading library and that allows the debugger to kind of know stuff like thread local storage um which is important actually for erno is thread local a little plug here for our uh gdb watch point um series so you can go to the undo website and go to watchpoint and look at all kinds of nifty tips and tricks uh dog is not included um gosh we really are uh really going over time um right uh address sanitizers um just go quickly through this so um and valgrind so these are the dynamic checkers right and uh and and they work by essentially but they kind of work the same way they essentially give you a a map of um which addresses are valid and which are not um and uh you could like you might think well that's what the page table is for right um so why don't we just use that and you can in theory but that will really explode the space that you need if you're gonna have every single local object um you know an int on the stack or something in its own page uh that's gonna be very very expensive in terms of memory um and uh and also probably very bad for performance as well um because it will ruin the cache and everything so so um instead they kind of have this shadow memory which is a software version of page tables if you like and and and sort of this is data structure that for any address will tell you um whether or not um whether or not that that memory is valid uh the drastilitizer uses quite a clever technique to sort of really compress that down so it can its shadow shadow maps can be quite small i mean it does depend a bit but generally and an eighth of um of the memory that your um that your program is using um yet still do bite line stuff um and and they kind of could put these red zones um above and below right so why why why do we need red zones and uh and the shadow um and uh and the answer is well you know that otherwise how do i detect i can't really detect buffer overrun in that case right if i um if i don't have red zones um because uh the memory immediately after this malloc will also be valid just like it's different right so i put red zones before and after they're not a page typically some social bites or something and um and and and then we can detect uh buffer overruns most of forever runs if i if i overrun the buffer by a large enough amount i might step right over the red zone right so um the sanitize sanitizers and and valgrind won't catch everything um so this is kind of the so so address sanitizer is obviously it's a compile time thing and so if your code might have done uh this just got a dereferencing address um or writing reading or writing the address then the compiler generates code that is poisoned um error otherwise just do it and that is poisonous looking through those shadow maps and it's got quite clever optimized things in the overhead it's quite quite low um and you can also do it with uh with locals as well so address enterprise is better than valgrind in this respect because um it will put sort of red zones around locals um and um and uh and and can detect i can detect that whereas um uh valgrind won't know to do that because it's not a compile time thing valgrind works with uh compiled code so it's a big advantage right because maybe i don't have the source code and i can't recompile um and uh and so that is very useful there or maybe i want you know maybe the bug goes away right you can imagine you know alignment things and stuff just sort of disappearing when you start doing this so valgrind is very useful um in in that sense um but it does have the limitation that it's slow and that um it can't catch quite as much stuff all right um so having spent all that time um uh let's talk about time travel debugging completely different thing i'm not going to give a demo you can assume you know roughly what it is there was some stuff on it lunch to learn um but essentially just in some words right so we let you see any um any piece of program state so in the debugger we can step back a line not just forwards we can put a break point and run back or the really cool thing we can put a watch point and run back and do a reverse continue and not run back until that data changed so this means we can go back to any line of code that executed and see the complete program state which is pretty awesome um but also i mean in kind of more the traditional sense of the word actually if you think about it in terms of like that's a lot of data to store right um and clearly we can't you know obviously would be absurd to think about storing the full program state for every single instruction executes um so you think well how would we do this well maybe we can just do a bit like a sort of mpeg encoding we'll just store the differences right it's executes it's only going to touch a few bytes probably um so i can just kind of encode that and that shouldn't be too bad right but even that is get is very slow and consumes huge amounts of memory because you know we every instruction is going to change some state otherwise you've got unless it's an infinite loop right it must change at least some register state or some awesome uh or some um memory state or at least the program counter otherwise it's that's that's it's halted um and so uh every single instruction well how many instructions did you get well let's just do some really crude arithmetic gigahertz machine billion fractions per second that's like gigabytes per second right of state that's changing um and uh that's just that's just too much so instead and there are there are time travel developers earlier versions of time probably bugging that that just did that right and they lived with it and it's fine and it kind of works for small programs in fact gdp has an inbuilt process record thing that does exactly that and um if you've ever used it it's extremely slow the reason it's extremely slow is it's doing a betrayal single step for every single instruction um and so that is very very heavy weight um but also generates huge amounts of log because it's like gigabytes per second or like what would have been a second um so more modern time travel debugging or software failure replay things um the rr and live recorder or udb the undo debugger what they do is to recompute prior state right and that's much much more efficient and it's gonna based on the fundamental kind of well on the obstacle on the observation the computers are fundamentally deterministic things if i run a program multiple times every time you give it the same starting state and every time it gets the same set of inputs it will always do the same thing right this is why random numbers are difficult to generate well except you know the catch is a the program computers are completely deterministic except when they're not and of course they're not completely deterministic if they were to be kind of useless or very limited use i just run them once got my own so that's it and of course any you know there are multiple points of non-determinism so anytime we make a system call or a thread switch or get a signal um certainly asynchronous signal um certain non-deterministic constructions or accesses to share memory these will do different things same in program state same instruction might give different results um so uh what we what these what these technologies do is to store in an event log all the non-deterministic thing and then i want to go back a line we'll go back to the beginning or back to a snapshot play forwards like just before that line right and then effect it looks like i've gone back a line as long as we've recorded record time all the system calls we'll share memory accesses etc so that i can now replay them to make sure that the replay does exactly what it did in record time right um but you also need um you also need a very precise notion of time for this to work because let's say i'm stepping back through a loop um and uh you know it's a small you know it's a mem copy type loop um i want to step back one iteration of that one line within that loop i don't want to go back to that line five iterations previously or the first time around the loop and of course programs are all loops right if you don't have any loops then your program is going to terminate very very quickly um so i need for this to work i need to know when to stop and i'm replaying i don't know exactly when to stop i also need to know when to replay some of these asynchronous events it's an asynchronous signal i need to replay that at exactly the right time otherwise my program is going to do something uh different um so uh there are two ways to kind of do this with this rr and liverpool to do this in different ways so rr will use cpu performance counters to count the number of retired branches that is the number of branches that actually um executed and this is quite good because the hardware will do it quickly that makes it quite a simple um implementation um but on the downside uh they're not always available and the big killer is they're not available in most aws and other cloud machines you can get an aws bare metal or complete socket you can they'll allow them otherwise they don't basically multi-tenant cloud machines tend to turn them off um and uh and and they um uh so just work in those environments and uh and there are things you can't do particularly around shared memory um um so the other way you can do it with jet instrumentation and that's what live recorder does um now that does impose runtime overhead um it doesn't you're not as big as you might think but there is some um because we're doing more stuff in software there um but it doesn't mean we're not relying on these sort of quite exotic uh cpu features so just work wherever and it does mean that it gives you more flexibility to do things like uh deal with things like shared memory and weird kind of non-deterministic instructions otherwise it's difficult to uh trap um so that is um so i've rushed forward apologies for rushing through um that oops um uh i'm but um yeah poor timing but hopefully that was interesting and made sense i'm now gonna switch over to duang who's gonna talk uh uh all about uh static analysis thank you very much greg hey everyone all right uh static analysis so when you know greg take you through all these different events techniques for debugging right from a static analysis perspective i generally like to put it this way usually if you have found these anti-patterns then the question is is there a way to formalize and benefit your team by coding up some kind of analysis that catches this kind of uh problems in the future yeah so uh this portion of my talk is focused around giving you a little bit more fearful what static analysis is and uh walk you through a demo on what is like creating a static analysis check so a static analyzer you can think of it as a compiler you start out with the source code right and it will essentially compile your code and generate uh the semantic representations of your code so the abstract syntax tree and then you will compute the different flows that your program could take and that's a control flow graph and it will collect other properties um and and then what it will do is we'll go through these different paths and look for problems right so in today's demo i'll show you for example if you were to allocate memory right and then if you uh your function exits and you forget to de-allocate right then you're leaking memory yeah um so a good analyzer will give you annotations and tell you the paths it took to get to a problem and tell you exactly what the problem is right and then you can consume this kind of information inside your ide in command prompt or through a browser right um so here i am going to show you an example of a problem that a static analyzer is good for in terms of finding yeah so on the left we'll see here's a here's the example code test code where we make a call to malik yeah and here is a simple uh conditional expression where sometimes it will free that allocated memory and other times it won't right so when this function exits uh obviously you can potentially leak this memory yeah so um on the right hand side here what i what i have is an example uh checker and you know i'm a solutions architect synopsis i worked on the coverity product and this is our um domain-specific language called codexm and so coverity uh can out of the box find you tons and tons of issues but you can create your own analysis with it as well and this dsl is available with the product and so if we start out at the way bottom here you'll see that i have defined the checker named after me right and so this is how it looks like i'll say okay report for [Music] going through all functions in the code base where where the function uh path match the sequence right and now i'd say wherever there is a [Music] allocation anywhere downstream in the past where it's missing a call to free right and so if we look for the up okay okay so i'm gonna define um a uh a call to a function called malloc see the function identifier up here yeah and i'm gonna define it as a node step okay and then similarly i'm gonna uh define a pattern called call free and with identifier is a function free right and you'll notice there's some syntax here that i just says i want uh to make sure that the call to the this free function takes an argument yeah because we want we want it to be this form okay and then this next two um uh pattern is built on top of the above right and one thing that's really important i want to highlight here is uh um in this particular pattern what we want to say is um it's a pattern of this there's a call out up here that matches this malloc right but one important concept here is uh i want to point out is if you think about the call graph you want to line up the malloc with the free to the same address right you don't want uh the malik of some uh of some other memory uh to be paired up with free of some yeah another memory then then it's mismatched right so you want you always want to to make sure they're lined up so this key is used to express that so basically you can see now we just built a checker that would say every time this malek before it goes out of scope we need to free it yeah so let me switch over to demo okay so um here is a demo code and you will see in fact um the one i talked about just um just one slide ago uh is this example code here right but um i also want to share with you a couple of quick static analysis concepts yeah uh one of them is called aliasing so you can see that um i have allocated um two blocks of memory here yeah and then um you'll notice that i declare another uh variable name and then i assign it to this um one of these addresses yeah so that's aliasing so if i free this address right um so in our case coverity actually um can keep track of aliasing and and these kind of assignments for you for free right so we because we free this particular address but we have not freed this one right that checker we i showed you earlier should then be able to catch the case when this is being leaked okay um and of course uh and of course the con the concept with the control flow is um you notice in the checkered example code i did not explain i did not explicitly have to code up for looking for uh these different paths right so again with coverity it's kind of computed for us for free it just automatically knows that this is there's a branching condition so there's a path where uh this can be skipped and then there's another password you know it's not skipped yeah so let me run this this is our checker um that i showed you guys earlier okay so the way converity works is we do call build we specify an intermediate directory and then we call the compile so i'm in the microsoft visual studio environment so i'm gonna do let's go okay so now that not that so so okay there we are right so you can see that coverity has successfully picked up this compile yeah so now we're gonna run analysis on it i'm gonna disable all the default checkers and i am just going to run that particular checker that i want to run okay so you notice it's found those three instances of resource leaks yeah um so now uh you'll see okay so for the simple case um right uh there's an allocation function goes out of scope uh it's it's leaking this memory yeah and then here is the second one right uh it's found uh in this particular case um we have allocated this block of memory but this one was not freed yeah so this is aliasing okay [Music] and then finally um here we allocate there's a case where we do not go into um this blocker code to free it right then it leaks [Music] okay so hopefully that gives you a feel for uh stack static analysis i know that's uh really quick um the thing i want to tell you guys is uh if you guys think about if you do code reviews uh there's gonna be a lot of things about your um application that only you or your team will know about right um like so this uh this uh the sequence of allocating and you forget to de-allocate this kind of anti-pattern it's actually really common right if you think about this like uh if you if you have a program let's say um if a function returns a value and you forget to check this value in certain way well it's a sequence of function calls right so did you call this function finally or not right so it all belongs to this uh similar type of uh anti-pattern yeah so um all right so um greg i think uh that's gonna be my portion on static analysis and and uh maybe we can turn it over to uh q a yeah so there were uh there were some questions which kind of tried to answer in the chat but as we have a few minutes we can kind of go into maybe a little bit more more detail on them yeah so um uh so uh honey asked uh uh which command was which command was used to see the code while single well single step right so i think let's just look at so um i'm going to uh just give me a second here so that was gdp 2e mode t-u-i stands for text user interface um so what if i share and we can see that so um yeah so this was like when i was going uh so to start and then um uh i actually so yeah i can do i can do layout source um and that's pretty cool right i often say this is the uh scores the highest and the on the uh least known most useful gdb command you can also do it with control x a um i can also go uh [Music] layout asm and then i get the assembly code and i can step through that sis short step by and i can step through that i can actually go and have it come up i can go control x2 and i get both um source code and assembly and i can go control x2 again and i get my assembly and my registers um which is kind of neat um so let's see what other questions we were asked um so yeah so writing so gb writes he's in three instructions right when he's gonna put a breakpoint and question from mike was um uh when it's single steps could put the pistol there um in threes back and and and then and then i'm sorry but uh all the um normal instructions back and then the real code back um unpatches the threes and and steps over the break point how does that work with multi threading i tried to answer this in the chat but i realize now i think i answered a different question um so it will um uh it will step just the thread that has so p trace gives you control over each thread um so it will uh uh unpatch all the entries and in fact it unpatches all the entries before it gets back to the prompt um and then when you want to continue when you continue it'll single step that one thread over that one break point um instruction then put all the threes back then resume all of the threads um and then another question follow-up question from mike is how does that work if the replace instructions is a potentially long running syscall instruction so again i sort of tried to answer it but i think i probably answered a different question so what happens if you put a breakpoint on the syscall instruction and [Music] i don't actually know shall we try it why don't we try it uh i think this will be interesting uh so uh cat um intra it's latest i've only got i've got short enough time now that if all goes horribly wrong i just can then uh duck out without being too embarrassed so uh feeling very brave um something worse when it goes wrong in the middle of the talk right so um so this has got a nice long-running instruction so start so let's go control a control two so i get my uh assembly code so where's the fifth score so score is oh no it's inside the middle of the library so i'll continue until we uh stop in the cisco um right and so now oh look it was dangerous yeah okay tui mode with assembly is um maybe not the best thing to be doing because gtb just crashed that needs debugging uh all right let's do it just very instead so i'm just gonna run and then right so now here's my school instruction um so by the way gb by default will turn off address based randomizations this is quite useful so i can just remember that because i've gone past instruction but i can restart and put a break point on that um just to make sure that that is not talking nonsense and that really is this is cool uh yeah this is just cool so let's put a break point there so i can do that like this uh star and then the address another breakpoint there so i'm going to run to that it's going to hit that break point and then i'm going to continue over it um what's going to happen it's hit the break point continue over it yeah of course we don't know do we whether it um i guess it doesn't matter because it just goes back into the cisco so that will return when it returns but would the other threads have started when i did that good question don't know um we need a multi-threaded program with i longer than that it's going to take longer than this the longer the remaining three minutes to find the answer to that but i have to go look at that i wonder whether gdp gets that right i wouldn't be surprised if it just doesn't get it right and none of the other threads uh run um but it might singles you might know cisco instructions it could it could smart it'll know it's just construction and single not single separate but do a um beat trace cisco um there's another um question um hey presumably so mike says prejudice race will send a signal back to gb for the cisco instruction when it's executed i think well it probably will if it does if gdp starts with pre-trade cisco i think that's what will happen um but yeah i i don't if you try if it starts with peter's comp i don't think it will i i'm gonna go on a limb i'm gonna bet i bet the gdp gets this wrong but um we'll find out we'll find out um and then there's another instruction a question from charles how will cpu instruction reordering affects search debugging and actually it kind of doesn't so you know having said at the beginning i'll it's all models and you need to know what the models are underneath um actually i think cpu instruction ordering on x86 is one where you as far as i know anyway i don't know of any instance where you would need to know that but it is completely transparent and the the silicon the chip is sort of clever enough when it gets a signal to kind of you know an interrupt it sort of unwinds everything and makes it all as if you know the instructions had execute all of the instructions up until a certain one all the instructions before the one where you stopped have executed number ones afterwards have so it's basically completely transparent you'll only see it i think in terms of performance um you do get exposed to things like memory ordering um multiple threads um accessing memory and some of the weak guarantees that are given in order to do efficient um multi-core uh multi-cpu package systems and some of that you can be exposed to but um i might be wrong i might be wrong but i don't think i think that is one as far as i know instruction reordering is something that um we can all just forget about unless we are really trying to squeeze every last sort of nanosecond of performance out greg we have uh yeah we have a question from richard northcott yeah his question is uh yeah regarding static analysis aliasing what are the limits of this thinking of nested function calls etc uh yeah that's a great question um so there's no hard answer uh i mean there's no hard number um basically uh i think you're kind of you're kind of bound by uh cpu and memory but basically if you think about the control flow graph uh stat analyzers will take a depth and a breath first type of approach to make sure as many paths as covered as possible and in a product like coverity we can limit how many paths we want and we can expand that to to be massive and so uh and then along the different paths the different anything that and it encounters it will it will keep track of them and so when an analysis like this uh occurs you know it will apply for whatever it's able to to pick up right so the whole idea behind static analysis is uh uh i mean in runtime you you only can hit the different paths and different cases based on the tests you have right and um a style analysis in some ways it's kind of confined but this time it's a limit on how much uh compute and how much memory you're willing to give it cool so that's that we are at time um so yeah thanks thanks for uh doing that with me duane that was that was great and thank you everybody uh for listening and some really good questions um uh so yeah at least some of you were awake the whole way through so that was pretty cool um it's interesting the first time i've done this remotely uh the one thing i thought i really missed is you know the demo we did with the optimized code and somehow it did something different i must have done something wrong and that's the sort of thing where if we were doing this live some kind soul from the audience will have pointed out what i've done wrong but because i can't hear anybody uh i i i it's still a mystery so um if anybody spotted what i got wrong there i'd love to know i must have done something wrong um but yeah so that's it that brings us to the end um thank you very much everybody hope you have a good uh remainder of the conference all right thanks guys thanks joy