Advanced Linux C++ Debugging Techniques with GDB Tools

[00:00:00]- So, welcome everyone.

[00:00:01]Hope it was a good lunch.

[00:00:04]So we're gonna talk about lots of cool debugging stuff.

[00:00:10]All right, saw some people coming in.

[00:00:12]This is a big room and these lights are kind of scary, but anyway, I'll try, I'll try my best.

[00:00:18]Okay, so just to start off and just to set the scene, the scope, of what I wanted to talk about today, so gonna just show a bunch of specific tools, some of them freely available, some of them, some of them commercial.

[00:00:32]Gonna either show or talk about just a random set.

[00:00:36]There's no particular rhyme or reason why I chose what I chose.

[00:00:40]If your favorite thing isn't here, then, well, deal with it.

[00:00:47]And it's really about detecting.

[00:00:50]It's about root causing bugs.

[00:00:53]It's debugging.

[00:00:54]So often, these things, scopes are best set, I think, by saying what they're not.

[00:00:59]So I'm gonna steer away from generic advice.

[00:01:02]I mean, there's some really useful stuff we can do, and I think a lot of what makes experienced programmers valuable is the tips and tricks they learn themselves for how to debug stuff, not necessarily using tools, but anyway, this talk isn't about that.

[00:01:18]It's not about testing or testing tools, right?

[00:01:21]There are loads of great testing tools, there's loads of great testing talks, that's already important stuff, just, that's not what I want to talk about today.

[00:01:31]It's not about how to avoid writing bugs.

[00:01:33]I'm gonna assume that nobody in the audience is perfect.

[00:01:37]And it's not about performance profiling.

[00:01:39]Well, not per se.

[00:01:43]In a sense, looking at a performance anomaly is a bug, right?

[00:01:46]If the program doesn't comply with the spec, and the spec might be that this thing needs to respond within so many milliseconds, then if it's not, it's a bug.

[00:01:54]So there's clearly overlap between performance profiling and debugging, and sometimes, the two are the same.

[00:02:02]But I'm looking at this from a point of view of debugging, not performance profiling.

[00:02:08]Certainly not exhaustive, as I kind of said at the beginning.

[00:02:14]So, oh yeah, and also it's not a workshop, right?

[00:02:18]So all the tools I show, nobody's gonna come away expert in those tools who wasn't already an expert when they walked in.

[00:02:25]And the point of it is, hopefully, you'll see some stuff that you just haven't really seen before, or you might have heard of, but weren't quite sure what it was, and then this gives you enough to maybe, then you can Google it and you can read the man page or whatever, and you'll never become an expert without actually using stuff.

[00:02:42]But I do want to talk a little bit about why I care, and give it a little bit of context.

[00:02:49]And I'm gonna start at the beginning.

[00:02:53]So this chap, chap called Maurice Wilkes, who I think had as good a claim as anybody in the world to being the world's first programmer.

[00:03:01]He's the first person to write code on a computer to do a real job, to do something other than just test this experimental machine that people were building lots of just after the Second World War to test that they work.

[00:03:14]He was actually, I can't remember what it was.

[00:03:15]It was some biological problem that was (mumbles) very complicated mathematics behind it, so he wrote a program on one of the first computers.

[00:03:25]And he said in his memoirs that he remembers that the realization came over him in full force, that a good part of the remainder of my life was going to be spent finding errors in my own programs.

[00:03:40]I kind of remember that feeling as well, when I first started to program, and I think we've all been there, we've all had that realization that it's just really not as easy as you might think.

[00:03:50]I mean no one thinks, people who've not seen programming before, I don't think they think it's easy.

[00:03:54]I don't think you realize just quite how impossible it is to get a program right.

[00:03:59]I mean, I like to think of myself as a reasonable programmer, at least, perhaps, I was before I got out of practice.

[00:04:09]What's the longest program I can write, and it will just work first time?

[00:04:14]20 lines, 30 maybe if I try really hard.

[00:04:19]And I think that programming is dominated by debugging.

[00:04:24]I think there's been studies that show most programmers spend more than half their time debugging.

[00:04:29]That might not be debugging some in-production failure, might not be what we think of debugging, but again, just ask yourself that question, how often does it work first time?

[00:04:39]I remember a couple of years ago when my daughter started programming in Scratch, and she said, "Dad, it's lots of fun, "but I have to try lots of times to get it "to do the right thing," and I was like, yup, there's that feeling still happening.

[00:04:51]And it's kind of cool 'cause then she could understand a little bit more about what this company with a funny name that Dad started actually does.

[00:04:59]So anyway, that's kind of why I think debugging is really, underrated's not the word, no one could rate it, but it's underappreciated.

[00:05:09]I guess, so there's the obvious thing, well, it's better not to write the bugs in the first place.

[00:05:14]Yeah, well, duh, of course it is.

[00:05:17]But none of us is perfect.

[00:05:18]I think a nice test to the statement as whether it's worth making is would the opposite of that statement be in any way sensible?

[00:05:25]And clearly this one, so obviously better to avoid the bugs, writing the bugs in the first place, and prevention is always better than cure.

[00:05:32]But whatever prevention you do, you're going to need cure, and when it comes to programming, you're gonna need quite a lot of it.

[00:05:40]And here's a nice quote from a splendid chap, Brian Kernighan, that probably many of you know.

[00:05:46]It's quite well known that everyone knows that debugging is twice as hard as writing the program in the first place, so if you're as clever as you can be when you write it, how will you ever debug it?

[00:05:54]Good quote, makes you think.

[00:05:58]And I think that's another of the differences with experienced programmers.

[00:06:02]Someone likened it to learning to ride motorcycles, and when, like these 18-year-old guys go on their first motorbikes, they tear around at 100 miles an hour, and the ones that survive, sort of the ones that don't get selected out of the gene pool, go on to be kind of middle-aged men riding around on motorcycles, nice and slow, well within their kind of margin.

[00:06:21]They've got lots of margin for error 'cause they know they're gonna make mistakes, and I think programmers, as programmers, we learn this as well.

[00:06:32]I think it's an interesting, if you think about what that statement implies, it's that debuggability is the limiting factor in how good our programs can be.

[00:06:44]So whatever metric for good you have, whether it's how fast it goes, whether it's how extensible and maintainable it is, whether it's how small it is, whether how many features it has, whatever the metric for good is, if you could make debugging twice as easy or half as hard, then you could make those metrics twice as good.

[00:07:07]It's the bottleneck, it's the limiting factor in how good our programs are.

[00:07:11]Yet it gets remarkably little attention.

[00:07:13]You look at the ecosystem out there, the number of talks available, the number of tools, the number of books, the meetups and conferences on things like performance profiling or testing, is huge.

[00:07:26]There's loads of it, but there's comparatively hardly any on debugging.

[00:07:30]Now, I appreciate, to some extent, I am preaching to the choir, 'cause you guys all turned up today, but still, that's my little soapbox bit.

[00:07:44]So, and if you just think for a moment, the magnitude of the task, we do this every day, this debugging bit, and just kind of normal practice.

[00:07:53]Modern computers issuing billions of instructions every second, and that's if you have one thread in one process, so that's kind of as simple as it gets.

[00:08:01]And you're looking for that one bad instruction often.

[00:08:04]And it is the ultimate needle in a haystack challenge.

[00:08:09]But here's another quote from that same chap, Brian Kernighan, says that the most effective debugging tool is careful thought with judiciously placed print statements.

[00:08:19]Now, splendid chap, he did say that in 1979, and the world has moved on, I think, a bit.

[00:08:27]In fact, when he said that in 1979, interactive terminals were kind of new.

[00:08:31]Print statements, more often than not, meant printing things out on a dot matrix printer or a line printer or something.

[00:08:37]So I think the world has moved on since 1979, and we do have much better tools at our disposal.

[00:08:44]There are times, we all know there are times, when good old printf debugging just is the tool for the job, so like all these tools, you have to choose the right tool at the right time.

[00:08:56]But yeah, world has moved on, so where are we now?

[00:08:59]I think there are really two kinds of debugging tool.

[00:09:03]You can categorize things how you like, but I categorize them like this.

[00:09:05]So you've got the kind of checkers, so dynamic and static analysis, and mostly what you're doing there is trying to look for did my code do a particular instance of a bad thing, a buffer overrun, for example?

[00:09:19]And then the other kinds of more general purpose debuggers, which are really about code exploration and trying to work out what did my code do, and think actually a good debugger, you can spend time in a good debugger not actually debugging, just trying to work out what some piece of code that you've inherited does.

[00:09:37]Perhaps somebody has been very clever, and you need to find out just how clever they've been.

[00:09:43]Perhaps it was you, a few months ago.

[00:09:46]But it's just a general, what did the code do, kind of debuggers.

[00:09:51]I'm gonna cover both types a little bit today.

[00:09:54]They definitely both have their place.

[00:09:57]It's usually, if you can catch it in a checker, if you can catch it, I mean static analysis probably falls into that prevention thing, the isn't prevention better than cure, but it's still kind of, it's about root causing and extracting bugs from your code.

[00:10:12]So running a, I'd say this is roughly what I'm gonna try and touch on today.

[00:10:15]Lots of stuff.

[00:10:18]I'm not gonna cut exactly to schedule, but as I say, the point is that hopefully we'll see enough of these things that you'll say, oh yeah, okay, that's cool, I see how that could be useful for me, and maybe not too intimidating as well.

[00:10:29]I think a lot of these things, you hear about them somewhere, probably one of the smartest people you know talks about using it, and you think, gee, that sounds complicated.

[00:10:36]Actually, lots of these things really aren't that complicated, but as per the title of this talk, there's going to be a bit of GDB wizardry focusing on GDB, just touching some of the stuff.

[00:10:47]I gave a talk last year, two years ago, on some advanced GDB stuff.

[00:10:52]I'm gonna cover some different, little bit of overlap, little bit different stuff.

[00:10:55]I'm certainly not assuming that you saw any of my previous talks already.

[00:11:02]And all the rest of the stuff is new, is new, live demos to go wrong in all sorts of kinds of new and exciting ways.

[00:11:10]Okay, let's start then with GDB.

[00:11:13]So I think, it was in the abstract anyway, this is all Linux-specific, obviously C, C++ specific, 'cause we're here.

[00:11:21]It's not really about C++, obviously, but it's about how you debug binary code, compiled code.

[00:11:28]So yeah, for GDB, it's certainly not intuitive.

[00:11:31]It can be very intimidating.

[00:11:34]It's perhaps, of all the tools, and having said that some of these tools sound complicated but easy to use or easy to learn, I think GDB probably is a good example of one that isn't.

[00:11:45]I think it is easy to use, just not to easy to learn.

[00:11:50]But once you've got the hang of it, it is pretty powerful.

[00:11:57]So the first thing I'm gonna talk about, and I find I'm always amazed by this, it's the one feature of GDB that I think is least, (mumbles) combination, least known and most useful.

[00:12:11]So here is a program, Hello, World with just a tiny little bit of extra stuff, and so I'm gonna compile it.

[00:12:20]Now, I need you to compile gcc you (mumbles) with -g.

[00:12:23]Actually, it's better to say that.

[00:12:25]That will generate, -ggdb3 will generate more, richer debugging information can be, and GDB can do much better job of in-lined functions, optimize the way data templates and all that kind of good stuff.

[00:12:38]So that's, and actually, using a very ancient debugger.

[00:12:41]That's probably a better argument to give it.

[00:12:45]Sorry?

[00:12:46](man mumbling) Can I increase the font?

[00:12:48]Yeah, like that? Good.

[00:12:52]Okay, so made my little program, so now I'm gonna run it in GDB, and I'm gonna type start, which is basically the temporary breakpoint on main, and then continue, and here we are.

[00:13:03]So this is definitely better than Kernighan's 1979 world bit but then really not that much, so I can look at my program, I'll type list.

[00:13:15]Yeah, this is feeling an awful lot like 1979 actually.

[00:13:18]So let's bring GDB forward into the, screaming forward into the 80s by Control X + A, and I get my nice cursors interface, and now this is much more useful, and now I can next and it's much more like being in a debugger.

[00:13:33]So that's a very, very useful feature.

[00:13:36]It does kind of, is a bit temperamental, to be honest.

[00:13:38]It's like most cursors applications.

[00:13:40]It's worse than most cursors applications if you're running your program inside it like I'm doing here.

[00:13:44]I think it behaves better when you attach to a running process because here, it's kind of fighting for the terminal with the process that you're debugging, with what GDB calls the inferior.

[00:13:53]But nonetheless, we can get some multiple windows, so here I can step through the disassembly as well as the source code, for example, so all cool stuff.

[00:14:02]I'm not gonna spend too long on that.

[00:14:07]Control + L is very useful in TUI mode because it refreshes the screen and you need to do that more often than you might hope.

[00:14:14]Sometimes there's no way around it.

[00:14:15]You just have to start again.

[00:14:17]Yeah, terminals are messed up.

[00:14:21]Very, very briefly, my good friend Jeff Turow is gonna be talking about Python and GDB later on on Thursday, is that right?

[00:14:30]I say good friend, only met him about half an hour ago, but we're a conference, so that counts.

[00:14:37]But just very, very briefly then, to introduce the Python built into GDB.

[00:14:42]It's really powerful.

[00:14:43]You can do all kinds of cool stuff with it.

[00:14:46]So go like that.

[00:14:51]Or I can... I can import print I am pid.

[00:15:06]It's pretty, pretty, pretty complete.

[00:15:07]So it's not just shelling off the Python process and running that, and if I look at the processes here, so I can, if I want to shell off the process and do that, I can do that from the prompt with shell, and now I can see there are the pids that GDB is pid to 3905 and sure enough, that's what it printed up there.

[00:15:29]Now there's a lot of, they've bound the Python interface to what's being debugged really quite tightly.

[00:15:35]There's all kinds of things you can do with breakpoints and exploring the data, and I'm gonna leave that to Jeff's talk to go into that into more detail, but those are just some of the commands you can just kind of get started.

[00:15:50]You can do, I'll just say heck of a lot with that scripting.

[00:15:53]It's very powerful.

[00:15:55]So I'm not gonna go into any of the details, as pretty-print is for STL.

[00:15:59]The only thing I will say, just one little note of advice that I've seen before is, generally, you can, generally, GDB will debug arbitrary binaries that you've made anywhere, and it works well for that.

[00:16:12]If you start trying to debug things like the STL using it's pretty-printers, you need to have used a similar, probably the same or similar distro to have compiled your program on which you are now debugging it.

[00:16:26]Otherwise, it gets, it all gets very confused with the pretty-printers that live on the machine on which you're debugging.

[00:16:35]So you can actually take a copy of the GDB binary and move that around quite easily.

[00:16:39]But then, yeah, that's the other thing I see quite a bit is the GDB, Python, if you're just using your nice Ubuntu or Fedora or whatever, and it's all packaged and it all works really well, if you start trying to take a GDB, for example, and run that GDB binary on another distro, it will kind of work, and even the Python integration will appear to work, but then it will try to use some of the Python libraries, and find that there's version mismatch between the GDB binary and the Python library, the Python interpreter that's inside the GDB and the libraries on the system that it's trying to use, so that's kind of some of the more commonly, just messed up configs that I've seen that cause all kinds of issues.

[00:17:29]My one advice, I think I said this wasn't gonna be any general advice.

[00:17:32]This is perhaps straying dangerously close to it, but the one bit of advice, again, this is just kind of pitfalls that I've seen before, keep your GDB in it nice and simple.

[00:17:42]I remember years ago, we had to help a customer who had some really weird behavior and all sorts of mad stuff was happening, and it turned out that they had a run command inside their.gdbinit, which was something we didn't think to test of ahead of time and we didn't quite handle properly.

[00:17:58]It's quite, I mean it's quite a good, if you can put your, a nice GDB in it, there's all kinds of functions and things in your source control and then source that from the GDB command line, that works.

[00:18:07]That works quite well.

[00:18:11]And the history save is good because that means your nice up arrow, get my commands from before, that saves then across sessions, so history save has nothing to do with reversible debugging or anything like that, but it just saves the commands but often, it's much nicer to type up arrow + Enter than actually have to type the thing in again.

[00:18:34]And yeah, pagination off and confirm off because if you're not living life, if you're not living life on the edge, then you're taking up too much space.

[00:18:47]Okay, little bit just about how GDB is implemented, because I think this is useful to really get the most out of it.

[00:18:52]You kind of have to, it's like old stuff.

[00:18:54]You can understand kind of the layer at the top and you can go so far, but if you do more layers down, you can understand, the more you shouldn't get out of it, and particularly, the way that GDB interacts with signals is, I think, kind of surprising at first.

[00:19:11]It really makes sense when you understand it but it's surprising at first.

[00:19:14]So the thing you need to know is that GDB is built on top of ptrace, which is like a really horrible API in the Linux kernel.

[00:19:23]I think it was inherited from releasing Solaris, possibly from longer ago than that.

[00:19:30]And yeah, it's an awful API but it works and so that's what, there's been a couple of attempts to replace it over the years, but none of them have really got traction.

[00:19:42]But so when GDP is running, when you're running the inferior, whether you've attached to a running process or whether you run it from the GDB command prompt, it's doing that under the control of ptrace, and the way ptrace works is when the inferior process, as GDB called it, when the tracee process receives a signal, it doesn't actually receive that signal.

[00:20:04]It stops at that point.

[00:20:05]Control is returned to the tracing process, which is GDB in this case, which will pick it up through a waitpid return, and then GDB can decide what to do and it can decide to just continue the program, throw that signal away, feed the signal in.

[00:20:20]So let's have a look.

[00:20:26]Oh, I'm gonna set (mumbles).

[00:20:31]So most of the signals, we've got Stop, Print, Pass to program.

[00:20:35]Most of the signals that, it'll do all those things, so when the inferior gets a SIGHUP, it'll stop, control returns to GDB prompt, and it will say, got a SIGHUP, and you can press continue, and if you do, that SIGHUP will then be passed into the program and if it has a handler, that handler will run, or if it doesn't, then it will do whatever the default action for that signal is, usually terminate, sometimes ignore, whatever, but some of them don't, so SIGINT, for example, that we treat specially, so when you're in your debuggee, your inferior process are inside GDB, and you hit Control + C to get control back.

[00:21:10]Well, at least if you've launched, if you've run the program from GDB prompt itself.

[00:21:17]GDB isn't doing anything special with that.

[00:21:19]It's just that when you type Control + C at the terminal, it will generate a SIGINT and deliver that to the program that's being run.

[00:21:26]Actually, every process inside the process group of which the terminal is the controlling terminal, I think, and so the normal thing happens, the process stops, GDB gets the notification that SIGINT has arrived and it returns to the prompt, and you type continue, and you'll notice the pass to program there for SIGINT is no, so that if you type continue, then that SIGINT will not be delivered to your program.

[00:21:49]So if your program you're debugging has a handler for SIGINT and relies on SIGINT being called, then you'll need to change that inside what you do with, you'd need to say handle SIGINT, and then you have print, stop print pass.

[00:22:10]I think if I want to go back to the original behavior, I'll do that.

[00:22:18]Okay, so I think that's, oh, SIGTRAP likewise, so when you hit a breakpoint, it'll just generate a SIGTRAP, and what GDB will do when it's a breakpoint is it'll change the code.

[00:22:31]I think it's architecture-specific, but certainly on x86, and probably most architectures, it will change the code.

[00:22:39]It will write the opcode to generate a trap.

[00:22:41]In the case of x86, it's the 0xCC opcode, which is a single byte opcode that generates a trap.

[00:22:48]Other architectures, it might generate different instructions, and it will just literally plonk that in the text section so when the program gets to it, the programmer receives a SIGTRAP, GDB stops, and returns to prompt, and again, I think if we go, we'll see SIGTRAP, which is here.

[00:23:08]Again, does not get passed to the program when you continue.

[00:23:13]Okay.

[00:23:18]So yeah, they'll actually, SIGINT SIGTRAP are used when you're normally debugging, but GDB doesn't actually hand, doesn't do anything particularly special with them other than responding to the SIGTRAP in the right way when it hits what it knows is a breakpoint.

[00:23:32]Watchpoints are, watchpoints are super cool.

[00:23:35]Really cool with reversible debugging which we'll show in a bit.

[00:23:39]So watch foo, so I'm sure most people will have had experience with this.

[00:23:46]And so yeah, you've watched and you continue and then when foo is not a foo, is a variable in this, assume leave a variable here, when foo is modified, it will stop and so you can run forward to the next time, and the foo is modified.

[00:23:59]It tries to be quite clever, and so foo is a local variable when it goes out of scope.

[00:24:04]GDB will actually set a breakpoint internally at that function.

[00:24:07]That's the end of that scope and then also, okay, that's no longer being watched because it's out of scope.

[00:24:12]Actually, if you're debugging compiled code, usually, what you care about is I want to watch that address 'cause I've got some other stray pointer somewhere that's stamping on this or something.

[00:24:22]So watch -l, which is new-ish.

[00:24:25]I don't know (mumbles), anything new within a few years old will watch -location, and that won't try and do the clever when it goes out of scope, stop watching it thing, so if you've got some local variable that's being trashed, it'll just watch that address.

[00:24:39]Read watchpoints, so generally, actually, if the variable foo is written to, let's say foo gets an integer and it contained 42, if foo is written to and it's updated with the same value as before with 42, then it won't stop.

[00:24:57]That's not considered, the variable beta hasn't changed even though you actually physically wrote to that piece of memory.

[00:25:05]Just waiting for it to change, and rwatch is a read watchpoint, and if the architecture supports it, x86 does, then it can stop whenever that variable's being read, which is useful.

[00:25:17]And we can have thread-specific watchpoints and we can apply conditions and we can combine these in all kinds of useful ways.

[00:25:29]So thread apply, I think, is another useful command.

[00:25:31]Most commonly used, I think, with backtrace.

[00:25:35]So, often, especially if you're debugging some, someone else sends you an error report or some sort of bug report, and you just want to say, yeah, thread apply all backtrace full.

[00:25:45]That's nice, that will give you, let me show you that.

[00:25:46]That will give you a backtrace of, we're at a multi-threaded program now, so I've got, here's one I made earlier, so that's, that program, and if I run that, it's just got these 10 threads which were just up, just running around updating those values.

[00:26:06]So if I run that... All good.

[00:26:13]So as you probably know, info threads tells me all of the threads in my process and where they are, and yeah, thread apply or backtrace gives me a backtrace for all of my threads.

[00:26:26]Thread apply all backtrace full, including all the local variables, so that's kind of useful.

[00:26:36]I've only ever seen thread apply used with those options but you can do other things, so I can say thread apply 1-4 print, and $sp is a convenience variable for the stack pointer.

[00:26:49]And so thread apple. (chuckles) There we go.

[00:27:05]Dynamic printf, so much maligned printf.

[00:27:06]Printf is the worse debugging tool in the world except, of course, it's quite useful, but obviously, the worst thing about printf is that you have to think in advance where to put the printf, and you have to put the right one in with the right date, print out the right arguments.

[00:27:21]Otherwise, you need to recompile your program and deploy it again and run it again with the printf that you wish you'd put in the first place.

[00:27:29]Now, dynamic printf is halfway to solving that problem.

[00:27:34]So it's kind of neat, so we can go, so let's do, so dprintf, and it's, the syntax is a little bit arcane.

[00:27:46]I think it's, here it goes, right, yeah, so mutex_lock is my function.

[00:27:55]And so that's where I would put a breakpoint.

[00:27:58]I've got a feeling you have to do this without spaces, I can't remember, so I've got these mutex things that I have got in my little threaded program I just showed, and I've got a magic (mumbles) and... Okay so kinda crummy, but my program exited.

[00:28:19]But that's okay because I can just start it again and run, and there we are.

[00:28:23]Okay, so I've... That's... I wonder how many times that particular bug is made in the world every second.

[00:28:36]All right, there we go.

[00:28:40]Cool, so don't print, it's cool.

[00:28:41]It's a bit slow, I mean it's fast enough in this kind of case that we don't care at all.

[00:28:47]Particularly if you're remote debugging, which we'll probably won't get time to look at, but we'll touch on, if you're remote debugging, then it's very slow because what's happening internally is GDB is hitting a breakpoint on mutex_lock, control is returning to GDB, it's then running printf commands, like calling those inside the inferior to do the printing that it needs to do.

[00:29:10]Getting control back, removing that breakpoint and continuing, all of which is very slow, and all of which is really slow if you are remote debugging, so you can do this dprintf style agent.

[00:29:23]I'm sorry, the first, sorry, I lied.

[00:29:24]The first one, GDB.

[00:29:26]GDB will just figure out what the printf would have been.

[00:29:27]Call will call printf inside your program.

[00:29:30]An agent, if you're running, if you're doing remote debugging, so you've got a GDB server on some kind of target, it will do the printing inside that agent and it can save a lot of time.

[00:29:42]And it's reasonably configurable, as you can see, so dynamic printf is cool.

[00:29:46]I mean you still need to put the dprintf in before the actual bug has happened.

[00:29:53]You still need to catch it in the act, but at least you don't have to change your code and recompile your code to get that, get more printf info out.

[00:30:02]So I just touched on calling inferior functions, and so this is very useful.

[00:30:07]You can just type call foo from the command line and it will call the function foo.

[00:30:12]It can be surprising.

[00:30:14]Print foo+bar, if you're in C++ might, we might have overloaded the plus operator and so GDB is smart enough to figure that out.

[00:30:22]Well, smart enough, and so sometimes, that can be surprising that that might call.

[00:30:27]Print errno will call a function in your inferior because errno is a thread local and it's actually defined as a function called get errno address or Something, and GDB will just call that when you call print, when you type printer errno.

[00:30:46]And this one caught me out.

[00:30:47]Does my little pointer thing work?

[00:30:49]Good, I think it does.

[00:30:50]This one caught me out.

[00:30:52]Passing literal strings, so from the GDB prompt, I type call strcpy( buffer, "Hello, world!

[00:30:58]The first thing it will do is call malloc inside my program.

[00:31:00]So malloc a buffer into which it can put Hello world, and so if you're debugging your own malloc implementation, then yeah, that can get interesting.

[00:31:11]Catchpoints are very cool.

[00:31:12]I'm not gonna go into them in detail.

[00:31:14]They're kinda like breakpoints but they stop on a nominated system call.

[00:31:21]If you say catch syscall or you can catch exceptions, which also is useful.

[00:31:27]So yeah, kind of like breakpoints but rather than giving a line of code on which to stop, they give some kind of condition, something your program might do on which to stop.

[00:31:38]Remote debugging, I touched on.

[00:31:40]I think I'm gonna just put that up there so you can, I mean it's quite, very simple to use on the same machine, so here we're debugging over a socket, so you need to run this gdbserver, which is this little, little stub application that GDB will connect over a socket or whatever, which itself will then, then GDB server will debug the inferior using ptrace.

[00:32:03]Yeah, you can do multiprocess debugging, which is good, but we're kind of running out of time, but very quickly, so I can actually get multiple, I can debug multiple processes at same time, and it looks very like debugging a multi-threaded application.

[00:32:16]So if I just, I have to set the set follow-fork-mode child parent, and set, actually, so the key one is set detach-on-fork, so by default, GDB will detach on a fork from one of the parent or the child process, depending on what you set the follow fork mode to, but if you say detach-on-fork off, then it will continue to debug both the parent and the child process after a fork, and you can list, just like you say info threads to see all the running threads, you can go info inferiors and see all the running processes and switch between them, like you say thread one to switch to thread one, thread two, say inferior one or two.

[00:32:57]So that can be kind of handy if we've got lots of processes to debug and to keep in your head all at once, or you could just start to copy to GDB, whatever floats your boat.

[00:33:06]You can create your own commands in Python.

[00:33:07]I think Jeff is going to talk about this in more detail, and so won't go into that.

[00:33:12]You can have little stop handlers, so (mumbles) it's a Python that get called when certain things happen.

[00:33:16]Also very useful.

[00:33:18]You can do temporary breakpoints, you can have breakpoints on a regular expression, which is really neat if you want to stop on every, and if you've got some library that starts mylib underscore, though you can go mylib underscore dot star, and it will put a breakpoint on every function in your library's API.

[00:33:37]One little note, because people often get confused about this.

[00:33:43]Typically, we have debug builds and release builds, and in debug builds, we run them with low optimization, and debug to find not, not find, sorry, and so you got all your assertions in and everything else, and it can go a lot slower, depending on, I've heard of applications going like 10 times slower when they're running the debug build and a release build, and so people go, oh, I can't run GDB on my program because debug builds are too slow, and that life is, the world is more complicated than simply having the debug or release.

[00:34:18]They're just sort of conventions, and whatever optimization level you have and what debug info you're generating are completely orthogonal, and so you can have minus 09 and minus GDB3, and lots of debug info very optimized code.

[00:34:35]It'll be kind of weird when you debug it because you think you're stepping forwards a line and the compile has laid out code which you didn't expect, so you need to kind of, to be aware of that, but it will work, and there will be absolutely no runtime performance impact.

[00:34:47]In fact, the only thing you'll use is a bit more disk.

[00:34:49]You won't even, if you're not debugging it, you won't page in the debug info sections from disk, so just to correct that common misunderstanding.

[00:35:05]All right, enough GDB.

[00:35:07]Let's move on to other things.

[00:35:09]So valgrind, everyone calls it valgrind, but actually, it's called valgrind.

[00:35:15]I think, so I'm told.

[00:35:16]Anyway, actually, so the most common one is memch-- It's a platform, you have all these tools, different tools.

[00:35:21]The most common one is memcheck.

[00:35:23]So they're kind of synonymous, valgrind and memcheck.

[00:35:25]When people say run valgrind on it, they often mean run valgrind with memcheck, which is fine.

[00:35:31]Then you've got these, actually so it does strike me, it's definitely called valgrind, not valgrind.

[00:35:36]I don't know how you say cachegrind 'cause that doesn't sound right.

[00:35:39]Cachegrind and callgrind, but anyway, there are the other, these other tools that you can run within valgrind.

[00:35:47]It can be rather slow, but it just works, which is really neat.

[00:35:50]So you don't need to recompile your program, you don't need to link against any libraries.

[00:35:54]It's in most distros, so you can just apt install valgrind or whatever, and then just use it.

[00:36:01]I'm reminded of a real-world story of using this when I worked at my last proper job before I started doing Undo was, we had an LD_PRELOAD library which was doing kernel bypass kind of before that became a common thing, and very often, we get this one thing that became consistent in my old life and working in Undo is customers would often say, well, your stuff's broken, it's definitely broken 'cause I run my program without your kernel bypass library or without live recorder and it worked just fine, and I run it with your kernel bypass in set or, and it's broken.

[00:36:41]And like a lot, like most, certainly a good chunk of the time, in both cases, they're right.

[00:36:47]It is our stuff that's broken 'cause as I said, programming is hard.

[00:36:54]But some of the time, actually, is their program's broken and they just didn't notice, and so the guy I was working with at the time, very smart guys which couldn't believe, couldn't understand why our stuff was broken, so he just got a copy of their program and ran it with valgrind, and sure enough, there was some uninitialized data that was being accessed, and we could kind of point them at that, so let's show, let's show that in practice.

[00:37:18]So here's my little canned version of that bug.

[00:37:24]So here's a nice simple program.

[00:37:26]Of course, I compile it as normal.

[00:37:32]Run it, there's nothing wrong, that's legal.

[00:37:36]Just is undefined.

[00:37:38]So, let's run that inside valgrind, and see what happens.

[00:37:47]And so, oh yeah, look.

[00:37:49]Now you'll see it's saying, it's an instruction level thing, so what it's doing, actually, I asked if Undo works a bit like this.

[00:37:54]What it's doing is it's translating the machine code as it runs in a sort of JIT fashion and doing analysis on that code, so it's not simulated, but it is (mumbles).

[00:38:09]And of course, if you printf an undefined value, the first thing you notice is there's a jump, a conditional jump, based on the uninitialized data 'cause printf is trying to turn your number, the number in this case, into a string.

[00:38:28]So it's kind of useful like that, but even more useful is you can combine it with GDB, and if I, now, the thing you got to remember that I just said, I said so valgrind is doing this translation, this binary translation of the code, so the code that you're executing is under valgrind.

[00:38:47]The code the CPU is executing is functionally identical to the original program, but it's got extra stuff in it, this different code, and so if you try to debug it through GDB in the normal way, you'll just see nonsense, because once it tries to, once GDB tries to look through ptrace, what it'll see is what the CPU sees, which is not what it was expecting to see.

[00:39:14]But valgrind has built into it, a GDB server, which you can connect to, and then you can start to do all the GDBs, and then the other thing we want to say here is if we do it like that, it'll just run to the end, so you can give it an error count like that.

[00:39:32]I think (mumbles), I don't think it matters but anyway, so this thing stopped after zero errors.

[00:39:36]I could say stop after 10 errors, and they stop after zero errors, which is gonna stop at the beginning.

[00:39:44]Sorry, there are different vgdb modes.

[00:39:47]Full is, I'm gonna use that 'cause it doesn't have any surprising.

[00:39:52]There is on as well, on or full on, just so trades off performance for being, it's not incorrect but it just gets a bit weird at times, especially with, it can miss watchpoints and things.

[00:40:06]So now, I start valgrind like this, and now it's nicely telling me what I need, tells me exactly what I need to type somewhere else to get GDB to connect to this, the server inside valgrind.

[00:40:19]So it says I'm gonna run GDB a.out, so I can run that without, here I can run that without copy/paste.

[00:40:29]But the next bit, I can't run without copy/paste 'cause it's more than nine characters, so, that line there.

[00:40:46]Okay, so here I am at the beginning of time, and then I can continue.

[00:40:50]As you can see, it's a little bit slow but it works just fine, and here I am inside this printf and I can get a backtrace, and I can see, what's wrong with my program frame 2, and here we are, accessing this uninitialized memory, so I can do all the all the GDB stuff to walk around and explore and try and get a bit more information.

[00:41:10]Now you can't combine, unfortunately, you can't combine valgrind with any kind of reversible debugging.

[00:41:17]That would be super cool.

[00:41:19]Can do it with AddressSanitizer and stuff, which we'll get to in a minute.

[00:41:25]Okay.

[00:41:27]Kind of getting on for time so let me try and speed on through.

[00:41:33]So yeah, a whole bunch of different tools.

[00:41:35]The default is memtool, which I think most people think of, as I say, all different things you can do with valgrind.

[00:41:41]Right, on sanitizers, which are kind of like valgrind but different, so unlike valgrind which will work on an unmodified program, the sanitizer's built into the compiler.

[00:41:50]Originally came in clang, and has been available in GCC for some time now as well.

[00:41:56]Slightly more arcane typing needed for GCC than in clang.

[00:41:59]I don't quite know why but that's what Google told me I had to do so that's what I did and it seems to work.

[00:42:05]Anyway, it's quite, it's much faster than valgrind.

[00:42:09]Valgrind will slow down by, can't be anything up like 100x.

[00:42:15]Can be more like 10 but it can be 100x, can be, yeah, can be very slow.

[00:42:22]The AddressSanitizers, and the other kind of sanitizers are much quicker, typically, 2x.

[00:42:28]So there's still, there's still an overhead because it's (mumbles), the compiler, basically, is instrumenting all your memory accesses, but it's doing it at the compiled time, so pros and cons.

[00:42:40]Also they do find different types of errors.

[00:42:44]There are some that's sort of an overlapping set of bugs that they'll find.

[00:42:49]So let's see, so I have, so I've got this out of bounds function, which is nicely written, so if I give it a number to (mumbles), it'll use that directly to indirect this, to reference this array here.

[00:43:07]So what did I say?

[00:43:11]Fsanitizer, it calls address and (groans) what's the, static-libasan out_bounds.

[00:43:28]Fsanitized, not sanitizer.

[00:43:32]Okay, so if I run this like this, it's fine 'cause that's, there's enough elements in my array to access.

[00:43:39]If I do it like this, it's a right activity array out of bound, and now my program has told me.

[00:43:46]Now, because this is actually running, this isn't doing the JIT binary translation stuff, this is really running on the hardware.

[00:43:55]You can combine this with other debugging tools just fine, so particularly, with reversible debugging, which is something close to my heart, just works.

[00:44:05]So let's first show some reversible debugging for those of you who've not seen it.

[00:44:11]So this is a program.

[00:44:14]Bubble sort of program which contains a bug.

[00:44:22]Which is one of those... Non-deterministic bugs so if I run it in a loop like that, it runs just fine until eventually, it doesn't.

[00:44:39]Now I've got, now I'm running with, I wanted to actually do this a different way.

[00:44:44]Anyway, let me, this has got the stacks motion detection but it actually slightly messes up this demo, so I'm gonna... I think that failed first time that time.

[00:44:56]No, it failed every time, okay, 'cause it's the wrong (mumbles).

[00:45:09]So... Non-deterministic bug, it will fail.

[00:45:24]Eventually, that will fail and I'll try to look at a core file and I find out the core file is useful.

[00:45:28]You'll have to trust me.

[00:45:31]So, but I'm (mumbles), so gdb bubble sort.

[00:45:38]So I quite like this little trick.

[00:45:41]So I'm gonna run it with process record which is the inbuilt GDB reversible debugging stuff.

[00:45:47]So now I need to run it a whole bunch times.

[00:45:49]It doesn't usually, as we've seen, doesn't usually fail.

[00:45:52]So I'm gonna do, put a breakpoint on main.

[00:45:55]I'm gonna put a breakpoint on exit.

[00:45:58]I'm gonna put a condition commands one which is gonna run a bunch of commands every time it hits breakpoint one, and I'm gonna go continue, and I'm gonna, sorry, I'm gonna type record to start turn on process record, which is the bit that we need for reversible debugging, and I'm going to continue, and I'm gonna put a break, I'm gonna put my breakpoint on dash it on score exit, so commmand 2 and just gonna rerun.

[00:46:22]That's it, all set.

[00:46:24]I think I probably would've done this because of my, but I need set confirm off for that to work, and off we go.

[00:46:31]And this will keep running it with print, and you can see it's slow.

[00:46:35]It's a very, very simple program and it takes, now the slowdown of GDB process record is kinda bit like tens of thousands of times slowdown.

[00:46:42]What it's doing is single-stepping every single instruction, and recording what changed, so it uses lots of memory and it goes very slowly, but come on.

[00:46:52]It definitely will crash.

[00:46:55]And when it does, we can step back.

[00:46:58]You know I'm gonna leave that... Oh, damn (mumbles).

[00:47:04]Let me try something in parallel, see if we can't, let's also try it with the rr which is record and replay, so let's do the same so, so rr record, my bubble_sort program.

[00:47:17]Does this saves the trace, that I can subsequently go and debug.

[00:47:23]So let's just keep doing it.

[00:47:31]Oh no.

[00:47:43]What the (groan) it's crashed.

[00:47:45]Right, so both crashed, so here we are inside the process record, so backtrace is garbage.

[00:47:53]Just like, I'm in hyperspace.

[00:47:57]I can look at the program counter.

[00:48:00]That doesn't even look like a sensible address.

[00:48:02]X to examine the contents.

[00:48:03]No, there's no memory there, but I can reverse-stepi, or rsi, and that will just go back one instruction and now I'm back in insane land.

[00:48:14]I can kind of see where I am.

[00:48:16]Okay, so I'm at the end of the function.

[00:48:21]Let's have a look, so this little arrow here tells me yeah, I'm at the end here.

[00:48:23]There's return instructions.

[00:48:24]A return on x86 will pop what is ever on the stack and jump to that, so that's the stack pointer.

[00:48:32]If I examine that, whoop.

[00:48:37]Sure enough, that's that garbage address, isn't it?

[00:48:40]So I've got garbage on my stack.

[00:48:42]So I'm gonna set a watchpoint.

[00:48:45]I'm gonna watch that address like that, and then I'm gonna reverse continue, and sure enough, we've gone back in time.

[00:48:55]Unsurprisingly, to when this array is being written, and the array SI is 35, and if I go show type info, oh no (mumbles) ptype array.

[00:49:11]I can see the array is actually only 32 elements long, and obviously, my bug is the rand P (mumbles) module or size of array, which is the size in bytes rather than the size of elements.

[00:49:24]I ran it concurrently with rr.

[00:49:26]Considerably quicker, and non-interactive.

[00:49:32]Kind of works, look more like strace, which we'll cover in just a minute if we get time.

[00:49:36]So rr replay.

[00:49:39]Now this was the one, the last one that it ran, and so this is gonna look kind of similar.

[00:49:45]Now I'm at the beginning of time.

[00:49:47]Continue to the end.

[00:49:48]Here we are, we've got this (mumbles) at this random point and I can do all the stuff that I just do.

[00:49:55]So reverse step.

[00:49:56]I should think for some reason actually two reverse, so anyway, so reverse step there, and so I can watch.

[00:50:04]So the stack is now that.

[00:50:06]And so I can watch just like I did before.

[00:50:15]To reverse continue, and there we go.

[00:50:17]Right, so it's come down the same thing, only it's much quicker, and it's got this separate record and replay step.

[00:50:23]It is quite, it's a little bit, it needs to be running on the right system, so it must be x86, must be a relatively new Intel CPU that has the right support.

[00:50:32]Doesn't work on AMD, doesn't work on ARM or anything else.

[00:50:35]I need to be running not in a virtual light, not in a cloud environment like AWS or something, because they don't have the performance counters exported that it needs, but if it's got all the bits it needs, it's very cool, it's very fast and works well.

[00:50:56]So I want to go quickly.

[00:51:01]Kinda gonna run out of time but so quickly on to ftrace, which is a different thing, so it's a function tracer.

[00:51:08]It's kind of part, this is one of those part profiler, part debugger tools.

[00:51:14]And I'm going to talk through a little case study that we did it at Undo just a little while ago, so we have this thing that's kind of like rr but works in different way, and so it doesn't have some of the restrictions, and we had a customer who's integrating it into their test suite.

[00:51:33]It's very useful in that, so you run, when you've got those failures that just happen one in thousand times, and they're not, they're intermittent, not reproducible failures.

[00:51:41]You've got a recording and you can just debug that.

[00:51:45]Very, very useful, and so we're trying to integrate it, helping our customer integrate it into their test suite, and we got to kind of, we've got an exchange where a customer came to us and says, yeah, Live Recorder keeps dying, keeps getting SIGKILL.

[00:52:00]Think actually in the process I'm recording was getting SIGKILL (mumbles) things are dying with SIGKILL.

[00:52:06]So we said okay, well, you got this quite complex test suite.

[00:52:08]You sure you don't have some kind of process killer running around, killing stuff, and said, yeah, no, definitely not doing that.

[00:52:13]Said okay, so we take a look.

[00:52:15]So we had a look around, and after a while, we came back to them, we said, you're really sure you don't have some kind of process killer?

[00:52:20]And they said, yeah, 100%.

[00:52:22]Said, okay, so we said, can you run this script which was an ftrace script and have a look, and they ran that, and then we said, oh, we've seen a process called watchdog that's sending SIGKILL.

[00:52:34]What's that?

[00:52:34]And they went, hmm, have a look around.

[00:52:37]Oh yeah, that's this process killer we have in our test suite.

[00:52:41]So the script we sent them was this, and (mumbles) the whole script, but the point, the interesting stuff with this was here, so ftrace is controlled.

[00:52:53]There are some, there's a wrapper tool you can use that's quite cool but the low level, it's controlled by this sys kernel debug file system, and then you sort of poke different things into that, so let me show you essentially what we did here.

[00:53:13]So this is the, so you can just look at the trace.

[00:53:19]If I want to clear that trace, I can just... Sys/kernel/debug/tracing it's just echo, something at it, and it resets to trace, and so what we did here was told it to, first of all, tell it to enable signal tracing, and then, say, we're in, this is the filter.

[00:53:45]They're interesting in any signal, any signal, basically, and then you go echo 1 into tracing on, and that starts tracing, and now if I look at my trace file, there we go.

[00:53:58]Started to see some signals.

[00:54:00]So everything looking quite normal here.

[00:54:02]Just some sig-17, sorry I was just SIGCHILD, but yeah, you can see these two events are the generation of the signal and this is the process that generated the signal.

[00:54:10]Oh look, cat and bash, so this is me.

[00:54:13]So I think I'm only tracing.

[00:54:16]I said last time I did this, I did it for the whole system and somehow I've set this to only trace.

[00:54:19]I think for this process creep or something, oh no, no, no, no, no, there are other things happening as well here Xorg and the like.

[00:54:25]So, yeah, here you are, sig-14.

[00:54:29]That is SIGALARM, I think, and so you can see these two events, generations (mumbles) on the delivery of the signal.

[00:54:34]Typically, you'll see the two paired, not necessarily always.

[00:54:36]Depends if the, maybe the process has masked the signal or something like that.

[00:54:42]By the way, it's another thing, if your program masks SIGINT and you're in GDB and you try to Control + C, it won't.

[00:54:49]That's why when interrupted, but the program doesn't receive the SIGINT from the kernel.

[00:54:56]Cool, so that is, that is ftrace.

[00:55:04]Oh no, now we have to go through this whole story again.

[00:55:12]Strace is probably better known than ftrace, and you can trace all the system calls, so, but obviously, you did a little bit more like, then strace also is built on top of ptrace.

[00:55:22]So ptrace is the process, gets interrupted every time there's a signal.

[00:55:26]You can also configure ptrace to interrupt, to return when there's a system call, and it will output all the system calls being issued by the command.

[00:55:33]You can do all kinds of neat things.

[00:55:35]You can follow fork, you can write the output to a command file, to an output file, et cetera.

[00:55:41]Ltrace is like strace but rather than for the system calls, it's for library calls.

[00:55:48]And actually can be system calls as well if you want, and you can configure it to print out sort of certain useful things.

[00:55:58]Oh, I forgot the really cool thing in strace.

[00:56:01]- k, strace -k command gives you a little backtrace for each system call that's issued, which is super useful because usually, the system call is not directly what you want.

[00:56:10]Anyway, sorry.

[00:56:12]So yeah, that's ltrace, so should we simply... Show it to you one of these.

[00:56:21]Can't remember what program did I?

[00:56:26]Okay, let's do date, ltrace date, and you can see all the library calls and the system calls, and obviously, you can be more selective if you want.

[00:56:42]Perf trace is a new one to me.

[00:56:44]I would find this, it's like strace but better and worse, so better because it's a lot faster.

[00:56:49]So strace will slow down the program being traced quite a lot, particularly, if it's making lots of system calls, because every time the process makes a system call, it stops, and strace gets control, and it does some stuff, and it prints out, the system call has happened, and then it continues, and that takes time.

[00:57:04]This is built on perf, so it works kind of like ptrace, like strace.

[00:57:10]It's so much, much faster.

[00:57:12]It's quite flexible as well.

[00:57:13]You can actually get it to perf all the perf events, so all kinds of events like cache misses and all kinds of stuff.

[00:57:21]You do need to be root, and it also doesn't, it's also, I mean I think this will probably change over time but right now, so strace, if you've got a string argument, so if you're saying right, string to file descriptor, strace will follow the point and tell you the string, whereas perf trace will just show you the raw pointer.

[00:57:38]That's me running out of time.

[00:57:41]Fortify is very useful, so that would get the compiler to check certain things that it can.

[00:57:46]Reversible debugging, we've done.

[00:57:49]Oh, look at that.

[00:57:50]Just a bit quick at the end but I got there at the end, so thank you very much for that.

[00:57:53]Whizzed through all the things.

[00:57:54]Could have taken a bit more time.

[00:57:57]I don't think I have time for questions in this session but I'm happy to answer any questions at the end.

[00:58:01]How strict are we on time?

[00:58:03]Can we do questions?

[00:58:03]Just couple.

[00:58:06]No one's saying no.

[00:58:09]By just seeing everybody, and just suddenly thinking do I want questions in front of everybody.

[00:58:16]Probably, I don't, because I probably won't know the answer.

[00:58:20]I think that'll be it.

[00:58:21]All right, thank you very much, everyone.

[00:58:22](audience applauding)