How C++ Debuggers Work: Breakpoints, Stepping, and More

[00:00:00]- Hi everyone, my name is Simon and if you're here how C++ Debuggers work then you're in the right room.

[00:00:07]Before I get started, the Microsoft we do have a survey which would be great if you could complete and you can win an Xbox One, woo.

[00:00:18]That's half hearted but I'll let away with it.

[00:00:20]Okay so this is a bit of my background in debuggers so you know that I actually kind of understand vaguely what I'm talking about.

[00:00:27]So I used to work for a great company in Edinburgh called Codeplay literally until this week.

[00:00:32]And there I worked a lot on debuggers so I worked on a debugger for HSA which is like a low level standard for heterogeneous systems.

[00:00:42]We have gpu's and dsp's and tiny image processors and whatever.

[00:00:47]I also worked a lot with open cl debuggers which is another kind of similar standard and I worked a lot with LLVM's debugger, lldb.

[00:00:58]There are few people at this conference who actually work on lldb so if you'd like to talk to them come see me and I'll introduce you.

[00:01:07]So a bit about what I'm talking about here.

[00:01:11]Presumably we all have used a debugger at some point.

[00:01:15]Hands? Everyone use debugger?

[00:01:18]That's the right number of hands, I'm in the right room.

[00:01:22]Anyone know what this is?

[00:01:27]The bug, yes. This is the bug.

[00:01:30]Grace Hoppers bug, yes.

[00:01:31]Anyone know what machine it was from?

[00:01:36]No, I believe it was Harvard's Mark 2 calculator.

[00:01:41]They found literally a moth.

[00:01:44]Our debuggers don't tend to look like this.

[00:01:47]If you're doing this then come speak to me after.

[00:01:50]I want to hear about it.

[00:01:53]Our debuggers sometimes look like this.

[00:01:55]Which is LDB running inside a terminal.

[00:01:59]Sometimes they'll look like this which is GDB running inside Emac's.

[00:02:04]Yup, some people like that.

[00:02:08]Some people might use the LDB cursor interface, it's quite nice.

[00:02:14]Visual studio debugger.

[00:02:18]My definition is, I definitely did not steal this from Wikipedia.

[00:02:24]An application which is used to test and debug other applications.

[00:02:29]And the most common actions you will use with a debugger are breakpoints, saying I want to stop here at this line of code or I want to stop at this function.

[00:02:41]You could step around, could be just instructions could be jumping over functions, jumping into functions.

[00:02:51]You might want to do expression evaluation.

[00:02:53]What is the value of X right now.

[00:02:58]Or you might want to say, how did I get here.

[00:03:01]What's my call stack.

[00:03:03]So these are the main things I'm going to talk about and a little bit more towards the end as well.

[00:03:11]A bit of an introduction to the platform I'm going to talk about.

[00:03:14]So I will be focusing on Linux system's on X86_64 with ELF as a binary format you don't need to know what that is yet similar for DWARF, which is debug format and using ptrace as the main interface library.

[00:03:33]So most of the concepts which I will talk about are transferrable to other platforms and systems obviously the function calls will change you might be working at a different level of abstraction but the kind of general concepts will apply.

[00:03:51]So this is a very nice look at what ELF is.

[00:03:57]There is an even more detailed version of this but it looks terrible on slides.

[00:04:01]So go to this Ange Albertini's website if you'd like to see an even better introduction.

[00:04:10]But the basic idea is that it's a binary format for executable's and objects.

[00:04:17]So it consists of a header which tells you about what this executable is and what machine am I compiled for and then a bunch of code which is separated into sections.

[00:04:34]So you don't need to know a whole lot about ELF but this the kind of main format for Linux executable's so it's useful to learn a little about it if you want to actually work on debuggers.

[00:04:48]DWARF, you noticed the funny joke ELF, DWARF Knut people are hilarious and Dwarf mostly consists of die's not like this, they look like this.

[00:05:01]This is a Dwarf information entry and these are used to describe the entirety of your programs from compilation units which is what we have here it's like files with all the includes already in.

[00:05:17]Functions, variables, types, pretty much everything you can think of can be expressed someway in Dwarf information.

[00:05:26]And this is a standard, you can go and download it you can have a read.

[00:05:29]It's actually for a programming standard it's quite understandable.

[00:05:33]So I would recommend going and having a look if you are interested.

[00:05:37]This is an example of a compilation unit die.

[00:05:40]So it tells you like that was compiled by clang 3.9.1.

[00:05:44]This is the file that is compiled for C++ you can get quite a lot of information about your programs just by looking at these.

[00:05:52]And this is what the debugger will be consuming when it's trying to understand everything about your program.

[00:06:00]Debuggers can do a lot of things without Dwarf just by looking at the Elf file, symbol tables, things like that.

[00:06:08]But for any like heavy weight debugging you need the Dwarf information.

[00:06:15]So as well as die's like this there is line table information.

[00:06:20]So this tells you which lines of my source code correspond to which machine code addresses.

[00:06:30]So you'll see that these have the addresses on the left hand side and then you have rows and columns for your source code.

[00:06:41]And then over on the right hand side there's some descriptions which are all in nice acronyms.

[00:06:47]So NS is new statement, prologue end, end text the end of the program.

[00:06:53]So this is what the debugger is going to look at to try and understand how your source code relates to the machine code.

[00:07:00]Not only how your C++, like your representation of your program maps to it.

[00:07:08]But also the literal text in the file.

[00:07:14]This is Ptrace.

[00:07:16]Ptrace is, well it's Ptrace.

[00:07:21]Everyone who's worked with debuggers at any time if you say Ptrace, their face will just drop.

[00:07:31]That's because this is Ptrace's one function you pass it a request, a process ID and then an address and data.

[00:07:42]And depending on what this request is maybe it ignores these other two.

[00:07:47]Maybe they mean something different maybe you just an all pointer all the time.

[00:07:52]So a few things you can do as like say I want my debugger to be allowed to trace me.

[00:07:57]I want to read and write memory, I want to read and write registers.

[00:08:02]And so you just pass these enum's into this function and it will do wildly different things.

[00:08:10]So Ptrace exists, it mostly does its job.

[00:08:15]If you want to read more about it there is a page.

[00:08:21]Breakpoints.

[00:08:24]So breakpoints I think are, I mean debuggers in general are quite compared to something like compilers people don't know a lot about how debuggers work.

[00:08:36]Compilers there's tons of books, there are blog posts.

[00:08:39]Most people who can kind of do a little bit of C++ will have some understanding of how a compiler works, right?

[00:08:45]You get in text, you build an AST, maybe you haven't been prepped in representation and you get to machine code and their number step to get you there.

[00:08:55]But debuggers breakpoints are like one of the most fundamental parts and people see it as black magic.

[00:09:03]This is some box where I say oh holy oracle, please place a breakpoint at line 20 of this file and by some magic you get a breakpoint at the line in that file.

[00:09:15]But they're actually not so complicated.

[00:09:18]So there are two main kinds of breakpoints.

[00:09:21]Hardware breakpoints and Software breakpoints.

[00:09:25]Difference between these two is that hardware breakpoints usually have some kind of special registers which you're going to write values into and the hardware is going to take control make sure that when you actually hit those addresses that something happens, you get a breakpoint.

[00:09:41]Software breakpoints on the other hand you actually take the code which you're executing in memory and you're going to modify it so that a breakpoint gets set.

[00:09:53]And we'll see how these both work a little bit more.

[00:09:57]Hardware breakpoints are limited because you're not using physical registers.

[00:10:02]You only have so many.

[00:10:05]Whereas software breakpoints because you're just dealing with memory, it's essentially unlimited, you know, apart from however much memory you actually have and what not.

[00:10:15]The cool thing about hardware breakpoints is that you can set them to break on reading or writing or executing an address.

[00:10:24]Where software breakpoints are only on execution.

[00:10:27]So if you want to say, debugger I want to be notified every time this address is changed which can be really useful if you're trying to track down some really weird bugs.

[00:10:41]But you can only really do that with hardware breakpoints.

[00:10:44]Software breakpoints are just execution.

[00:10:48]So I talked briefly about hardware breakpoints and we're more going to focus on software ones.

[00:10:53]So in x86, you have four debug registers which used to write addresses into these and then when you get to that address the debugger will get notified by means we're going to see in a little bit.

[00:11:07]It has a register which you can read the status of and see like what's going on with them with all these breakpoints I've set.

[00:11:17]And you have something which you can actually control to say I want a break on reading, writing, executing.

[00:11:23]I won't go into much more detail about hardware breakpoints especially because they're very hardware specific.

[00:11:30]Software breakpoints on the other hand tend to be implemented in very similar ways so if you read a debugger read the source code for a debugger then when you get down to actually setting breakpoints it's all very similar on x86 and this is how it works.

[00:11:49]So say we have some assembly code This is x86 assembly.

[00:11:54]You don't have to understand what it's doing.

[00:11:57]So on the right we have the assembly on the left we have the actual hexadecimal representation of these instructions.

[00:12:07]I want to set a breakpoint at this new instruction Maybe I know the address of this and I'm explicitly telling my debugger set a breakpoint right here.

[00:12:19]What the debugger then does is it takes this first bite this ox48 and it's going to set it off to the side remember it for later.

[00:12:33]What's then going to do is replace the old value with this special ox cc and what this is int3 instruction and this is going to trigger a software interrupt.

[00:12:50]So if our pc is sitting at the top of the program counter we go down, we stop, software interrupt.

[00:13:03]Basic idea is the operating system, if you don't know how interrupts work the operating system is going to register some interrupt handlers.

[00:13:11]And when an interrupt is triggered then the handler for that interrupt will be invoked.

[00:13:18]So for int3, which is our one up here then the operating system will put into place something which will be called for int3 and on Linux this is the function which gets invoked eventually do int3.

[00:13:35]So this is the actual code from the kernel.

[00:13:38]The most important part here is that in the middle of this we have our do trap and this sigtrap thing is important this is debugger magic.

[00:13:51]So if we look at the man pages for signals sigtrap is the trace and breakpoint trap.

[00:14:00]Now a unique signal will get sent to our process and our debugger can then say okay we hit a breakpoint.

[00:14:09]And this is how the magic works.

[00:14:11]So I'll show an example of this all put together we start off we have our debugger is awaiting input.

[00:14:18]Sitting there waiting for us to type continue or breakpoint or whatever.

[00:14:24]Our debuggee is stopped, not doing anything.

[00:14:28]We have set a breakpoint at this second instruction here note the cc on the left hand side.

[00:14:34]So what it's going to do is the debugger is going to wait for us to do something and we type in continue.

[00:14:42]So it's going to continue the debuggee and then it's going to wait on a signal as it's using waitpid which is a Linux thing.

[00:14:50]This is essentially saying okay do nothing until I get a signal and then I'm going to wake up and I'm going to do something about it.

[00:14:58]So we continue the debuggee so this is now running forget about the debugger for a minute.

[00:15:05]We go down and we hit int3, we stop, software interrupt we drop down into our interrupt handler which issues a signal, now the debugger is woken up because it was waiting on a signal now we've got one.

[00:15:24]I got a signal, yay.

[00:15:26]And now there are a bunch of methods which we can use to say okay.

[00:15:30]Where am I stopped, where did this signal come from and we can report this in some way to the user.

[00:15:38]Does anyone want me to go over that again?

[00:15:39]Because that's all the magic of software breakpoints.

[00:16:03]Yes, the question was about if we have something like this continue debuggee and then waitpid like what happens if we have the pathological case where thread scheduler like preempts us right after we call continue debuggee.

[00:16:23]Debuggee runs and we haven't hit our waitpid and we miss a signal.

[00:16:30]I honestly cannot remember off the top of my head I think it would be possible to miss signals with something like this unless you did some extra synchronization and I can't remember what kind of extra synchronization you would do.

[00:16:42]So there would be methods of doing this the overhead may or may not be worth it.

[00:16:51]Source level breakpoints, oh sorry question.

[00:17:00]What's stopping the debuggee from continuing?

[00:17:03]It's the operating system, so this ptrace thing is known by the operating system and interacts directly with the operating system.

[00:17:12]So when something is halted, it won't be scheduled.

[00:17:18]So the operating system will make sure it's not going to run, yup.

[00:17:34]What was the question, sorry?

[00:17:39]Okay good question, I didn't cover that but I can go over it.

[00:17:43]So the question was, I talked about earlier when the debugger will save out the value that was previously in this cc and then the question was how, when does it get restored back and continue execution.

[00:18:00]So there are few methods to do this the most simple is to actually manipulate the program counter so you go back over the instruction you replace the ox cc with the original bite you single step and then you continue execution then after you've replaced the breaking point.

[00:18:19]Because you don't want the case where you continue and maybe you have a loop and it should instantly hit the breakpoint and that's actually not, this happens a lot.

[00:18:32]So you want to take the breakpoint off put the old value in single step, put the breakpoint back on and continue.

[00:18:39]That's the way a lot of debuggers do it.

[00:18:41]Another possible way is actually having some memory put off to the side.

[00:18:48]And relocating that instruction into the block changing the program counter so that it essentially will eventually jump back to the right point.

[00:19:01]And it means you don't have to single step and replace the breakpoint.

[00:19:05]I think GDB calls it displaced stepping so it's very useful for multi threaded environments if you're operating in a context where some threads are running and some threads are stopped.

[00:19:17]Does that answer the question?

[00:19:20]Okay, yes.

[00:19:45]That's correct, yes.

[00:19:47]So the question was about the interrupt and so the cc, most interrupt instructions are not just one bite.

[00:19:55]They're like an interrupt and something else but int3 is special because it wants to be used for debugging so there's a special one bite instruction for it.

[00:20:03]And Linux will register a handler in the interrupt vector table and descriptor table I can never remember which one's which.

[00:20:12]And that is the do int.

[00:20:24]So the bites after int3, after the ox cc are not touched because when we eventually get rid of our breakpoint put the ox 48 back, then we want the instruction to still be as it was, so they're not touched.

[00:20:39]Yes.

[00:20:45]How does it a single set, well we have slides on that.

[00:20:48]You'll see in a minute.

[00:20:50]Yes.

[00:20:54]When you disable the breakpoint and not delete it so one thing is debuggers can just swallow breakpoints.

[00:21:01]So they can just, a breakpoint is hit but the debugger continues and doesn't tell you about it.

[00:21:08]So it could do that or it could.

[00:21:15]Yes, so in the debugger, the question was data structure set aside to remember what's disabled and what's deleted.

[00:21:22]And yes, the debugger will have some data structures to say okay well the user set this breakpoint and currently asked for it to be disabled but may want to enable it again later so just remember the address and then maybe just remove the breakpoint or just swallow any ones that come up.

[00:21:42]Is there another question, yes.

[00:22:08]Sorry can you repeat, I didn't quite understand the question Yes so, maybe this is a little bit misleading.

[00:22:22]I put the arrow I was supposed to have an arrow from the do int up.

[00:22:34]But it got lost somehow.

[00:22:35]But yeah the do int will send a signal to the debuggee and the waitpid is waiting for a signal on that process.

[00:22:43]Okay that's the last question for now, I'll take more later.

[00:22:48]Source level breakpoints, very simple example we want to break on main.

[00:22:54]So what we do is we look up our Dwarf information we find a die with the right name and we look at the low pc this gives us an address.

[00:23:07]We can then set a breakpoint at this address and it all works using what we just talked about.

[00:23:13]In reality, debuggers will skip over the function prologue things like that, but this is essentially what it's doing.

[00:23:21]So yeah, going from being able to break point on an address to certain breakpoint on a function like this.

[00:23:28]I mean you have to have a Dwarf parcer, but it's not a huge step.

[00:23:34]How about we have an overloaded function.

[00:23:37]So most things I've talked about so far have applied pretty much to assembly or nothing c++ specific.

[00:23:43]In c++ we can overload functions.

[00:23:47]So what happens if we want to set a breakpoint on just do_thing double and not do_thing string.

[00:23:54]So one thing, what's going to be different about these two functions when they're compiled?

[00:24:01]Mangling, exactly.

[00:24:03]So do_thing double will be mangled like so do_thing string we mangled like this.

[00:24:13]Because c++ is fantastic.

[00:24:17]So what we can do is look for something if we want to set on just the double one we can mangle the name and then just look for something with that linkage name look at the low pc, rinse repeat. Same as before.

[00:24:30]Yes, Victor.

[00:24:36]Yes, mangling can depend from compiler to compiler so if you're doing, there are other methods to do this I just picked a simple one.

[00:24:43]But yeah, this will only work if you knew that you're working with the itanium api.

[00:24:48]So if you've compiled something with msvc mangling and you tried to do this but mangled your debugger mangled using itanium abi then they would miss.

[00:25:07]Yes, so on Linux, Itanium abi for mangling is like defacto standard essentially.

[00:25:15]How about lines.

[00:25:17]So if I want to set a breakpoint on foo.cpp line four.

[00:25:21]Then we talked about dies and we talked about line table information.

[00:25:27]So this is line table information and we can see there is line four over here there are fewer entries for line four this could be because there are one line with multiple expressions, which generate multiple assembly instructions.

[00:25:46]So we wont' set one on the start of the statement so we look at this NS and set a breakpoint there.

[00:25:53]Have the address, it's the one on the left there Again, going from instructions to source is just a case of looking at Dwarf and mapping it back.

[00:26:04]Stepping, here's the answer to your question So the main types of stepping you want to do are over single assembly instructions, stepping over functions calls, so you don't want to go inside a function if you call it.

[00:26:21]Stepping in is when you do want to go and stepping out is when you want to finish this function or turn out.

[00:26:30]So on x86 with sufficiently high kernel version number you can just do this for instruction stepping.

[00:26:41]I will go into a bit more complex things you might have to do.

[00:26:46]But ptrace has a single step enumerator so you can just pass this and it will do an instructional single step.

[00:26:56]For stepping out, you can find the return address I'll talk a little bit about that when I go into stack unwinding.

[00:27:03]But you can find the return address and you can set a breakpoint there.

[00:27:09]Now if you're stepping in you want to set a breakpoint at the return address, in case you're not actually stepping into a function.

[00:27:17]Actually you're turning out and you don't want to just continue into cyber space and never gain control over your debuggee again.

[00:27:24]So you want to set the breakpoint at the return address and you want to set a breakpoint at the next instruction in this function or the callee.

[00:27:33]Great, simple.

[00:27:35]Not really.

[00:27:38]What is the next instruction?

[00:27:41]If I'm on some assembly instruction, how do I know where I'm going to go?

[00:27:49]Software breakpoints.

[00:27:51]You want to set software breakpoints but you need to know where to set the breakpoints.

[00:27:57]Yeah, so you might just be going to the next instruction you know, you're doing an add, you're doing a subtract, you're doing a move, you're doing another move.

[00:28:05]Simple, you just set a breakpoint at the next instruction.

[00:28:08]Maybe you're doing a jump, maybe you're doing a conditional jump, maybe you have no idea what's going on.

[00:28:15]So actually, answering what's the next instruction requires understanding your actual target.

[00:28:21]So in reality, you need to inspect the code to work out possible branch targets, set breakpoints there.

[00:28:30]What this essentially ends with is your debugger ships like an instruction emulator for your target.

[00:28:36]It's what LDB does.

[00:28:38]I'll talk a little bit more about how that makes a mess of some things, later.

[00:28:43]This is okay for now.

[00:28:46]Stepping over is very, very similar set a breakpoint at return address and the next instruction in this function.

[00:28:51]We still have that same question but the answer is the same.

[00:28:54]You know you need to work out your branch targets.

[00:28:58]Registers and memory.

[00:29:00]This is a useful thing if you're doing like really low level stuff, you want to be able to write or read your registers.

[00:29:07]Reading memory is useful for a whole host of problems.

[00:29:10]So reading registers is actually way more simple than you might expect.

[00:29:15]There's just a ptrace call for it.

[00:29:17]So you say, I have this user_regs_struct thing which I construct and I just pass it in and I get out something like this.

[00:29:25]Which just has a field for every single register which I might want the value of.

[00:29:33]Fairly simple.

[00:29:36]If I want to then write the registers, then I just do the read, I set some value and then I write it back.

[00:29:46]Not that magical, eh?

[00:29:49]For reading and writing memory it's the same kind of thing I have these peekdata and these pokedata calls.

[00:29:56]The unfortunate thing about this, can anyone see something, yes?

[00:30:05]Exactly, it's a word at a time.

[00:30:08]So if you're wanting to read a big amount of memory maybe I have a massive array which I want to show to the user.

[00:30:19]Or the user has requested I want a memory dump of like this entire bit of my program.

[00:30:25]Doing all that reading a word at a time is super inefficient.

[00:30:28]Because we have a siscall and we have a contact switch down into kernel mode and then we come back up every single word.

[00:30:35]So that's not great.

[00:30:37]So there are calls like process_vm_readv and write which can do multi word reading and writing.

[00:30:45]So this in one case where you can use ptrace but you don't really want to.

[00:30:53]Stack unwinding.

[00:30:55]So stack unwinding can get very complicated.

[00:30:59]There is in fact, an entire talk on exceptions in c++ and how the stack is unwound on windows that's James' talk later today so I recommend going to see this if you want a more comprehensive view.

[00:31:13]I'm going to cover the kind of most simple case it should give you a flavor for the kinds of things which will actually occur.

[00:31:24]So this is what a stack could look like for x86 64 given the system 5 abi.

[00:31:31]So you have all your arguments, your return address the old frame pointer, so the pointer to the previous stack frame.

[00:31:41]And then your local variables.

[00:31:45]Everyone okay with this, any questions?

[00:31:49]Cool.

[00:31:51]So we have a code like this.

[00:31:56]Bar plus foo, foo, hi I'm going to set a breakpoint, here on the cout.

[00:32:07]Then what are we in? We are in some stack frame we don't really have any information apart from that.

[00:32:14]So if we want to show the user how we actually got to this function and what functions did we call on the way then we need to walk up the stack.

[00:32:25]So what we can do is you can say, okay well we have the frame pointer for the previous stack frame.

[00:32:32]So we can just go back.

[00:32:35]Now we found another stack frame but we have no idea what it's for.

[00:32:40]Like, above this return we have some arguments and then we'll have some local variables and things like that.

[00:32:48]But we don't really know what this thing is it's just stack frame.

[00:32:51]So we can use the return address to look up the Dwarf information.

[00:32:57]Because the dwarf information says okay here's the range that this function's code lives at.

[00:33:03]If I know my return address I can say okay it doesn't match this function, it doesn't match this function, here's the function we're in.

[00:33:10]We know it's between this low value and this high value.

[00:33:15]And that's how we can find out okay, our last stack frame was bar.

[00:33:19]And we can then send that to the user.

[00:33:22]And then we just do the same thing again we go back along the frame pointer, check the return, and we know we've done baz call, bar call, foo.

[00:33:34]Then we stop when the frame pointer is zero.

[00:33:38]In reality above this, we'd have main and like underscore start and things like that.

[00:33:43]But for sake of brevity.

[00:33:45]So this is how stack unwinding would work if you have easy access to frame pointers.

[00:33:52]Sometimes you don't.

[00:33:55]Expression evaluation.

[00:33:56]Now you might notice as I get further into this talk things get more and more complicated And Expression evaluation could certainly be an entire hour talk on it, it's multiple talks.

[00:34:08]So I'll talk about the kind of broad aspect of it.

[00:34:11]We want to print my integer.

[00:34:15]It's a local variable has some value, it's on the stack.

[00:34:21]Fairly easy to get a hold of.

[00:34:24]So what we can do is look at our dwarf information.

[00:34:28]Dwarf information is so handy and this tells us its names we've found the name, we might have multiple things named in the dwarf information you have scopes so you can work out which scope you are in and which is the right one.

[00:34:41]And this tells us where our variable is located so this might be, oh it's just in register n.

[00:34:52]Or it could be, in this case, it's an offset from our stack frame.

[00:34:57]This thing is stored on the stack.

[00:35:00]So we then need to go and find where our stack frame is and offset it by negative 8 and we'll find our value.

[00:35:05]And then using ptrace or the vm process then we can go ahead and look where that address is and read the value.

[00:35:16]Yes question.

[00:35:26]Yes, so the question was if something is relocated like if it starts off in register 12 and then is moved to register 17 because the register allocator decides it needs to move things around is that reflect the dwarf information.

[00:35:41]The answer is yes.

[00:35:43]There are various different ways that dwarf information can represent addresses, and one of those is list of ranges.

[00:35:52]So it can say between this pc and that pc it lives here.

[00:35:56]Between this pc and that pc, it lives over there.

[00:35:58]Here it's on the stack, here it's on the register and that can all be represented by dwarf.

[00:36:04]Answer question?

[00:36:06]Yeah.

[00:36:09]And yes, so I just mentioned that there are various ways for dwarf information to represent addresses.

[00:36:15]So this becomes annoying for debuggers to handle because, you know, maybe my frame base is just in register 6 and I can go read that register and I can offset it and it's all fine.

[00:36:27]Maybe I need to go look a the call frame information this is like a section in our ELF, which stores information for stack unwinding for exceptions.

[00:36:37]This is like the real way which you could do stack unwinding in the absence of pin pointer information and things like that, but it gets quite hairy so I'm not going to show that.

[00:36:48]And this example right in the bottom shows the ranges.

[00:36:52]I didn't really expect people to be able to read this but this just showing that there are a bunch of different ways in which it's represented and that's how the ranges kind of look when you dump them out.

[00:37:06]Getting a little bit more complicated we have some local variable my_int and another one called a and we went to multiply them.

[00:37:16]So we could kind of do a lot of stuff in the debugger but what some things do, this is what LDB actually does generates function, so this has some special magic in it to say what local variables I'm using and things like that.

[00:37:35]It generates a function, generates llbmir from it.

[00:37:38]You don't need to understand what this is doing just note that it exists and this is actually what lldb actually produces.

[00:37:46]And then that ir is interpreted to get out the final result.

[00:37:51]So the debugger isn't having to implement its own c++ expression parser and evaluator.

[00:38:02]If you want to do something like call a function maybe this function has side effect.

[00:38:07]Maybe we want the side effects to happen maybe it's going to change some values or print something out and we want this to actually be occurring as if we called the function in our code.

[00:38:19]So we can create some ir and interpret that because the side effects won't be properly represented.

[00:38:27]So instead we can compile ir, lower to machine code, map this code into the address space of our debuggee and then execute that function.

[00:38:47]Just by manipulating our registers.

[00:38:51]So this is how you actually end up calling functions within the address space and having all the side effects work.

[00:38:59]So it's pretty interesting.

[00:39:01]There's a whole lot of research in to various ways to achieve this and it's really cool.

[00:39:09]If you go and Google some stuff afterwards or come talk to me or some other debugger experts here.

[00:39:14]Yes, question.

[00:39:25]So the question was do the debugger need to know how the debuggee was compiled.

[00:39:31]I'm trying to think if it would need to do anything to match calling interventions or abi or anything.

[00:39:41]Probably not because you can find the address of functions by looking at symbol table or the dwarf information.

[00:39:49]So you know where the functions are gonna live.

[00:39:53]You just need to make sure that the addresses are correct.

[00:39:59]Does that answer?

[00:40:02]There's another question here I think.

[00:40:09]Yes so if our expression that we're, like if we're not calling a function here but say this expression is inset or like right into a memory address then yes we have to do the same thing.

[00:40:27]Why do we not always do that?

[00:40:29]I think this has changed in lldb sometimes I think it used to do more interpretation than it does now.

[00:40:37]But I'm willing to be proven wrong on that.

[00:40:40]I think maybe it now just always gets it.

[00:40:44]But yeah I would have to check.

[00:40:47]Victor had a question.

[00:40:52]What if the functions in line that's a very good question.

[00:40:54]So dwarf information does give you information about in line functions but a lot of the time not very useful because in line functions are subject to other things other than just in lining because you have more local information.

[00:41:10]I do have an example coming right up.

[00:41:13]So just a second.

[00:41:19]Yes.

[00:41:26]So I think this is how it gets mapped in they can get calls and m-map essentially and then writes into the map memory.

[00:41:41]It has to generate a call to m-map that it calls m-map within the address space and get the address out.

[00:41:55]So the functions actually in the debuggees.

[00:42:03]Yeah come talk to me after.

[00:42:10]This is getting to the end lining.

[00:42:13]How many of you have tried doing something like this?

[00:42:17]Quite a few, did it work?

[00:42:20]No? Did you get something like this?

[00:42:23]Yeah.

[00:42:25]This happens a lot.

[00:42:27]Especially with templates everywhere in c++ things get in lined, you don't have a definition for the function so you what do you call?

[00:42:37]It's not there.

[00:42:39]Pardon?

[00:42:42](laughter) Who are you gonna call, yes So what you can do is just write this So you could just write this in some translation unit output actual definition for this in satiation I have actually used this quite a lot it's awful but it works quite nicely.

[00:43:09]So if you want to be printing out a bunch of expressions which are requiring functions of class templates which would otherwise not have definitions outputted then you can make sure definition is outputted somewhere and then the debugger can call it.

[00:43:28]So this is like a hacky debugger tip.

[00:43:33]Multi threaded applications, again could be like an entire course.

[00:43:40]So you could do ptrace has options you can set.

[00:43:44]So you can say I want to get a trap every time the clones is called.

[00:43:50]And that is what happens when you create a new thread.

[00:43:54]And then there's some magic you can do to say oh what's this.

[00:43:58]Sigtrap from a clone call and if it was then you can work out what the new pid is and add it into your internal data structures and things like that.

[00:44:08]So this is like the thousand mile view of what the setup is for dealing with multi threaded applications.

[00:44:17]Shared libraries, Oh sorry, Victor, yes?

[00:44:31]Yeah that's the question I didn't want to get into much.

[00:44:35](audience laughter) The question was when you set a breakpoint do all threads stop, does just one thread stop the answer is complicated because debuggers will have different modes which allow you to have non stop mode or all stop mode and some dealers do it better than others.

[00:44:54]Yeah it's another big topic which I didn't quite want to cover.

[00:44:58]Yes.

[00:45:01]Isn't it wrong to stop only one thread?

[00:45:03]Maybe, maybe not.

[00:45:04](audience laughter) Well if you stop only one thread maybe your gooey can keep running.

[00:45:25]Right yeah, if you have a multi threaded server and maybe you're debugging infraduction.

[00:45:31](audience laughter) Maybe you're a terrible person.

[00:45:37]Then you can do this, yes.

[00:45:40]Yeah come talk to me after.

[00:45:43]Don't debug infraduction.

[00:45:46]Okay shared libraries this was something in which I didn't understand for the longest time.

[00:45:51]Because it's really badly documented until you find the header file which explains everything and it's great.

[00:45:59]But if you wanna trace when shared libraries are loaded and unloaded because maybe you, if you use shared libraries at all you've probably got to the situation where you want to set a breakpoint on something in the shared library and the debugger says oh we couldn't find this do you want to wait and we'll try and set it if the library turns up.

[00:46:17]This happens a lot.

[00:46:19]This is how it's implemented because it has to know when something is being loaded.

[00:46:23]Because maybe you just hit a breakpoint on function you continue.

[00:46:32]If shared library has loaded with that function in it you want that breakpoint to be hit rather than just waiting for the death of the universe or the program terminates or something.

[00:46:43]So somewhere in your program there is a way to get this somewhere.

[00:46:50]There is this kind of data structure which has a bunch of tags and information about your program.

[00:46:58]This is like dynamic process information.

[00:47:02]And one of these entries points to something like this, r_debug.

[00:47:09]Now the most important things in this are this link map which tells you every single shared library which is loaded, its name, where it's loaded.

[00:47:18]So this is the kind of thing we want to be, it's a link list.

[00:47:21]So we want to walk the link list and find what our shared libraries are.

[00:47:27]The really cool thing, at least I think it's really cool I'm really nerdy.

[00:47:31]Is this.

[00:47:33]So this is an address of a function which will get called every time a shared library is loaded or unloaded.

[00:47:42]So what can we do in order to read the link map every time it's something our debugger can do.

[00:47:51]Set a software breakpoint.

[00:47:53]So we set a software breakpoint at this address and every time the function is called we know that the library is being loaded or unloaded we can then walk this link map, see if there are any differences.

[00:48:06]There's some ways for the operating system to tell if there are changes, things like that.

[00:48:12]But we can essentially walk it up our data structure, set break points.

[00:48:16]And this breakpoint is hidden from the user.

[00:48:20]So the debuggers are setting breakpoints all the time without you actually noticing it.

[00:48:25]But they're doing it to do things like this.

[00:48:27]Trace, shared library loading and it's all hidden.

[00:48:32]There are some ways to get the debugger to show you these things.

[00:48:35]I think lldb has a option to print internal breakpoints so you will see things like this being set which is pretty cool.

[00:48:45]So this kind of how it goes the debugger sets breakpoint, magic function, when it's hit you walk through the link map, update data structures.

[00:48:52]Remote debugging.

[00:48:54]So this is kind of how a remote debugging session looks.

[00:48:58]You have the debugger on your host, you have a debug stub on the target.

[00:49:05]Now this debug stub is supposed to be, supposed to be, a very thin wrapper around the operating system, debug interface.

[00:49:14]Like ptrace.

[00:49:15]In reality, this thing is massive.

[00:49:18]Like, lldb debug stub for x86 ships like an entire instruction emulator.

[00:49:24]So yeah, it's not really a tiny thing anymore which is a bit frustrating.

[00:49:30]The idea is that the debugger communicates with the debug stub, the debug stub operates with the debuggee in some os specific manner like ptrace.

[00:49:41]And then just everything sent between the debugger and the debug stub.

[00:49:47]Now, the interesting thing is your actually using this all the time.

[00:49:52]Because this how quite a lot debuggers operate, just doing local debugging.

[00:49:59]Because it means it's a whole lot easier to implement a bunch of this stuff.

[00:50:03]You just always have a debugger connecting to a debug stub.

[00:50:07]It just so happens that the debug stub is on the same machine as you run on the debugger.

[00:50:13]So this is how lldb operates all the time for example.

[00:50:16]I think GDB does as well but I don't know GDB as well.

[00:50:22]Yes?

[00:50:31]That's a very good question.

[00:50:32]What if the debug symbols are not available?

[00:50:34]And yes debug symbols should generally, are not available on the debug stub because the debug symbols information is high level information which only the debugger has access to.

[00:50:47]The idea is that the debug stub should just be handling addresses and super low level stuff.

[00:50:53]So generally the debug stub does not have access to the debug info.

[00:50:58]It's just the debugger works out all the address and sends it to the debug stub.

[00:51:04]Okay.

[00:51:14]Yeah, so I'm going to show how it sends request just now.

[00:51:17]So there is a remote protocol called the gdb remote protocol.

[00:51:22]That's what gdb and lldb use.

[00:51:24]So it looks kind of like this.

[00:51:27]This is just blown up a bit, so I can explain all the different parts.

[00:51:32]So you have packet start, some types which is a little identifier.

[00:51:37]Arguments and then check sum.

[00:51:38]So this is just sent over a network or over any kind of communication channel.

[00:51:45]So for example, this is a debug breakpoint packet.

[00:51:50]Which sets the breakpoint at some address and there's an architecture specific thing I can't remember what that's actually used for on x86.

[00:52:01]And that's how you write, or that's how c++ debuggers work.

[00:52:06]Thank you.

[00:52:16]Okay I have resources there if anyone would like and I will now take questions.

[00:52:19]Yes.

[00:52:24]Why can't we properly debug over fork.

[00:52:27]This annoys me too.

[00:52:29]GDB has a very, very good, well comparatively very good follow forks mode lldb does not.

[00:52:36]So the gdb follow forks mode has, in my experience mostly worked okay.

[00:52:40]So it's just something you can set and say okay when my program forks, I want to follow my child rather than the parent and maybe you had bad experiences with that.

[00:52:53]You want it to follow both.

[00:53:02]Right, yeah.

[00:53:05]I haven't thought much about debugging both at the same time.

[00:53:08]Maybe modern GDB has more support for that or something.

[00:53:30]Yeah GDB just connects to the child as soon as it can rather than doing something nice.

[00:53:40]I know some people who have worked on follow forks mode stuff.

[00:53:47]But can't remember the details.

[00:53:51]I'm not so sure about that part myself.

[00:53:54]Other questions, yes in the back.

[00:53:57]Sorry can you come to the microphone.

[00:54:05]- [Man] How do watch points work?

[00:54:07]- So watch points you use the Hardware break points because you can set them on reading and writing as well so you say I want a set watch point at this variable I know where that variable lives because I have the dwarf information and then I can set harder breakpoints to trap on reading and writing.

[00:54:23]- [Man] So you can only do that on registers then?

[00:54:26]- No the registers are having addresses written into them.

[00:54:31]So you're watching addresses.

[00:54:36]Other questions? Victor?

[00:54:39]- [Victor] So you have a protocol between the debugger and this stub.

[00:54:45]Is that standardized or.

[00:54:48]- Yes, well it's partially there is a document which shows all the different gdb remote packets, I think I have a link to it in my resources.

[00:55:02]In reality, debuggers will have their own extensions so lldb has its own packets which gdb doesn't support.

[00:55:12]The packet format will have multiple versions so if you're using different versions of gdb or your debug stub, then maybe there's a mismatch and things like that.

[00:55:24]So it's not like a standard, but it's documented at least.

[00:55:30]Or at least most of it's documented.

[00:55:33]Yeah.

[00:55:36]- [Man] I think somebody over there was actually trying to ask this earlier.

[00:55:40]Maybe I just didn't understand the response but some debuggers have a pause button, right?

[00:55:46]How are you inserting a breakpoint wherever you happen to be at the time.

[00:55:51]- Okay, so the pause button on debuggers usually just sends like a second to the debuggee process.

[00:55:59]So it's not like taking a breakpoint wherever you are it just sends a signal and that will stop the process.

[00:56:08]- [Man] Question about core dump debugging works.

[00:56:13]We don't have a process that will work.

[00:56:16]- Yeah for core dump debugging, I never looked into a whole lot.

[00:56:24]My naive understanding is that you just have your core dump will have enough information to give you some experience and then your debugger is responsible for using that information as best it can to provide.

[00:56:54]- [Man] Petrie skulls to get registers to get memory are replaced by reading from core dump, right?

[00:57:02]- Right, yeah, as I said I haven't looked into core dumps much but I would guess that's what's done.

[00:57:17]So if you evaluate an expression which results in an exception I guess when you're getting the call to the function then the compiler which you're using, would have to I would put code to handle to deal with the exceptions.

[00:57:39]Like report back to the user that an exception was thrown and not just end up terminating everything.

[00:57:55]Sorry can you stand closer to the microphone?

[00:57:59]How does the debugger know whether it's an exception.

[00:58:03]Well I mean the debugger is using like a full on compiler so the lldb calls aren't inclined to actually compile the expressions which you use.

[00:58:17]So it's not one of these things where it's just hacked together.

[00:58:22]C++ subset is like actually using clang so it has everything in clang available.

[00:58:41]Sorry can you say that again?

[00:58:43]- [Man] Do you know why the Linux kernel only got single step support recently because x86, I think has single step hardware support.

[00:58:50]- Yeah do I know why ptrace only got single step recently I mean it wasn't recent, recent.

[00:58:54]This was a number of years.

[00:59:00]I can't remember exactly what kernel version it changed at and I'm not sure why it took longer maybe it was just ptrace didn't have that actual part available.

[00:59:11]Before then to do single step you'd have to do something like the branch target.

[00:59:17]Finding and then taking breakpoints there.

[00:59:25]- [Man] When debugging application that was built with clang sometimes I can't see the internal representations of memory but I can with gcc.

[00:59:32]Is there anyway I can instruct the debugger what I've built with so that I can give it better, rich information.

[00:59:37]- Sorry can you say that first bit again - [Man] Oh so sometimes when we build an application with clang you can't really see the internal data but we go to gcc and see your strings for whatever you need is there anyway to instruct the debugger like hey I'm building a clang, use clang internals to represent this data - So yeah the debugger will know what you've compiled with because it will be set in the start of the compilation unit and it said which compiler is being used.

[01:00:06]So will have that information available to it, it's just a case of what it does with that and whether it implements all the necessary machinery to do that.

[01:00:18]Thanks, one more question.

[01:00:19]We have seconds left.

[01:00:22]- [Man] How do you debug when the variable is optimized away - When a variable is optimized away, then the debugger will just often say this is being optimized away.

[01:00:32]So if it's not there, you can't visualize it.

[01:00:36]- [Man] So then do you add printer statements for the variable and not make it get optimized - Yeah you could, in your program, try to do things to stop it being optimized away like stick it in a volatile or something like that.

[01:00:54]Okay thank you very much.

[01:00:56](applause)