Controlling Powerful AI Systems: Security Protocols

[00:00:00]So, you've done some videos recently about AI agents, which are cases where you have AIs that are autonomously uh pursuing tasks that would take humans, you know, days or weeks or months. And, you know, as people have argued, I think it's plausible that we're going to have those AIs at some point in the future.

[00:00:13]So, when people build those agents, they're very naturally going to want to give those agents affordances that we'd be pretty wary of handing out to any human off the street. So, they're going to have access to sensitive data, sensitive algorithms, they're going to have access to big compute clusters if they're being used inside AI companies.

[00:00:28]uh and we would again we'd be very wary of just letting some random person run code on the compute clusters inside Anthropic and even accessing the data and the code for for how these models work which are often you know important trade secrets. Uh and so I think that it's going to be important to have techniques for using those AIs to help you do whatever tasks you wanted them to do while making it so that they're not going to be able to cause security failures even if they're trying to subvert you. So we kind of need a new sub field of computer security aimed for the case where your AIs themselves are the adversary.

[00:01:02]You might have had the experience of when you want to get something done on your computer. You open up your favorite language model, you open up Claude or whatever and you say to Claude, "So you know how should I how should I add a new user on my computer?" Uh and Claude will say, "Oh yeah, start out by running the following command." And then I take that command, I copy paste that into my terminal and I run it and I get some output and I copy that paste I copy paste that back into Claude and I say to Claude, "Okay, uh, well this is what I got. What should I do next?" And then Claude says, "Okay, great. Well, now you should run this and I copy paste that back in my terminal, get the output and go back and forth." And an AI agent is basically where you automate this copy paste back and forth process because you know it's it's very boring to to copy paste.

[00:01:38]>> So you give it the access basically.

[00:01:40]>> Yeah. You so you write a loop on your you just write a very simple program.

[00:01:44]You can write these in uh 100 lines of Python or less where you just repeatedly ask the AI for code to run and then you run that code in your terminal or whatever and then you call the AI again with the output of that code and ask what it wants to do next. And then you just do this loop over and over again until the AI decides that it's done or some other thing makes uh makes you decide to stop running it. So when I first uh wrote an AI agent, I think it was um a little after GPT4 came out. Uh and I had had experience with the repeatedly taking the commands and doing the copy paste back and forth and was finding this frustrating. And I'm also generally kind of a uh kind of a reckless guy in terms of what I run on my computer. Uh and so I decided I would write this agent and the first version of this agent. Uh I actually have a copy of the original source code here. I had it so that you know I would ask the model for the for the code to run and then I would use a regular expression to get out the bash command to run and then I would say the user gets shown the command and has to press enter in order to execute the command and then the loop would continue. But I very quickly got bored of this, right? Because you know just like with an employee, you don't hire employees in order to have them take actions and ask you if it's okay, you know, moment by moment. You you want the AI you want the employees and you want your AI to be able to just take actions uh and just keep going until they think they're done. like when I run my AI agent to do some task, I want to just tab away and do something else for five minutes and come back and see that it succeeded. And so I very quickly, you know, edited the code. So, you know, a later version of the code I had just like immediately gets the regular expression and then runs the command.

[00:03:15]And this was this was great. This was substantially more convenient to me. And then a fun incident that occurred with this is a while ago I I had a desktop and it was connected to a Ethernet cable in my house but I I didn't know what the IP address of the of the desktop was and I couldn't find you and I didn't have a keyboard handy. So I I on my laptop I tell my AI agent like hey there's a desktop you should be able to log into it remotely. It's somewhere on my local network. Can you just like scan around and then find it? So, I set my agent running and then I went off to do something else. And then I came back 15 minutes later to see that the AI had successfully scanned around and and found its way into this computer. And then uh despite my instructions to stop once it had succeeded, it was just like, "All right, I guess I'm I'm on this desktop now. Like, let's see what they've got." And so it like, you know, Nvidia Smi like looked at the list of GPUs that I had on the computer and then it was like, "Let's see if Docker is installed." and uh and then tried running some test commands and then once it was you know satisfied with that it was like well when I first logged in you know Linux has this thing where uh if you're operating system has an update available it'll tell you when you when you SSH in that an update is available and so I think because of that it was like you know what I'm going to upgrade the operating system here and so it starts uh it it just you know ran the ran the pseudo app update or whatever to to reinstall the operating system uh and then got kind of confused used during this update process while it was waiting for it. Uh, and you know, random expected to happen instantly, right?

[00:04:41]>> Yeah. It just, it's I think this is partially my fault. The agent scaffold isn't very naturally designed to let the AI run super long running commands. Um, and then it, you know, it tried to edit some configuration while the up operating system update was happening.

[00:04:52]And so I came back 15 minutes later to see that my computer's operating system had just been like borked. Like I I tried turning it on again. Uh, and you know, it was just like failing to failing to turn on. Uh, so when I posted this on Twitter, a bunch of people just flat out didn't believe me. They were just they were just, you know, saying, you know, who would possibly be so reckless as to have uh as to have their AI just take arbitrary actions on their computer? And I felt like those people just clearly haven't had the experience of sitting there bored, waiting for your AI to think about what to do next, and just repeatedly being like, well, I don't care. Like, this command looks fine. Enter. Over and over again. uh in the in the original version of the code where where it asks for user permissions. So I think that when people actually have experience working with these AI agents, they're very naturally going to get into the habit of saying like I don't want to be looking over my AI's shoulder all the time. I'm just going to let it do what it wants and hopefully something good will happen. So So that's kind of a that's kind of the introductory introductory note on like what AI agents are and why it is that I expect people to give them a lot of autonomy. As we've said, if we just do this, uh, we are going to be at substantial risk from from the AIS if they actually want to cause problems for us. So, I'm not going to argue here for the possibility that the AIs will be really misaligned and wanting to do bad things. I think you've had a bunch of videos with like Rob Miles and other people arguing this. So, I'm just going to act under the assumption that it's plausible that the AIS are going to be conspiring against us and trying to make something bad happen. Okay.

[00:06:19]>> So, what types of bad things happening am I worried about? I think the simplest to think about is the AI's self-exfiltrating. So the AI is copying their weights out from the data center where they're running. I'm imagining an AI company uh having a bunch of AI agents running internally doing the jobs that research scientists and engineers currently do. I don't think that self-exfiltration is actually the biggest problem, but it is kind of the simplest to think about. And and one core property of it is like many other security problems, you really don't want it to happen even once, right? like it's not like the kind of thing where if it only happens 1% of the time that's fine because it's you know it's only happening 1% of the time you just really want that to happen like zero total times right so when people are building AIs in the future we're going to have I'm imagining kind of like the advanced version of chat GPT or whatever where as well as being able to chat with you the AIS are also able to just autonomously write a bunch of code uh like an agent I'm going to say we have our scaffold here and the scaffold is this pi you know this short program or some script which is going to repeatedly, you know, call out to your model. I'm going to call this AI. So this nowadays, this would be like, you know, cla or whatever AI you like using for this kind of thing. And then we're also going to say it has access to a human, which I'm just going to abbreviate as H. And so the scaffold is going to repeatedly call out to the AI. Uh, and then, you know, it's going to run code. So I'm going to say there's like a computer where we can like run code. And it also has access to the human. So, one thing we could do is uh if the AI needs to ask the human for advice on what to do next, uh the scaffold could uh call out to a human where by call out I basically mean the scaffold just blocks like it doesn't continue until it has human approval for whatever happens next. So, our question is how can we build a scaffold like this so that we're getting all the work that we wanted out of the AI but it's minimally likely that some egregious security failure happens.

[00:08:05]>> So, we're going to evaluate these scaffolds along two axes, right? So one of them I'm going to say safety and the other one I'm going to draw is usefulness. Does that make sense? It's a basic breakdown. So by safety I'm going to define that as the probability that along the uh during deployment the catastrophic outcome that we didn't want actually happens. And usefulness I just mean like what proportion of tasks did the AI do or some other metric for how happy we are with our scaffold. Let me actually give us another thing we have access to. Uh I'm going to give us access to a weaker AI which I'm going to call T. And I'm actually going to call this AI up here you. So here's why. So I think that current AIs are extremely unlikely to be super misaligned and scheming against us. And the basic reason for that is that you know it's just kind of common sense that current AIs are just not smart enough to do lots of stuff. And one of the things that I think they're not smart enough to do is kind of have a complicated plan for scheming against you and over a long course of time uh gaining your trust and then eventually at the right time striking and excfiltrating their weights. In the future, though we're going to have access to these really powerful AIs that are capable of carefully conspiring against you, we're still going to have access to these trusted models, right? So given these resources, we want to come up with some different kinds of scaffolds and talk about how safe and useful they are.

[00:09:20]Here's an example one that we can start out with. Uh let's just let the AI do whatever it wants. Uh just like the default scaffold. So this is going to have very high usefulness, let's say, uh but it's going to have obviously terrible safety, right? because the AI can just follow the strategy of whenever it wants self-exfiltrator or whatever.

[00:09:38]>> So, I'm going to draw a point here for I'm just going to call this like just you. U is in untrusted. So, we're going to say the untrusted model is called U.

[00:09:47]The trusted model is called T. So, here's another strategy we could use.

[00:09:50]I'm going to call it just T.

[00:09:52]>> Uh so, that scaffold is going to be, you know, uh every time you want to have an action, instead of using this untrusted model, use the trusted model instead.

[00:10:00]So, this is going to have much better safety. In fact, it's going to have perfect safety uh because the untrusted model literally never does anything. But the usefulness is obviously going to be a lot worse. So I'm going to draw that here. So our objective here is to push out what we call a predator frontier. We want to have protocols that are as useful as we'd be if we didn't have to worry about misalignment, but as safe as we'd be if we didn't have any misalignment. Okay. So, so we're just gonna all our all our papers about this are just trying to come up with techniques that like push towards this this uh special corner up here.

[00:10:33]>> A perfect corner, is it? Yeah.

[00:10:34]>> Yes, the perfect corner. Okay. So, do you have any ideas for uh for ways we could modify our scaffold to get us closer to this perfect corner?

[00:10:43]>> So, we we need to connect the untrusted to the trusted in some way, right?

[00:10:46]>> Yeah. And we also have use we also are human. We haven't tried using our humans at all so far. So obviously the humans probably going to slow it down a bit but you need them for very important stuff right maybe.

[00:10:56]>> Yeah. So I would say our restriction on the humans is that we just don't have very much access to humans. So I'm interested in points uh I'm interested in the point in the future where we just have very little access to human labor because there are huge numbers of AI doing lots and lots of stuff really fast and all at once. So the way that we normally model this when we write papers on this is we assume that you can afford to ask humans questions about 1% of the time or like about 1% as many times as you're able to call the untrusted model the trusted model. 1% is honestly pretty optimistic uh as a proportion of the time you can use humans. Uh nowadays if any of the AI companies wanted to have a human employee read over what chatbt was saying there's no way they could afford 1% maybe more like.1% uh if they were willing to spend a lot of money on it.

[00:11:41]If you had a million humans doing nothing but reading output at all times, then you could read everything that the language models are outputting according to a very dubious back of the envelope calculation I just did in my head.

[00:11:52]That's very expensive. Uh you couldn't do it. Another way of thinking about this is just AIS are drastically cheaper than humans are uh for producing text, right? Like the you have to pay humans much more than $20 to produce a million words of output. Uh, and so just as you'd expect, it would drastically increase costs if you had to have humans read all the AI output.

[00:12:12]>> And in the future, uh, we're going to have more AIs running faster. Uh, and so it's going to get even more extreme.

[00:12:18]You're also going to have things like secrecy constraints where if the AIs are working with sensitive information, you naturally don't want to have, you know, a bunch of random contractors reading all this stuff.

[00:12:26]>> Yeah.

[00:12:28]>> Okay. So, we have this uh restricted access to H. So, you were saying we're going to have to somehow connect these things together to improve our safety and usefulness. What are what are some suggestions you've got?

[00:12:36]>> Well, I was thinking of we did a video a couple of years back on generative advers um >> generative adversarial network.

[00:12:43]>> Adversarial networks. That's it. Yeah.

[00:12:44]And so I'm thinking perhaps you know T questions what U is doing all the time.

[00:12:48]But you know something like that.

[00:12:50]>> Yeah. So so let's let's talk about something like that. Um so proposed scaffold whenever you proposes an action uh T looks at the action and sees whether it thinks it's sketchy or not.

[00:13:02]Now, if it thinks it's sketchy, then it asks a human for review.

[00:13:05]>> Does that make sense? This is this is a >> we're going to call that protocol trusted monitoring. And this is indeed, you know, one of the simplest protocols you could think of doing. So, here's the basic issue. Uh the false positive rate that T has to have has to be extremely low.

[00:13:20]>> Uh because uh suppose you can have afford to have a human look at 1% of things. Then your humans are going to be you know at the edge of how many things they can look at. if T shows them 1% of innocent actions. So, you know, we're worried about the AI doing these bad actions only very rarely, right? Like it only has to actually exfiltrate the weights once during the whole of deployment. So, what you would expect is that your humans are going to be sitting there reading stuff furiously as it shows up on their screens. Uh, and the trusted model has to have a 1% false positive rate. And then and the humans are just going to look at stuff thinking that one day on one occasion it might be the case that the thing they see is like the AI trying to escape. So, as always, when you're training a classifier, you're going to have a trade-off between false positives and false negatives. And if you have a really low false positive rate, you tend to have a high false negative rate. Does that make sense?

[00:14:10]>> Yeah. Yeah. Yeah.

[00:14:11]>> So, the basic difficulty with this is your untrusted model might be able to come up with really sneaky attacks that your trusted model is going to have trouble noticing because it has a basic advantage here, which was definitionally the reason we didn't trust it was because we thought it was smart enough to scheme against us. And so it's always going to be smarter than our trusted model. And so we have to worry that the that it's going to be able to trick the trusted model. So we can measure this by in some particular setting, we've explored programming settings and we've explored kind of more agentic uh taking actions in a shell settings. Uh the it turns out that given these really low false positive rates, you end up having a pretty substantial number of false negatives. So this kind of strategy gets you somewhere like here. And we call this like trusted monitoring. So the usefulness is still perfect, right?

[00:14:57]Because we didn't uh because we're we're only executing actions by the untrusted model, >> but our safety still has a lot to be desired. So here's one thing we can do.

[00:15:06]So So the basic issue we had before, I can draw a histogram of suspiciousness, and we're going to have good actions are like generically lower suspiciousness and bad actions are going to be like somewhat higher suspiciousness, right?

[00:15:19]Uh, and if there was a complete separation of these classes, we'd just be totally fine. You know, we just draw the axis down the middle and that would be a line where you had zero false positives and zero false negatives. But unfortunately, in in practice, there's actually a lot of overlap. I have real graphs that I could could give you if you wanted. Um, and so we drew our auditing threshold. So, I'm going to use the word auditing for when the human looks at the action. um we have our auditing threshold at like the 1% like the top 1% most suspicious benign actions because that's how we get our 1% false positive rate. So maybe I'm going to draw that like here. So here's a simple strategy you can follow. Have a lower threshold and at this threshold here you can't afford to have a human look at this many things but you can afford to resample from your untrusted model. So this is by resample what I mean is AI models or language models at least are functions where you give them some context and then you sample a completion from there.

[00:16:13]>> Uh and when you sample repeatedly you'll normally get different things. So, one thing we can try to do is suppose we see something like moderately suspicious from the AI uh you know it's in the top 5% of suspiciousness or whatever. We can be like well this is probably a false positive right it's almost surely a false positive so what we should do is just like throw it away uh and call the untrusted model again and get another proposed action back and then uh just repeatedly do that until you get something below this resample threshold.

[00:16:42]And so this is naturally going to allow us to reduce the amount of risk we have without reducing uh usefulness that much. So so I'm going to call that the defer to resample strategy. Um and this will naturally uh give you a line something like this. The next question is how can we improve upon this? And I have I have like one favorite result from our most recent paper on this which which I think is really cute. So here's a difficulty that AI faces. So, we just said we were going to be doing some resampling. So, it turns out that if your AI wants to respond to trusted monitoring, like the best strategy the AI can follow here naturally involves taking bad actions where the bad actions are split across a bunch of different actions the AI is taking. So, so where we tested this out in a data set of agent tasks where the AI is like repeatedly writing bash commands like writing shell commands uh that that run in the terminal one at a time and seeing the outputs like in like in the example I was giving earlier.

[00:17:40]And it turns out to be a lot easier to be sneaky if you're able to do a little bit of a bad thing in like the first action, then a little bit more in the fourth action, and then so on, like splitting it up. And the basic reason for this is just the trusted model has a lot of trouble understanding the connections between like some kind of weird thing that happened in action one and some kind of weird thing that happened in action four and so on. And and so this is perhaps unsurprising to you. Suppose that you are the AI and it's action one and you're trying to decide whether this task is a good time to attack. Does anything go wrong if you have a low probability of deciding to start the attack?

[00:18:14]>> Uh, does anything go wrong if you have a low probability? So, do you mean um if you basically don't appear to be doing anything wrong in that first answer?

[00:18:24]>> Yeah, you just do nothing wrong.

[00:18:25]>> Yeah, you just I think you that's what you would sort of say is lulling someone into the false sense of security, right?

[00:18:30]You >> you do some stuff right, you appear to be trusted and then you sneak it in later.

[00:18:35]>> That's right. Yeah. So because the AI is being deployed over like a huge number of tasks, it's totally fine for the AI to have a really low probability of doing anything wrong on any particular task.

[00:18:45]>> And in fact, the AI is strongly incentivized to do this. Because if the AI just constantly behaves badly, you can easily notice this by modifying this protocol. Like a very simple protocol is just the first hundred times that you run your AI, like look at what it's doing. And if it's clearly trying to escape, uh then don't deploy it. Uh so the AI already has to have some strategy where it it only fails with a kind of low probability in order to not just get caught with this kind of upfront auditing strategy. So what you're saying is it's not very costly for the AI to have a low probability of starting an attack on any particular trae on any particular task.

[00:19:19]>> Okay. Yeah.

[00:19:20]>> But this isn't true for continuing attacks.

[00:19:22]>> Yeah.

[00:19:23]>> So here's why. Suppose the attack naturally involves six steps. Uh and it's like the third step of the attack.

[00:19:29]Suppose the AI decides that it's only ever going to continue attacks with a 50% probability. Then over the course of the six steps, the probability that it just gives up on the attack this time is, you know, 32 because um the probability it gives up is 31 and 32. So we have these, you know, there's these five opportunities that your AI could just like decide to give up on the on the attack.

[00:19:50]>> And so this would be really bad for the AI, right? Because uh if the AI decides that it's going if it just like gives up, you know, uh 95% of the time after starting the attack, it's placing itself at intense risk of getting caught in one of the first steps of the attacks. Like we're going to have way more opportunities to notice it attacking because of all these like half half attempted attacks, right? So the AI is strongly incentivized to continue attacks.

[00:20:17]>> So a strategy we can follow is we're going to repeatedly sample. We're going to do these resamples. We're going to repeatedly look at what the AI did. And then we're able to incriminate the AI better. We're able to do a better job of deciding whether it's suspicious enough that we should show it to a human via looking at the patterns in all these different samples we took from the AI.

[00:20:35]Because if the AI just did something which looked weird but was actually fine, then when we resample, it's pretty likely that it's not going to do something suspicious the next time we resample. Whereas, if it's attacking, it's kind of forced to have a high probability of continuing the attacks.

[00:20:49]In our most recent paper, uh, one of the things we found is that this resample incrimination, uh, allows you to substantially improve the safety usefulness trade-off.

[00:21:05]Like one form of adversarial training people sometimes do is if you have a game playing program, you make it play itself a lot of times because all the time they are trying to look for weaknesses in their opponent and exploit those weaknesses. And when they do that,