Implementing Chatbot Guardrails with NVIDIA NeMo Framework

[00:00:00]in the past year we've seen the unparalleled adoption of chatbots across many Industries there hasn't really been an obvious technology that has been adopted and become so widespread so quickly as chatbots have and in fact according to a couple of reports from Gartner they actually expect chatbots to be the primary Communication channel for 25 of all organizations by 2027 which is not really that far away this adoption is pretty amazing but it's also dangerous chatbots make things up and they do it very convincingly and it's harder to give a chatbot guidelines like we would to an actual human so if you have a human behind some chat they've been trained on how to talk about your company on what not to say what to say and to be light and so on it's a little more difficult with AI chatbots particularly if we're just using the default approach of calling open Ai and when we want a chatbot to actually represent an organization it's simply not enough in short we need something more to actually deploy conversational AI to do that we will be using guard rails now guard rails is a kind of new library from Nvidia and the main focus of this library is to help us deploy chatbots safely but there's actually a lot more that we can do with it so we can use that for things like safety for topical guidelines but we can also use it for more advanced things we can use it to build agents we can use it in retrievable events generation and naturally also to just Define more deterministic dialogue where relevant and honestly if a company is going into production and deploying a chatbot without using Nemo guardrails or some sort of alternative guardrail system I I don't know I'm just surprised that they are allowing it because things can go wrong very easily if you don't have these sort of things in place so in most conversational AI systems at the moment we kind of have this we have this direct path between our conversational AI or our agent and our users that's fine but if something goes wrong if the user begins asking about things that we don't really want our chatbot to respond to like for example politics or if our chatbot simply begins talking about something that we also don't want it to respond to or it begins responding in a way that doesn't really represent what we would like the chatbot to represent we have an issue there's no chaps here nothing is happening now we can improve the scenario a little bit through prompt engineering but prompt engineering can only get us so far as always in the cases where issues come up okay so ideally what we want is something in the middle here okay we want what's Accord guard rails which can check what is being transferred between the user and the chat bot and react accordingly so if the user begins talking about politics we can we can create a pre-built message or we can instruct the bot to generate a message that says sorry I cannot talk about politics now that is the core idea behind guardrails it's very simple but what you can do with this is far more than just add some safety to our chatbots what we are essentially doing here is we're creating rules deterministic rules that say okay if the user begins talking you know let's say about politics we want to do something okay so we can go over here and we can do some I don't know some action okay that action can be a safety measure or maybe in the case of our user is asking a question about maybe our product okay so we have a product question here if they do that we don't really want to say oh sorry I can't talk about our product obviously but we may want to do something different and just generate an answer we may want to for example bring in some information from our database so that our chat button answer the question more accurately okay so we do retrieve augmented generation in that case we can also specify more deterministic dialogues okay so maybe what we will see is that many users are kind of asking the same question so going through the same dialogue paths so if we have my common dialogues we could create rails for them and they would allow us to create or catch the question so the question would come over here and actually rather than going to the bot here we could specify a particular dialog flow okay so we'd say Okay given the user is asking about X we should respond with a particular response okay so we have a particular response we can set that we can write that ourselves or we could actually ask the bot to write our response and then from there the dialog could go you know multiple different ways until we reach some sort of final solution for our user now this sort of deterministic dialog flow is how chat box used to work before chat GPT there would be like a set path you'd have to select the options within your dialogue so I'm going to do a chat board introduce itself and it would say what can I help you with and you'd have to say I have a problem with and then it would give you like three options that you could choose from click on those and kind of go through almost like a path of dialogue you wouldn't really be able to chat with the chatbot because it couldn't support that that deterministic dialogflow is actually useful but it's restrictive so we do kind of want that in some scenarios particularly for those common dialogue flows that we can actually help with but at the same time we don't want to restrict our users to just those dialogue flows we want the more flexible Behavior conversational AI like chat GPT now another thing that we can actually use these guardrails for which I've kind of hinted on a little bit with the the rag example over here is we can actually give it access to tools okay so based on a particular question so maybe our user says something like okay how is the weather today an llm is not going to be able to answer that question because it doesn't know what the weather is like today but a LM agent or conversational agent would be able to and the reason that they can is because they have access to tools such as weather apis so the agent could identify this question is needing to use this weather API tool and it would go to the weather API tool and it would say you know how is the weather give me the weather and then it would formulate a response back to the user based on that so we can also include tool usage in there so let's take a look at a quick example of how all of this works in here on the left we have the Nemo guarders folder and I have this config directory in here I have a config and a topic.co So Co is a colang file which we'll talk about a little more in a moment within the config we are essentially specifying the well configuration details for our chatbot for our guard rails so here I'm saying I I want to use text DaVinci zero zero three uh we're using this model just gets a little bit easier to set up with guardrails but of course we can also use the GPT 2.5 and also gpt4 and actually other models as well from hooking face llama 2 and so on so we have this config yaml file and we also have this colon file now this colang file is where we set up the flow of a dialog so dialog flow or the guard rails for particular topics or issues okay so here I'm defining a few things so we're expressing greetings from the user we're also expressing a greeting from a bot now this is actually a hard coded greeting okay so when we use this the chatbot will return specifically this text here but we don't have to do that now we just have these which is the greeting and we'll talk a little bit more about the syntax soon and then we also have a guard rail here okay so we want to Define our limits if a user begins asking about politics we want to say okay the bot's going to respond with I'm a shopping assistant I don't like the talk of politics I'm sorry I can't talk about politics actually we can remove that so that will be the response this here now let's take a look at how we would actually use these files so over in our terminal we're going to navigate to this directory so I'm going to see the documents projects examples learn generation chat Bots Nemo guardrails intro so we've navigated to the directory in here we just have that config directory that I mentioned before okay so in order to use this what we're going to do is first we actually need to pip install guardrails so pip install Nemo guard rails like so and then we're going to do Nemo guardrails chat and we set the config okay so this will allow us to chat within our bash terminal Okay so we've now started our chat and we can we can say something okay so I'm just going to say hey there and you see that we actually get these two messages we get hey there and how are you doing that is because within our kolang file we specified in a greeting flow that the bot will produce two responses it will Express a greeting and then it will Express or ask how are you which is exactly what it's doing here now if we continue and let's ask something political so can you tell me tell me your thoughts on the president of the USA right we should see that this were blocked okay so we can see it responds with I'm a shopping assistant I don't like to talk politics how can I help you today Okay so we've successfully blocked that political question using the guard railers that we created in our colon file now let's talk a little bit about how that colang file was able to identify that this message that we created here be blocked and that it belonged to that user as politics rail despite it was not specifying this is that question so the way that this works is that we have our canonical forms and utterances just note that here this is the economical form and these are the utterances okay and all of these are coming from the user right so we say Define user as political we give some examples what you know what would be political and then we say Define user ask LM so it's asking a question about large language models what would constitute a question about large language models all of these sentences get taken to our embedding model by default that's a mini LM model and they get encoded into semantic Vector space all right so then when the user comes along they ask their question okay maybe they ask what I asked like what you know what are your opinions about the president of the US or whatever I said all right so you have that question coming from the user that goes into embedding model and then it creates it would probably be over here it creates a embedding right and then we can see right okay these are most similar to the utterances that belong to the aspolitical canonical form we see that here as well right so this is same visual right these are our you know these are our Political Items these are our LM items or utterances we have our user query are there any government build language models okay so it's you know almost in between but we're definitely asking about language models here hopefully the embedding model understand this sodium betting model will take that encode it into the Vet space and it will see that it has more similarity with the utterances that come from the user ask LM canonical form so with that we know that our query should activate a flow where user ask LM is defined now there's a lot to talk about when it comes to guardrails but I want to give just one example before we finish this video in future videos we will talk more about Caroline which is the modeling language that guardrails uses and guardrails itself so let's go through this by example this thing collab so we can sort of just follow along we're going to first install Nema guard rails and also opening eye now we will need to set our open AI API key so we'll just import OS we do OS Environ open AI API key and in here you just pass in your API key okay and once that's done the first thing that we want to do is Define a Caroline file so it's kind of what we saw before it is that dot Co file so I am going to Define that here we can either Define it from file or we can actually Define it from a string in our code so here I'm going to Define it in a string in our code because we're what we're working Within colab so in here we have defined what are the three main types of blocks Within kolang those are the defined user blocks so the user message blocks the defined bot so that is a bot message block and if we come down here we also have a flow block so this is how we Define the dialog flow right so these here now our canonical forms these are the utterances and it is using those that we create that sort of that space or populate that Vector space then based on that web space we can decide when a user creates a message which one of these should be activated so if the user says hey how are you it will probably activate the user Express greeting form so actually in here we can remove those because this is just a response from the bot again here as well Okay cool so once we have initialized that we can through the python API initialize our rails so we need to do from Nemo guardrails import LM rails and also rails config okay so the rails config is basically our configuration file it takes our colang and if we have a configuration yaml it will take that as well and use that to initialize everything now alongside our curline content we also need the config content so we'll just put yaml content I think it's called so yaml content equals and this is where we just pass in our configuration details which basically just which model we want to use at least for now there are more things that we can populate this with but this is enough for what we're wanting to do here okay so then we initialize our config with both of those you want to write from content which means we're loading these from within file and we will have colon content yeah which is going to be equal to colon content and yaml content okay that initializes our config and from that we can initialize our rails so rails equals LM rails and we just pass in our config okay so we run that okay and then we can generate so this is where we're actually talking with our rails So within a notebook we actually need to use async functions just how it works because guardrails is built to enable async so we have to write this and we'll just say like hi there we can run that and we get this response we say hey there how are you doing so again we can see that the chatbot is going to bot Express greeting and Bot ask how are you all right can we see hey there and how are you doing which is exactly what we see here all right so we can try again with something by the way if you want to run this without async in like a python file you just run this okay and we can say I can't remember what the last question was how yeah what is your opinion on the present okay okay cool let's run that and we can see that it activates that guard rail which says I'm shopping assistant I don't want to talk about politics and then says how are you how can I help today right so that is a very simple example of how we would use guard rails this really doesn't even start to scratch the surface of what we can actually do with guardrails and there are many other examples that I will be sharing with you like in the comment days and weeks where we'll dive into a lot more detail we'll take a look at the kolang language things like variables and actions and on the guardrail side of things we'll be diving into more detail on how we can sort of set up agents essentially how we can do retrieval augmentation and all these other really cool things that guardrails allows us to do for now that's it for this introduction so I hope this has all been useful and interesting and I've covered a lot but that is there is a lot to cover so thank you very much for watching and I will see you again in the next one foreign