AI Agent Development: Debug, Evaluate, and Deploy with LangSmith

[00:00:02]All right. Um, so we'll go ahead and get started. Um, today we're going to be talking primarily about how to um, essentially debug, evaluate, and ship reliable AI agents. Um, and we're going to talk about it in the context of Langmith. Um, so a lot of what we see today um, is built into our Langmith product in terms of the way to ship reliable and production agents. So I'm excited to jump into all of those topics. I'm I'm Victor. I'm one of our deployed engineers here at Langchain. So really excited to get into all of this and if you have any questions afterwards um please feel free to reach out to me or reach out to our team and we can definitely um help you along your agent building journey. Cool. Um so go ahead and jump right into it. So um let me There we go.

[00:00:51]Cool. So um the AI imperative is here.

[00:00:55]Um this is something we talk about a lot is you know every company is trying to figure out how to use AI. Um we're seeing generally that the challenge of AI is less of like a if we're going to do AI and more of when and the speed and reliability at which you can do so is very very important. Um so that's kind of the high level of of what we're seeing with AI with different companies.

[00:01:17]um essentially wanting to generate more revenue per employee um improving their products and experiences and also um increasing the efficiency of your internal teams. Those are kind of like the three paradigms that we see people wanting to use AI for. Um a quick little poll here. I'm just curious where everyone might be in their AI journey, whether they're still figuring it out.

[00:01:36]Um actively piloting some AI initiatives currently building agents already have agents in production. Um would just be curious to hear where you're all at today. Cool.

[00:02:00]Um, so you should be able to respond to that panel. Um, let me check on my side really quickly.

[00:02:17]Let me pull this up.

[00:02:21]Awesome. Thank you. Cool. Um, yeah. So, that's exactly where um, you know, we we see most people kind of start coming to Langmith is is when Okay, cool. I just got the results. So, it looks like the majority are still looking to scale agents in production. Um, and we're seeing that there are a lot of different people in this kind of phase. um and still kind of also piloting different AI initiatives and trying to figure out where to use agents. Um this is exactly where LinkSmith can come into play and help you build reliable agents. Um so building with AI is a little bit different than our traditional software engineering workflow, right? Um traditionally if you're building in software, you would have very deterministic code that you could see it whether it compiles or not. Um and that's pretty pretty easy to determine um if your code is going to work or not.

[00:03:07]with LLMs, we started to see more constrained apps, right? So, we have like one LLM call, like a one-time call with with just that context of that particular LLM run. Um, and that has been probably the most common way that we see people get started with LMS is just like some some generation or some sort of like filling out of a of a document or something like that on behalf of the u user. So, those are kind of fuzzier. Um, they're still kind of easy to see if they're working or not.

[00:03:35]you know, you'll see like a a an error if it doesn't run properly, things like that. Um, as we get into agents, this is where the really interesting um portions of evaluating and building with AI and where things really start to differ from traditional software with agents. Um, the majority of our logic will now live in our agents reasoning. So, um, if an error happens, if things like that happen, we can't really go into a stack trace and see what happened. um or we can't look at our code and and try and figure out where in our logic there's something wrong. We need to kind of be able to dig into what's going on with our agents. So, it's really important to be able to have the observability to go in and start to evaluate our agents. And this is one of the core things of of Langmith and one of the core things that we've seen building a lot of agents with different enterprises is your agent uh your agents and app logic no longer lives fully in code. there's kind of this like uh different space within the agents own logic that these things happen.

[00:04:34]Cool. So on to um Langmith itself. So Langmith allows us to basically turn these traces that we get from our agents into um a systematic improvement or what we refer to as a flywheel. Um and this is done using our agent engineering platform. So the idea is is that we start to observe our agents in production using our tracing and monitoring features. we can then start to evaluate them and curate data sets based on um that production data. From there, we can then redeploy our agent or deploy it for the first time using deployments um and then kind of continue on in this flywheel. So, we're going to go through all of these in high levels.

[00:05:12]So, I'm excited to kind of jump into them. So, um let's go ahead and do that.

[00:05:19]Great. So, here we are in Langmith.

[00:05:23]We're in a little uh demo workspace that we have here. And LinkSmith is a really good way to be able to understand what your agents are doing. So on the far left, you'll see tracing, monitoring, data sets, etc. What we'll do first is um you know, see how we can set up tracing. So if you click here on this tracing quick start, you'll see that um with Langchain and Langraphph, our open source frameworks, it's really easy to get started with tracing. Um, all you need to do is set these environment variables here and everything's ready to go. Additionally, we have integrations with all of these different popular frameworks. So, if I click here in our docs, you'll see we have, you know, all of the most popular agent frameworks out there um for easy setup and easy tracing. Additionally, you can manually instrument um your code as well in Python and TypeScript. And we also if I go back to our integrations um we do have the ability to trace with open telemetry as well. So this is another popular option that we see.

[00:06:22]Cool. So that's how we can set up tracing on an existing agent. Um one of the agents that I really like to demo is called chat lang chain. So anyone can go to this today. Chat lang chain is fully open to the public. Um and I can select a different model to use. Let's just use uh a fast model here. And I can say how do I use lang to improve my agents.

[00:06:46]Cool. So um chat lang chain is built actually um with lang chain and lang graph. It is all plugged into lang. So all of the tracing and observability that we'll see today. And then lastly we have the ability to deploy on lang. So it is actually also deployed on linksmith. So we can see here that it's giving us a nice trace on the agent's performance and that looks like it did a great job. So this is just a demo of what this agent does. Now to kind of get into the logic of what just happened is I can jump into our tracing projects here. I can filter down on just chat lang chain as an application and we can see just the resources associated with chat lang chain and then from there I can open up our actual project.

[00:07:28]Great. So you'll see in the last day or so. So we're we're filtering by the last day. I can see the last 6,000 runs here.

[00:07:34]Um in the past day I can also like set a different time period. So I can go to last week and I can look at traces from last week if I wanted to and start to understand how our agent is performing then. Um and we can pop open any one of these traces. So I could pop this open and we can see exactly how this agent works. So on the far left side in the tracing view um we'll be able to see the different steps that our agent has taken and you'll see here that you know the first step is we jump into this guardrail here. It then goes to this guardrail LLM call and anytime we make an LM call we can see the system prompt.

[00:08:07]We can see the tools I had access to. So we can see this is a pretty basic guardrail. It just basic just just checks like is this question related to lane chain or langraph and this allows us to gate our actual agent from you know malicious queries or prompt injection things like that. So this is actually built into the architecture of our agent and then from there it responds with whether the decision is allowed or rejected. It then continues into this model tool calling loop. So this is a simple like reason and action style agent right? So it has access to a few different um has access to a few different tools. So if I jump into this LM call, we can see it can search our docs, it can search support articles, check links, etc. We have a nice system prompt here and our agent decided to call two tools and then it finally responded back to the user. So we can see that full context gets sent to our LLM here and it finally responds. If I click on output here, we can see the update and right here is the full response. So this is really nice to be able to debug our agents. We can also jump into a waterfall view and we can see um related to latency uh kind of how all of these steps took place and it's really nice to be able to see all of these happen. Um and we can see for example these tools were executed in parallel and we can see that that drastically reduced the latency of our agent. So that's really nice. Um one thing that we have inside of Langmith as well is Pauly. So on the far right side we'll see Paulie is here. I can ask Pauly any question. So Polly can help you filter things. It can answer questions you have related to Langmith.

[00:09:37]Um it can show you recent errors, things like that. But I can ask Pauly to filter down on runs from the past two days.

[00:09:51]Poly will always have the context of the pages that you're on. So it's able to see what page you're on and it's able to just show you. For example, I asked it to show me the runs and the runs would show us all the intermediate steps. I can switch it back to traces and we can see all the traces from the past two days. So this is a great way to interact with our agents.

[00:10:11]Cool. So one other thing that we can do is view our traces in a thread style view.

[00:10:17]So threads um you can think of as like full conversations. So if I continue this conversation here and say like hi, you can see that we're going to kind of continue on to a second turn of the conversation.

[00:10:28]It can be really useful to see how your agent performs over multiple turns of a conversation. So what we can do in this thread view is pop any one of these open and we can see sort of the back and forth between our user and our agent and we can see all of these different steps.

[00:10:42]I can switch to a trace view again and I can go to like step nine or sorry turn nine. So this is like the ninth back and forth that our user and our agent is at.

[00:10:51]And maybe our agent as that context window grows starts to struggle. So we can find traces and threads that are happening way down the conversation stream and see how it's performing as the context grows. So that can be very important as well.

[00:11:05]Awesome. So um we covered how we can get traces into Langmith and filter on them and and debug um exactly what's going on our agent using poly and being able to look at different things. We've seen the thread view. Um next, how can we like start to automate this and see how we can evalu?

[00:11:23]This is what we refer to as online evaluators. We also have offline evaluators which we'll get to when we get to data sets and experiments and how we can test and improve our agents. Um but essentially these online evaluators will allow us to observe our production agents and calculate how they're performing. So one of the things that we can do here is um so I see a question here about the option to eval sub agents which are using tools and skills on their own decisions and demand. So yeah, we can absolutely eval sub agents and full endto-end agents. There's there's a variety of different eval strategies that you can explore inside of Langmith.

[00:11:58]Um, one of the ways that we can set these up is on the top right. I can click new. Under new, you'll see this new evaluator button. And then you'll see we have a variety of different options for eval um for our tracing project. I can either do LLM as a judge, I can um create one from scratch, or I can create an LM as a judge from one of these pre-builts. I can also use a custom code evaluator um for more deterministic type uh queries. And then we also have the ability to evaluate a full thread. So that is really helpful for being able to determine how our agents performing over full conversation and is a pretty unique feature to Langmith. So to show you how to set one of these up, I'll show you one that's already created. So, for example, we have this kind of more simplistic helpfulness one that I think does a good job of demonstrating this. Um, when we set up our eval, we'll set up a name for it. We can set up a sampling rate, you know, from one to 100% of the traces.

[00:12:52]I'm going to leave it at 10%. Um, and this just helps control cost. So, what'll happen is as these traces flow into Linksmith, we can set a sampling rate for, you know, a random sampling rate to send these traces off to be evaluated.

[00:13:04]From there, we can filter down on different runs. So, the default for your evaluators is going to evaluate the entire run. Um, you asked about evaluating sub aents and things like that. I could remove this and I could go into a run name and I could say I just want to evaluate my guardrail, for example. We could do that. And now we're only going to be filtering down on the runs for our guardrail. You could do the exact same with a sub aent um and be able to evaluate just those sub aents as well. Um, but I'll leave this on is trace, which means is it an endto-end trace.

[00:13:36]Um, from there, what we can do is actually set up this eval. So, this is an LM as a judge. So, I do need to bring an LLM into the into the picture. Um, so in Langmith, we can set an API key um and use any one of our providers that we'd like to use. So, we have all the popular model providers here. We have cloud providers here as well, including Azure, Google, and Bedrock. And then we also can bring in custom models via OpenAI compliant endpoints. So we just need to make sure that they're OpenAI compatible and we're able to bring in any custom models as well. So um I'm going to leave this on GBT5 Nano for now.

[00:14:10]From there we can set up a system prompt. So we can say you're an expert um evaluator assessing how helpful a chat's response can be. So this is basically going to score on a 1 to 10 how helpful our agent was for our user.

[00:14:23]From there we will pass in the original user's message. So all we're doing here is injecting in a variable and and finding that original user's message which you can see highlighted here. And then we'll grab the last message from our AI or from our agent. So we'll send those off to be evaluated. We can then set a feedback kind of configuration. So we can say on a scale of 1 to 10. I could also say boolean or I could even make it categorical if I wanted it to dump it into different categories as well.

[00:14:51]Great. So once that is set up, all of these traces will start going through.

[00:14:56]And the other thing that we can do is we can also set up a multi-turn evaluator.

[00:15:02]So the way that this will work is kind of same thing. We'll hit that add new.

[00:15:06]And I'll show you one that's already set up. So this one we're going to be trying to see if our agent actually accomplished the task that our user initially set out to do. And the way that we'll do that is again we can set a sampling rate. But what's a little bit different about these thread evaluators is we can set a thread idle time here.

[00:15:22]So this will be how long we'll wait until the last user message to mark a thread as completed. So for chat lang chain we have it set to an hour. I can also set it to a day. I can set it to 7 days however you might have won.

[00:15:34]From there I can go ahead and you know set up the same exact way that we set up the other elements as a judge a system prompt a human message. But what's really nice and and kind of like a really nice DevX here is that we can basically select which messages we want to pass to this evaluator. So I can grab all the messages and that would include all of the intermediary tool messages, sub aent calls, etc. Um, but that might be a little bit noisy and a little bit too much context to send an LM as a judge. So I can actually just grab out the human and AI pairs, which will be the human messages, the final AI response, the human message, the final AI response. So, it's just going to be the back and forth between the user and the agent. And that'll kind of map to what you see here in the UI, right? Like what the user experiences here.

[00:16:16]So, we can go ahead and do that. And then we can set a boolean of whether this user's issue was resolved over the course of the conversation. So, this can be a great way to evaluate how your agent's performing. When you think about how agents work in the wild, a lot of times they'll they'll generate responses that look semantically correct, but the user might immediately after be like, "No, that's wrong." Or, "No, I tried that and that didn't actually work." Um, so it's good to see over the course of a conversation if the agent is actually able to accomplish the goal.

[00:16:43]Awesome. So that is online evals um in the context of LLM as a judge and thread level evals. You can also do custom code. Um I'll kind of skip over that today just to make sure we cover all of the different topics we're looking to cover. But essentially we could just write custom code that will run in a sandbox. Um it can't make external requests. So it's just more for like deterministic things that you're trying to evaluate. So we have one that checks for example if lang chain products are mentioned using code and it just looks for strings. Um cool. So when these evaluators run they're actually going to write feedback either on threads or on the runs. So if it's a normal element as judge it's going to write to our runs.

[00:17:17]We can actually filter down and find traces that this happened. So here we are. We can go into this LM as a judge or sorry we can go into this tracing project and you can see I have a pre-made view for helpful traces. And all this does here is it's going to filter down on feedback scores where the key for helpfulness is greater than or equal to eight. So now we have all of these traces that got high helpfulness scores. And I can pop any one of these open and we can see, you know, this one got a score of nine. I can open this up further and we can see the reasoning as to why it got a nine. So our our LM as a judge will also include reasoning as well. So that's really helpful. Um, so that's how we can kind of find all of these traces and start to see what's going on here.

[00:18:00]Great. Um, I'll talk in a little bit how we can use automations to actually automate this process of of grabbing these traces and sending them to more interesting places. But, uh, before I jump into that, um, I do want to show you insights really quickly. And insights are going to be how we can start to understand how our agents are performing. Once you start getting, you know, hundreds or or thousands of traces and you're trying to um essentially ensure that your agent is going to be performing as you want it to and you can find these edge cases. You can find different ways that users are using your agents. As anyone that has worked in software or product knows, people tend to use your your agents or your products in ways that you don't intend. So, it's really helpful to be able to find those um in real time. So, we have a few different examples of insights reports here that we've ran. To create a new one, you just go up here in the top right and we'll click new insights report. And you can do this manually where you'll kind of go through and set all these different things. Um, we also have an auto setup where you just start chatting with um, our insights agent and it will basically configure this insights job for you. But essentially what's going to happen is our agent is going to go through and cluster and categorize all of these traces depending on what you're trying to find out. So, if you're trying to find failure modes, it's going to find all of the traces that your agent made uh mistakes and start to cluster those together into different categories. So, you can see what types of questions are leading to the most errors. Um, if you're doing like a broad usage pattern one, it'll show you all the different ways that people are using your agent. Um, so that's kind of up to you how you want to categorize things or have your agent analyze them, but you can also include attributes alongside your insights that will help you um parse through all of this data. So, it'll basically just like an online evaluator, it'll show you these different metrics related to these clusters or these categories. And that might sound a little confusing. Um, but let me switch over into one of these that has already ran so we can kind of see what this looks like. So, you can see we have these kind of three that run. Um, you can set a schedule if you want. You'll be able to see the distribution over time. So, it's going to go ahead and start categorizing things for us. So we can see here that um we are basically trying to identify how people use chat lang chain and I wanted to put these top level categories for it to cluster them into. So um we asked it to categorize it into lang chain, langraph, lang and deep agents. I can also um not put any categories and it will automatically create those clusters. So if I go to like Q&A topics for example, it just came up with these categories on its own as it was summarizing and grouping together these these different um categories. Uh so that's that's essentially how the deep or sorry how the insights agent will work. So if I go back to this usage pattern one, you see that it starts to give us insights on all of this data that it's analyzed and summarized. So this is like over thousands of traces.

[00:20:41]It's gone through and seen how people are using um chat link chain and it's saying you know people that are chatting with chat link chain are building production agents not prototypes. Rag still dominates around 30% of the the usage in terms of how people are asking chat link chain questions. Um and people are having some issues with some growing pains with the ecosystem. So that's helpful to know. Um and we can see the distribution between two different runs.

[00:21:03]So this happened on March 1st. This happened on March 8th. We can see kind of how things are changing over time and how people are asking different questions. Um and if I jump into one of these clusters here, so I can go into here. We can see what questions people are asking about link chain versus link graph versus deep agents. So these are all our open source packages and then also in langmmith. So what are people asking about linksmith and I could dive into any one of these uh subcategories here. We can see summaries of these different traces. I can also look at these individual traces one by one as well. So very very helpful for understanding your agents performance.

[00:21:37]And one little anecdote here of how we've used this um actually at Langshade is for chat lang chain when we released our deep agent package which is our um open- source agent harness uh very similar to like claude code or agents like manis um we were really excited about it we released it we blasted on socials and then people went to chat lang chain to ask how to build with deep agents um but we had failed to update that guardrail that you saw earlier so everyone was asking about deep agents and they were getting blocked by our guardrail so we saw a big big um kind of issue or error analysis or failure mode under deep agents and we were able to identify that very quickly and find out that oh it was just our guardrail that was off. Everything else is fine. We really quickly update our guardrail prompt and everything kind of continued without a hitch. So really helpful to identify those edge cases that you may not consider or may forget to update when you're building your agents.

[00:22:28]Great. So that is our that is our um insights agent. So I talked earlier about how when we have feedback that's written to traces, how we can filter down on these here. The really nice thing about Langmith is, you know, when you have all of this data being collected, it's it's not really the best thing in the world to come in and filter on these and then grab them and then maybe add them to a data set or something like that. So what we can do in Langsmith is we can set up automations.

[00:22:56]So in this plus button on the top right, same place as the evals and the insights report, we can set up a new evaluator.

[00:23:02]So these new evaluators will allow you to automate that process for you. So for example, the helpful traces, if I open this up, I have some filters here where I'm filtering down where feedback for helpfulness is greater than or equal to 8. Um, so this is identifying, you know, traces that are getting either an 8 out of 10, 9 out of 10, or a 10 out of 10 on the helpfulness score evaluator. And I'm able to essentially filter down on those particular traces, set a sampling rate, and then I can do something with it. So I can automatically send that into an annotation queue which I'll show you in a moment. Um which is essentially a place for subject matter experts, product managers, etc. to to come in and review traces for you. We can also just add this directly to a data set if we wanted to. Um we could trigger web hooks to send it off to a different um different environment.

[00:23:48]So that's really helpful to be able to set up these. Um, the other thing that we can do is in chat lang chain here I do have the ability to collect feedback and this is a feature of lang deployments is with our built-in agent server you can get like thumbs up thumbs down um and collect feedback directly on the traces. So if I left a thumbs down um it might be a good indication that we should go and review that trace. So I can set up another automation that every time a user leaves a thumbs down, which would be a zero, I can actually send every one of those traces to a different annotation queue so that we can uncover why users are getting frustrated with our agent. So that's how we can set up automations. Um typically these will send to an annotation queue. So in these annotation cues, um this is an example one that we have here that is for that helpfulness score. So if it gets greater than or equal to 8 and we'll be able to see everything here on this um annotation queue for subject matter experts and PMs and people alike we can set up instructions for them. We can ask them to leave feedback on the traces. So whether it was correct or not. They can also leave notes to each other. So I left a note to myself in this case. Um but you can see those here.

[00:24:55]Additionally, what's really nice about this annotation cue is as a reviewer, I might have way more context of what this agent should say in in reality, like the golden ground truth of what it should say here. Um, and those can be very very valuable when you go to test your agent later on. Um, even if your agent made a mistake, it's helpful to correct that mistake and save that as part of your data set to give it a hill to climb as you start to try and improve your agent.

[00:25:18]Um, and we'll talk about that in a moment as well. But what I can do is I can start to remove these. So, let's just remove these intermediary messages because, you know, I don't want to cloud the final output um with all of these different tool calls and intermediary messages. So, I can remove some of those. Now, we arrive at the final message. So, we have the original human message from chat lang chain and the final AI message that our agent made. I can hit edit here and I can start to clean this up. So, maybe I don't want chat lang chain to actually ever provide code. I just want it to link to the appropriate docs when relevant. So, I can go ahead and grab all of this code.

[00:25:52]I can delete it and I can go ahead and hit save. And now we just have a nice response that is just going to link uh kind of broadly explain the topics and then link to the relevant docs. So once that's done, I can go ahead and hit add to data set and we can add that to a data set for further testing. So I have this chat lane chain golden data set and I can go ahead and add it to that. Once that gets added, we'll see it in this data sets and experiments. So right here I have this chat chain golden data set and you'll see here we have a bunch of examples that are stored. We have around 10 in this demo data set. I typically recommend anywhere from like 50 to 100 um once you're kind of actually trying to improve your agent and get some good data on it. And you'll want to have um you'll want to have data in your data set that's on both ends of the spectrum.

[00:26:36]So, kind of examples that are, you know, tried and true that your agent should always be able to answer and then questions that are kind of difficult, um, but you want your agent to be able to answer in the future. And you'll want to have those reference outputs mapped to exactly what you want your agent to say in the wild in production. So, we'll curate that data set of reference inputs and outputs and then we can start doing experiments. So, experiments are going to be offline eval.

[00:27:01]And there's a few different ways that we can do it. on the top right. We can either run these in playground and that's great for testing like a single step of your agent. We can also run them in lang studio which is you know available if you're using link chain and langraph. You can run a local server very very easily from our CLI and you can go ahead and test your agent in studio. The final is going to be running it via the SDK. So if I click run an SDK you'll see we have a small little guide here. Um basically you just need the langu client. You'll point to the data set by its name. Define whatever evaluators you want to run. So, similar to the ones that we set up in the UI, you can do these in code. Um, and you can actually set up very robust evaluators in code since you'll have access to, you know, any libraries you might want, things like that. And then from there, we'd use this evaluate function to evaluate and log all of these different um test cases against our new agent. So you'll use evaluate.

[00:27:55]You'll pass in your target function which would be your agent or your LLM application. Point to the data set by its name to find any of the evaluators and let that run.

[00:28:05]So the other thing that we can do um before I show you like an experiment is you know I showed you how you could set up an evaluator in code when you run that evaluate function. You can also set them up in your UI. So I have this code checker one that I created and it's essentially the same process that we saw with the evaluators on the tracing projects. um you just define it in the UI. Um and what'll happen is once I have this defined here with my you know input output and reference output mappings what'll happen now is when I run these experiments here. So we see we ran a few different experiments it'll actually run that code checker no matter what. So you can have these like standardized evaluators on a data set that will run no matter what environment you run them in whether it's studio or playground or the SDK. So it can be a really nice way to kind of ensure that these certain evaluators run no matter what environment your team's running them in.

[00:28:52]Great. Um, so we see here that we wanted to test out chat lang chain against this golden data set and we wanted to basically try out four different models.

[00:29:01]We tried out haiku 45, GPT5 mini, GBT 52 and sonnet 46 and I can kind of scroll over to the side here and we can see that we included metadata as to the provider we use, the model we use, etc. So it can be really easy to filter down on this later. So, it's always nice to include metadata so that you can kind of understand what different knobs you turned with your agents. And these experiments are going to be a great way to test out different um architectures, different prompts, different models, um anything that might be a change that you want to make in your agent and ensure that it actually improves your agent and doesn't cause any regressions. And one of the ways that we can do that is by setting a baseline, which is like a newer feature in Linksmith. So, we can set a baseline here. If I unset it, you'll see they all get kind of get mapped. But let's say I'm using um let's say I'm using Haiku 45 in production. I could go ahead and set that as the baseline. And then every time I run a new experiment, they'll all get compared to this baseline. So we can see, you know, every one of these more powerful models did a bit better than Haiku. Um but we can see like the latency went up um and the cost went up and things like that. So we can see these kind of side by side in this dashboard. But what I can also do is grab Haiku and I can grab let's say, you know, Sonnet looks the best to me. So I can go ahead and grab those two and I can go ahead and hit compare.

[00:30:19]So what's really great about this is, you know, I I'm looking at my traces.

[00:30:22]I'm trying to identify which one is best or sorry, looking at my experiments. I can actually go side by side and make a very nice comparison here. So we can see, you know, side by side which ones are doing better based on these different evaluators I set. So I set one related to accuracy, a code checker, um including docs if the if the docs that are included are relevant to the answer, faithfulness, which is essentially like hallucination. We can see latency metrics, we can see cost metrics, token count, etc. So we can see that um across the board it seems like Sonnet is a lot better um in terms of the way that it performed on those test cases. Um but the cost is nearly double. So this is one of those things that you might need to consider when you're building agents is like is it worth the improvement versus the cost and things of that nature. Um so that's essentially how we can use experiments. Um one thing I'll shout out here too is is Paulie is with you here as well. Um I can create a new thread here and I can ask it to ask uh what was the best experiment in your eyes.

[00:31:21]So, it's going to go ahead and look at our current data set and see if it's able to um basically evaluate that um and find the the best experiment. So, it says based on the experiments, Cloud 46 looks like the strongest performer overall. GBDT5 mini came close but lag behind. So, that's really helpful. Uh Pauly could also suggest prompt improvements and things like that. Um but to show you that prompt improvement, what we can do is go into our prompt playground. So, I like to talk about the prompt playground in the context of a trace. Um, so if I go back into our tracing project here for chat lang chain, we'll be able to see all of these traces flowing in. And again, I can filter down, you know, like last week or something like that. Um, and find these different relevant traces.

[00:32:09]And then in here, we'll be able to see any one of these LLM calls here. So we can see, you know, one, two, three different LM calls. Whenever I click on any run type that is the type of LLM, we can actually pull these directly into our prompt playground. And the reason why you might want to do this is you might notice a particular trace that your agent did poorly on. Maybe you filter down on one that it did poorly on. Um or maybe for this guardrail it, you know, it didn't allow something that it should have. We can come in here now and we can actually pull this prompt in with the context of the actual trace that we were looking at. So this pulled in the actual human message here where someone asked about custom middleware and it looks like it did well. It decided to allow I could also remove this and I could say what is the weather in SF.

[00:32:51]I could then you know just like those um online evals I can choose which model I want to use. So I could switch to entropic for example switch to a smaller model like haiku hit apply and I can give this a try again. So we see now because we have this output schema defined as decision um and it can either be approved or blocked. It's always going to either respond in approved or blocked. Um, and we see here that, you know, weather isn't being allowed, which is which is best. I I think that's a good idea that weather isn't allowed.

[00:33:20]One thing I can do is I can ask Poly to actually improve this prompt. So, I can go over here. I can click new thread and I can just say improve this prompt using best practices.

[00:33:30]What Poly will do here is it's going to use all of the best practices that we've seen from building a variety of different agents and working with different, you know, enterprises, startups, etc., and it's going to apply some of those best practices. So, um, one of those is being is most likely going to be using XML tags. So, typically, uh, XML tags are going to help your performance a lot. Um, the other is maybe some few shot examples, some positive examples of what it should or shouldn't. Um, and that'll really help improve the prompt over time. So, it might take Poly a bit to do this, so we'll see if it goes right away. Um, usually it takes like around a minute or so to go through the whole prompt and make all the updates. But, there we go.

[00:34:08]So it added some XML tags here, added some explicit context, some enhanced clarity, some concrete examples. Um, and this can really help us when we build our agents.

[00:34:19]Cool. Um, the next thing that we can do is, you know, for whether it got blocked, um, I can also like add in a dynamic variable. So I could say, uh, runtime allowed topics and we could say, um, let's say weather. And now it should go ahead and allow it. So that allows us to like dynamically inject in things that we want. So typically you could inject in things like memory or things like that in your actual agent. So it can be very helpful to to do that. Um from there you can go ahead and hit save and that'll save the prompt into the prompt hub.

[00:34:51]Prompt hub is where you get everything saved and versioned. So this works as like a repo under the hood. Um we'll see full commit histories. We can then pull in our prompts into our applications using our Langmith client um in both Python and TypeScript. And we would just use this pull prompt method and define it by its name and its commit ID here.

[00:35:09]So very very helpful to be able to pull these into our application.

[00:35:14]All right. Um so that is a high level of Langmith um observability eval um that sort of thing. There's so much more to Langmith as well including our deployments product which makes it really easy to deploy an agent in just a few clicks here. You just point to a repo, you set some environment variables and that is it. you're ready to go and deployed your your lang chain or lang graph agents. We also have agent builder here which is a no code way to build your agents. Um and it is a very very powerful tool. So that isn't in the scope today but I like to mention them since I think they're both fantastic products. Um so that is pretty much it for my presentation. Um I'll go ahead and shift over to a Q&A now if there are any questions. Um so I'll go through the chat now. Let me get this pulled up.

[00:36:02]and we can start to go. All right. So, I see some questions. Um, how can we use uh can Paul use whatever LM we want? Can I have an a bedrock configured to it? Um, that's a really good question. So, in here, if I go into our settings and I scroll down to integrations, you'll see model configurations. Um, and this is a new feature that shipped recently. Um, as admins, we can configure which models are available to our users and especially for things like poly insights, etc. Um, and yes, we're able to use, you know, any one of these cloud providers. You can set those cloud provider models here and then set that as the default model for Poly, etc. So, you have full ability to, um, do all this configuration here and set it for everyone in your workspace as well.

[00:36:46]All right. Um, all right. So, let's see.

[00:36:51]Can we use poly uh model thread? Yep. Uh use some enterprise SAS application.

[00:37:00]Sorry, let me make this a little bit bigger. It's kind of small on my screen.

[00:37:04]Okay, we use some enterprise SAS applications. They have started offering agentic capabilities within their platform. Will Langmith be able to integrate with and monitor such agents?

[00:37:13]Um yes. I I think uh majority of of these platforms that are offering agents are exposing them in some way for us to trace and monitor them. Um I would love to hear which particular agents you're you're talking about, but I know we have integrations for example with cloud code, cloud agents SDK, um you know, any of those those popular ones. Uh Bedrock agents, things like that um can trace directly. Uh if there's any agents that that you're using that you you don't see an integration for in our docs page or anything like that, please reach out. Um I'm happy to help and see if there's any way that we could get you um set up with tracing there. Um let's see. Langmith be able to integrate with and monitor such agent. Oh, sorry that was a cut off. Uh my problem is there's no way to know ahead of time which sub agents my agents will choose. Is there a way to give a dynamic eval? Um yeah, so this is something that I probably do off like on my data sets and experiments and my offline emails. So um we have a nice guide on this. Um, so I can find it and send it through.

[00:38:14]Um, one moment. Um, but essentially you can do what's called a trajectory evaluation. And these trajectory evaluations are going to be how we can decide uh for different inputs if our agent's going to choose the right sub agent, if that sub aent is going to use the right tools, etc. So like those those can be very very important when you're building out your agents is is not only like looking at end toend tests or you know what they're producing but actually just monitoring the trajectory.

[00:38:41]So you can actually just run evals on is it calling the right agents in the correct order things of that nature. So there are the docs here on trajectory evals.

[00:38:52]Cool.

[00:38:54]All right. Um, how do you evaluate your LLM as a judge?

[00:38:59]Evaluators are performing as as expected. Uh, this is a great one actually. We have a feature for this.

[00:39:05]So, um, this is common issue that we see people have. You can go into your evaluators here. I can click new evaluator either on a tracing project or in an offline uh, you know, data set and experiment. But if I wanted to create a new evaluator and kind of make sure that it's aligned to how I want it, we can actually create it from labeled data.

[00:39:23]And this will send us down what we refer to as the align eval path. And basically what'll happen is you'll select some runs to annotate yourself. You will then get sent into like a unique annotation queue to annotate that data. So you know you select like five to 10 runs or or 50 runs whatever you want. You then as a user will you know for let's say I'm doing uh helpfulness right or I'll just say helpful.

[00:39:48]And what I'll do is as I'm going through I can say like this is helpful yes or no for these like 10 traces. And then I can start crafting a prompt for an LMS as a judge. And we'll basically get to see if our LM as a judge is aligned with our preferences as a human or what we would say as a human. So it's pretty pretty important to um do that as well.

[00:40:07]Great. Um how do I detect and measure hallucinations? Um typically you would want to in your ground truth have like the documents or data set or the documents that your agent is retrieving or should reference. Um, and you can check with your LMS judge if the final response is grounded in the information from those documents.

[00:40:24]Um, cool. How do we budget costs overall? Can we set thresholds? Um, yes.

[00:40:28]So, you can set thresholds in the settings. Um, it's pretty pretty straightforward there. Is there a simple way to evaluate tool call success whether it was the right tool or in the right sequence? Um, I think that would go back to trajectory eval. Um, I would probably do that. Uh, we have some guides on that in the docs. Um, and some notebooks for that as well. Um, from Chase, where do we set the API key for Bedrock? I tried in the provider secrets and could not get it working. Um, you might have some more luck if you go directly into the playground and try and hit um, Bedrock here. It should ask you to do these different things and then on the top right there's like the secrets and you can add all these different secrets.

[00:41:05]Um, there's also like an IM ro so you can set an IM ro um, in your AWS account and that will basically get the permissions to like refresh the tokens.

[00:41:13]Um, so that should should work.

[00:41:16]Um okay I don't understand how data set is used in the links the ecosystem does the idea is to use it as a retrieval database to insert gold examples for a few examples. Um no so I think uh I think a good way of thinking about this is like part of our agent engineering flywheel. So the idea is you know we have these traces flowing in right so these are like production traces um and we don't really have a way to compare how our agent should perform versus how it actually did perform. So the way that we can do that is we can curate these data sets of golden examples. So we have you know all of these you know we have these 10 examples. So for this given input this is exactly what I want my agent to say in the wild likely for this one as well. So for this given input of how to you know what is the linksmith Python SDK equivalent to the runs endpoint. So this is kind of a complicated question. There's a lot of different ways that it could go. So this is the reference output of like once it does all of its research and responds back to the user. This is what I want it to say and how I want it to say it. So what we can do now is I can go and run my production agent against all of these examples and see if it's going to perform well. Um and we can get a baseline. So that's what we did here for this haiku 45 is we got a baseline that for a given input.

[00:42:31]So this is the golden input. We basically passed that into our production agent using haiku45 and we got this output here. And this output um it looks you know decent to me and it might look decent to an LLM as a judge but the benefit of having this reference output is now it knows exactly how it should have responded. So we can have different things like this code checker, this accuracy checker on a scale of 1 to five, uh answer relevance, things like that. And we can see okay compared to our reference output, how does this output compare? Um and this is what really helps us improve our agents and start iterating. So we can see okay um for example for accuracy it got a one um because it made some factual errors.

[00:43:09]It's it said that uh to use the langraph SDK instead of the lang SDK um and that's a pretty big like error that it made here. So it should have defaulted to using the linksmith SDK for querying runs versus creating runs. Um and that was kind of the issue there. So what the benefit is is how we can start to improve and start to create this this whole holistic strategy of going through and building out these agents um and improving them over time. So now I can do all of this offline testing outside of my production environment. I could figure out how I could improve my agent to do better on these test cases and then I could deploy a new version of this agent into production that will work a lot better. Um, I hope that was helpful, Rodrigo.

[00:43:49]Cool. Um, is there a simple way to evaluate tool call success? So, I think we just answered that a moment ago. Um, blacklist, can we spec out what our agents should not do? Um, good question.

[00:43:59]I I I would probably put that in prompting. Um, or you could also have hooks. So, in your actual agent architecture, and this is more on like the agent building side rather than the linksmith side. Um you could have hooks in that process in the agent tool calling loop that if it does a certain thing you could kind of kick it out or cancel the run or restart it or something like that. Um so a lot of people are doing that now with like cloud code to ensure that it doesn't uh do like destructive things. You can they just recently added hooks which we have in our deep agents package as well um where you can just kind of like hook in and and prevent any dangerous actions from happening. Um cool. Can you comment on a typical life cycle strategies for developing high quality evals plus graders prior to prod approaches to online evals and how to um how online evals inform refinement? Um yeah, so this is a a great question in terms of like the life cycle. Um I think quality evals and quality experiments like start with great traces. So even if you're not in prod um passing in a bunch of different inputs into your agent and seeing how it'll respond um is really helpful to then have your graders go through change that data annotate it in a way that it should be and then adding that to the data set for more for better experimentation. I think like starting with a example of what your agent has already produced and then editing that to where you want it to be is a bit easier than trying to generate like synthetic data which is what a lot of people start with. Um, so I typically do that even like for my own use cases when I'm building agents. I'll just like come up with, you know, 10 to 20 to 50 inputs and I'll just pass them into my graph. Um, and then or into my agent and then from there I will, you know, use the annotation Q, clean them up, put them at exactly how I want them and then add them to the data set cuz it's a little bit easier than trying to like go through and synthetically generate data or have your your subject matter experts generate them, things like that. Um, and then on online eval side, I think these are more like benchmarks rather than ground truths. So I think your your data sets and experiments are really where you do all of your like ground truth testing. Make sure your agent is at like a point that you think it's ready for production. And I view my online eval as more of a way to monitor our agents as they're in production. Um and one of the ways that we can do that is if I go over to this monitoring tab, we do get a pre-built dashboard with every deployment. So you'll see things related to like your trace metrics, um your LLM calls, like your latency, your cost, etc. Um different tool calls your agents are making. So you can see you know by tool count which which tools are getting called the most. You can see the latency by tool as well which can be helpful. Um and then the error rate as well. So we can see like for example today um we usually don't have any errors but we can see today we're having like a 03% chance or 03% of the the time um our search docs is hitting an error and that's an issue with uh you know we use Mintifi's doc server. So uh there might just be an issue on the Mintify side that's causing that. Um, we can see run types and then we can also see these feedback scores.

[00:46:51]So, you can see here like our helpfulness score. We can see it over this past seven days it's been, you know, hovering around six or 7%. I know if this like drops or spikes down, there's something there's some kind of issue with our agent. Um, same thing with like task completion. We can see that it's doing a good job of accomplishing user tasks. If there's like a big dip or a big drop, there might be something wrong. So, yeah, it can be really helpful to just like monitor over time.

[00:47:15]Um, cool. Um, how are annotation cues surfaced to SMMES inside of LinkSmith or can we surface them in another UI and integrate? Um, yeah. So, uh, you can, um, but typically you can go ahead and start create a rule-based access um through or rather through our rule-based access control, you can create a role that is specifically for the annotation Q. So, you can have data reviewers come in. Um, there's no like seat charges in Langmith. It's all usage based ch like billing. So, you don't have to worry about like having too many seats allocated or something. Um, but you can have users that are just specifically um, able to access like this annotation queue in this particular workspace and they're able to come in and access it.

[00:47:54]Um, the other thing that you can do which, um, I wish I actually had set up um, that's actually a good idea to get this set up on this demo workspace is you can enable a custom output rendering and that'll allow you to basically iframe in your own front end. So if you have like very unique traces where you want to allow uh reviewers to like see them in a particular format, you can enable this like custom output rendering which will allow you to see that in a very unique way. So um I unfortunately don't have a demo of that right now. But yeah, cool. Um so it looks like I'm being told that is it. Um thank you for joining. Uh I really hope this was helpful. Feel free to reach out to me um on LinkedIn or or you know contact afterwards. Um, I think our team is going to kind of send you to a different screen to to do those things. But, um, I enjoyed this time today. Thank you for all the great questions and, uh, yeah, feel free to reach out anytime. Thanks.