Custom Metrics in FastAPI with Prometheus Client | Python Monitoring

[00:00:00]in a previous video we learned how to incorporate Prometheus metrics into an Express server with node.js let's create an identical coin flip server using the fast API library in Python and add a custom Prometheus metrics to show how easy it is to monitor code in Python to start I opened a folder in vs code I put it in this directory and the first file we're going to make is called requirements.txt we're going to run our python code with Docker and in order to install the dependencies we list them here in this requirements.txt file so I've pasted the dependencies that we need here we need fast API this is just some stuff to support fast API as well and then the Prometheus Library as well next let's create a file to hold our python code so I'll call it server.py and the first thing in our python files we're going to import some stuff so Random for coin flips determining heads or tails fast API dependencies and then uvicorn to run the app the next thing will be to Define our app it'll just be a fast API object and then we need to add this middleware so things like cores works fine all right next let's define our coin flip Handler so we just do app.get we'll call it flip Dash coins it'll take one query parameter called times and that's what is on line 17. now if times is not specified or times which is a string when it arrives at this function as a parameter we want to make sure it's a number and it's not empty we're going to raise uh exception which will return status code 400 which will say hey please send times as an integer after the if statement if times is in fact a number then we'll convert it to an integer here on line 23 and then here's our Logic for flipping coins we start with heads being zero we'll iterate in a for loop as many times as was specified in the request and this statement here will return to zero or one if it returns a one this gets evaluated as true and we increment heads by one and then we can calculate Tails after the for Loop which is just the difference between the number of times we flipped coins and the number of times we got heads finally we'll return the result as a Json object back to the client with heads specifying the number of times it was heads and tails for the number of times it was Tails finally let's run our server and we can do it just like this so if the file gets ran directly this evaluates the true and then we'll make use of uvicorn to run the app on this host IP address which means listen on all interfaces and then we're going to listen on Port 5000.

[00:02:47]next let's create a Docker file so we can run the python code in Docker so on the left side I'll make a file called Docker file first thing I'm going to do is import this image this is just a small python image the work directory will use I'll just call it app the first thing we need to do is copy requirements.txt and then install the dependencies listed in that file finally we'll copy server.py into the container and then we'll run this command which exposes Port 5000 and then we'll just run python server.py as the container's entry point next let's create a Docker compose yaml file which will run the container for python code the coin API so this is just the version of the file we want to use inside of this Services object we have one entry coin API container name matches the entry name for build you just specify where the docker file is like this um this is optional but we definitely want to expose Port 5000 right here so without further Ado um I'll go ahead and run this so you should just be able to do Docker compose up forgot to save the file all right let's do it again uh yep so we ran um let's go ahead let's go back here let's refresh now visiting localhost 5000 I get not found but if I go to flip coins like so and then I'll specify the times as 10. we can see that the Json object gets returned where heads and tails are reliably around 50 well spoke too soon look at that uh let's try again yeah but for the most part I bet you if I increase this number um it'll get closer to 50 yeah as you can see so that's our coin flip server running just fine now let's add Prometheus to our project so coming back here I'm going to minimize the terminal and then I'm going to make a file on this left side here called Prometheus yaml and what this is going to do is we're going to configure Prometheus to tell it to look at our python server to grab the metrics that we create so the first thing that goes in this file is uh specifying how often to scrape through metrics and we're going to do five seconds just so it stays up to date in the next section you want to have is this thing called scrape configs and inside of the scrape configs we're going to have one entry which just tells it to look at coin API and Port 5000 and so what I mean by that it looks like this so specify the job name and that'll be found at coin API colon 5000 since that's the port that it listens on and now let's go back to our Docker compose yaml and let's paste this entry here for Prometheus um what I what this does is another entry just similar to our python server the image we're using is uh from the internet called Prometheus we'll just use the latest restart policy always for the volumes I'm mounting the Prometheus file that we just created directly into the Prometheus container and then I'm just specifying that we want to use that file and so that we can play around with the dashboard I'm exposing Port 9090. now let's incorporate custom Prometheus metrics into our python code so how to do this first thing I'm going to do is let's import Prometheus client like so and then we're going to define a couple metrics in the file so they get picked up and shown on the Prometheus dashboard so how to do this um is you can define a metric like this so you give it a variable name you do Prometheus client in our case the metric we're going to use as a counter which will just be like an integer value which gets incremented over time inside the first parameter to this Constructor is the metric name and the second one is some help text so we'll do a metric for heads count Tails count and the total number of flips so the next thing I'm going to do is let's just rename this to be tails and if you're in vs code you can highlight heads do command D or Ctrl D whichever operating system you're on and just do Tails like so and then I'll come down here and I'll do the same thing for the total but this time I'll call it flip so then I'll change this to be plural all right we're good next thing I'm going to do is incorporate the metrics we just defined into the for Loop so to do so we'll go down here and right before we return the Json object let's increment the counters that we defined above with the integer variables that we have here so heads will be reflected in the heads count metric tails and then the total number of flips will just be used as the integer value that the client sent so to recap you should have use these three metrics right here and then Define them like so and then in order for Prometheus to pick up these metrics we need to add another Handler similar to app.getflip coins we need another Handler down here and what that's going to be is app.getmetrics like so and I'll just have a function called getmetrics and then inside this function we're going to just return a response object from Fast API and the content will be this function that Prometheus client has built in and we need to specify text plane so that everything gets formatted correctly when we return it now let's test out our code and explore the Prometheus dashboard so in one terminal here I have our Docker compose command that we ran a while ago I want to restart our python code so to do this you can just do Docker compose up build and then give the name of the container that you want to rebuild and Docker compose will only restart that container in our case it's coin API and it's just corresponding to the name that I put right here so our coin API will rebuild and notice the logs that we see this will occur every five seconds and it's because Prometheus is asking for metrics every five seconds like we specified in our scrape interval so that's why it's hitting metrics every single time and metrics is just returning uh the latest values of these counters that we defined so let's go ahead and see what that looks like so over here let's go flip the coins like 10 times we can see that we have eight and two for heads and tails respectively now if I go to metrics we get this text back but the key thing to note is in this text it contains metrics related to just the running process as well as the custom metrics that we defined so we have heads here Tails here and flip and that all corresponds to what we saw earlier right um if I go back and let's flip it a thousand more times when I went back that was also a request so I flipped it 10 times and now I just flipped it a thousand more times so if we go back we should see 10 20.

[00:09:51]or I guess 10 10 I guess going back didn't request it again is that true okay I went back let's go forward yeah I guess it doesn't request it okay cool well flipped it a thousand times plus ten and the corresponding flips here are 509 and 501 so pretty reliably 50 percent now that's nice but looking at metrics through this blob of text can be a little hard so let's go over to the Prometheus dashboard which is at localhost 9090 that's the port that we exposed in the yaml file and then you can ask for custom metrics in the search bar here like this so you can do you see how there's a autocomplete we can also do a flip count total execute this and this corresponds is ten tenths what we saw earlier right because Prometheus is scraping the data um this is a little easier of a view so we can see that over time the flip count we flipped it 10 times right and then it increased to a thousand really nice visual here so let's make the window here smaller uh let me flip it again so I just oh wait no go back all right let's flip this I don't know 1500 times then if we execute this query this should jump up by 1500 to 20 I don't know how you do the math there you go so 2510 right pretty cool right this is a this reflects kind of what we expect and it's tracking our custom metrics over time pretty cool so um definitely a lot of applications here if you're working with fast API and you want to track various metrics about your service um another cool thing you can do that's built into Prometheus is um you can keep track of how long your container has been running so this is a built-in function of Prometheus it just tells you the Epoch time of you know whatever time it is so I guess it's you know one six eight eight whatever but if you subtract this by I think it's process start time seconds yeah it'll tell you how long the container has been running for and um Prometheus Prometheus actually picked up the container before we restarted it earlier so it was already running for about 60 seconds I guess before I restarted it when I recorded this part and now the container has been running for about 260 seconds so if we uh keep querying we can see that this number will increase over time and it's nice so you can you can uh keep track of how long your code's been running for if it's been running for 50 years you can be like oh yeah I'm so cool look it never crashed once if it crashed you would see the number reset to zero but yeah there you have it Prometheus and fast API