A Practical Guide to Deploying Machine Learning Models with TensorFlow Serving

[00:00:00][Music] hello everyone and welcome to this new project in this project we are going to learn how to deploy a machine learning model or an AI model using tensorflow serving so this project we're going to be very very interesting that would be the first step into how to take our models that we train beforehand you know throughout the entire course it's a little bit step further and actually start to deploy it in practice so we're gonna take that model that train model we're gonna run it on a server and then we'll start to make requests to that server and try to get feedback so we're gonna send an image to a server and then we're getting the server we're gonna get that apply back and tell me what class not that image belong to which is again that's pretty incredible because that would be kind of the foundation if you want to build an app if you want to make a startup if you want to do whatever this is kind of the basics or the foundation of how to take AI or trained AI models a step further alright so let's get started first we are going to use the fashion data sets that we have done beforehand so we're going to first in the first couple of steps we're going to learn how to train our model and build our model and that would be pretty much as we have done in the previous lectures that's why I will go through them a little bit faster and then afterwards in the next section I'm gonna show you how to actually save our model and how to save different versions of the model and then how to deploy that model using tensorflow to pronounce serving so again in this project we're gonna build train and test a model to classify fashion images using tensorflow 2.0 which is similar to the steps that we have done in the past and what's new that we're going to deploy that model in practice using tensorflow 2.0 serving so again if you guys recall our fashion dataset consists of 70 70 thousand images and they are divided into 60,000 for training and 10,000 for testing and the data set consists of images that are 28 pixels by 28 pixels and all of them were great scale images and we had ten classes and if you guys remember these are the classes we have class zero that belongs to a t-shirt plus one a trouser to pull over three dress for coat sandals shirt sneaker bag and ankle boot each image consists 428 pixel by 28 pixel and the total is 784 pixels in total all right and again all the numbers of these pixels range from 0 to 255 again all these are basics please if you are a little bit confused please go back to earlier sections of the course and you should find a ton of details regarding the images how to represent images digitally and so on all right please note that this section of the of the course you'll find on the of the project you will find that some of the code sections are based upon the tensorflow 2.0 documentation and here is the link to the documentation of the tensorflow to point out serving so first let's go ahead and import our packages as usual and then afterwards we are going to import our data set so here we went to TF the Charis code data set the fashion amnesty we loaded the data and here we have X trade white rain X test and white dust pretty simple and then we normalize the data so we divide by 255 so we normalize it and then we checked out the shape so now I have 60,000 images each is 28 by 28 that's for exchange for X tests we have 10,000 images each is 28 by 28 and what we're going to do here they're going to reshape our training data to be in form of 60,000 28 28 and 1 which is what we call it batch format instead of 60,000 28 by 28 and then again that would be very beneficial when we actually build our model so that's how we reshape it so I'm gonna say extreme dot reshape extreme the three chef's 0 which is basically 60,000 and here we're gonna reshape the pixels make it in a form of 28 pixels by 28 pixels by 1 okay and then afterwards we check out the shape so now beforehand it was 60,000 28 by 28 now I have 60,000 28 by 28 by 1 and then we're gonna do the same for the testing data so we received the testing data in the same fashion and we check it out the shape and now we have 10,000 images each is 28 by 28 body 1 let's go ahead and check out the classes so here are the list of classes so it created a list and within that list we created basically a bunch of strings in here of the different classes so we can basically print them out moving forward we check out the length of the exchange again 60,000 and what I'm going to do here is I'm going to create a grid and within that grid we're going to print basically 16 images 4 rows by 4 columns and we're going to visualize just we're gonna plot the image along with their training label so all my desired label my apologies so here I'm gonna pick a random number that random number ranges from 0 up to n training so between 0 and 60,000 and that with my index and every time I'm gonna say you know what go ahead and plot or use use image show we're going to show our X train our image I'm gonna pass along our index here and then we're gonna show its true label so I'm gonna say Y train I'm gonna pass along my index too so here I plot the image it's it's again it's a bag class 8 here it's a shoe class 9 it's AB a class 8 and so on and these are the different basically input-output data so that's my input data my image and that's my outputs that's my desired output that's what I want the model to be able to give me after training alright let's go ahead and build train and test our model so first we are going to use Cara's to build our model and as I mentioned Cara's is a default API when it comes to building and training artificial neural networks in tensorflow 2.0 so here we say t after Cara's the models of sequential we have done that so many times and then we build the first layer which is convolution layer has 32 filters each 33 by 3 the activation is raloo and the input shape here is 28 by 28 by 1 and then we added max pooling again the idea is to try to compress the image store feature Maps try to reduce the size another convolution again with activation of Lulu and then we flatten it up and then we add the bunch of dense Network so dense layer so we had a dense layer of 64 neurons another dense layer and that's the output layer it has 10 neurons because I have 10 classes and here we have our softmax function all right looks good and here we have 500,000 parameters to Train again pretty incredible afterwards we're going to specify how many epics so we specify we're going to run for 5 epochs and we're going to compile our models so that's how we compile our model so here we select TF trained add an optimizer and then we specify our loss which is going to be sparse categorical cross entropy and our metrics we caná-- be our accuracy so let's run that so that's how we compile them out and then here you are going to use the fit method on my model and we're going to pass along my X train y train and the epochs as well so let's run that and then after we train it actually we're going to evaluate the model as well so here we're gonna take our train model we're gonna use the evaluate method on my train model and pass along our testing data and here we go the models already at 84 85 percent accuracy again pretty incredible given in just one epoch now it jumped to 90 percent accuracy and it's getting better and better over time all right looks good so that's my second epoch again we have done that before that's why I'm going through it really quick here we have 92% accuracy and then we should reach I hope 96% I hope so 93 right now epic four and it's going up hopefully we'll reach 96 probably we gonna stop at 94 maybe 95 so 95 great ok I hope it's getting better so again within only 5 epochs we have been able to achieve around 95% accuracy on training and we have been able on testing maybe able to reach almost 91% accuracy on the testing data set and that's pretty good all right ok that's all what I have for this lecture I hope you guys enjoyed it and that's basically here haven't discussed anything new yet all that is just the basics the basic foundations now we train the model in the next lecture we're going to learn how to save our model and how to start to deploy our model on the server using tensorflow 2.0 serving and that's all what I have for this lecture I hope you guys enjoyed it and please enjoy tensorflow 2.0 and happy learning hello everyone and welcome to this new lecture I'm getting super excited because in this lecture we're going to learn how to save our model and how to deploy our model using tensorflow 2.0 serving so if you guys recall from previous lectures we took a model we trained it we fit it with the training data and now we have all the knowledge all the learnings or the train weights are captured within that object which is CNN object alright so first we now need to save our chain model and it has to be saved in what we call it in a saved model format so the model will have a certain version number and that version number will be saved in a structured directory so what's important to be noted here is that we want when we train AI models in general and we deploy them in practice we we generally all these models we have different versions of them why because again they are not mature yet we are continuously learning as we acquire data from the customers and these models are being trained all that training data is being fed back to the model to update the weights so now we have version 1 and then version 2 and then version 3 and maybe these models are deployed across different let's say regions we have models let's say deployed for example let's say in Brazil have on other models deployed and let's say in Egypt for example and then some in Canada or whatever so that's why we have different versions of the models so you have to keep track of these different versions and maybe name them as well so what you could do is to save the model is gonna say TF dot saved underscore model dot simple safe and this is a function that can be used to build a saved model that is suitable for serving using tensorflow serving so if you actually just apply simple save that should save the model and should save it in a special special structure that we can use afterwards tensorflow serving to be able to deployed our model afterwards so after the model is saved we can now use tensorflow serving to start making inference requests basically send that model an image and the model will respond back but we're not gonna do that locally though we're not gonna be doing this let's say on my computer for example we're going to be doing this while running the model on a server and that's kind of the the new part here the new element here so what I need to do is I need to specify what's my train model is called which is well call its servable so the train model now is called servable and now i have to specify or give it a certain version number which what version of that model I wanted to run so what happen is is that saved model would be able to we're going to be able to save and load our models so when you say the model that means we're talking about variables we're talking about on the graph of the structure of my models and all the metadata contained within our model so basically every single information every single trained weights all the architecture of the models how many neurons which neurons are connected to what and so on all that it will be captured when we run the simple save function and then what's really good about it as well that it was it's what called language neutral its recoverable which means that I don't need you know to to save the model with different formats for example what's good about tensorflow serving is that if you use simple save it will just save it in a certain structure that can be language neutral then I can call it easily easily using tensorflow serving and I'm gonna show you that guys in the next couple of lectures so let's see how can we use simple safe so first I'm gonna say simple save we have to specify which session am i referring to I have to specify what's my directory we're gonna be so I'm gonna save it in a certain directory and I have to specify what's my inputs what are my inputs and what's my outputs that's all what it is it's again it makes sense I need to specify again my directory where I'm gonna be saving the model what's my inputs and what's my outputs that's all what it is and if you guys wanna take get more information go ahead to tensorflow that org guide and then save model to this link all right so let's get started first we're going to obtain a temporary storage directory so I'm gonna say model directory you can call it whatever you want equal temp file the get temp directory and temp file if you guys recall we actually imported it beforehand so here we went a very important temp file okay so that's the first step so we imported beforehand now we can use it and then afterwards we can say ok show me my model directory so it will tell me ok now I have a temporary location a temporary directory called temp TMP and then we need to specify the version number so I'm gonna say my version of the model we're gonna be equals to 1 so my version equals to 1 looks good and now I would be simply I was just going to concatenate just you know mix up my model directory which is temp I'm gonna add backslash we're gonna add my version number so to do that I'm gonna say o s dot pass the join I'm gonna specify my model directory which is what we have here which is a temp and then I'm gonna take my version number convert it to a string format so a string of 1 so now it's a string join them together and that will be my export pass that will be my direct but that's that's where I want to save my model so and now I'll be able to print basically my export path which is gonna be export pass equals 2 and there'll be back slash temp by slash 1 and that's basically the directory we're gonna be saving my model so let's go ahead and save it it's actually pretty simple pretty straightforward so we're gonna say let's save my model so I'm gonna say if OS docpath dot is directory of export path so basically I'm asking if the model does not that path exists beforehand or not if it already exists then I'm gonna say ok already save the model so my order my mother's already saved let's clean it up so I'm gonna say go ahead and remove so bad so exclamation mark are M stands for remove so I'm gonna remove basically that directory and basically afterwards so I'm gonna say TF dot save model underscore simple save and that should basically say my session so I'm gonna save first of all get my session and then I'm gonna specify what's my export path we're going to look like and then I'm gonna specify my inputs and my outputs so my input here will be my CN n dot input and then my outputs because I have multiple outputs they're gonna say create a list comprehension so I'm gonna say for I in CN n dot outputs every time I'm gonna go call it I got name with I or it's kind of you know like when I run it you guys we're gonna understand what I'm talking about so when I run that okay looks good and when I list it here we go so now have been able to basically save my model and my saved model will be saved underscore model dot PB that's basically what all the intelligence all the intelligence have been captured already and has been saved and please bear in mind that now we have the model and that model basically is saved in that specific path which is temp underscore one let's take a look at it so I'm gonna say go to my files go ma go up here and then go to my temp you should find the temp here go to one and here we go so now I have my save model dot PB and I have my variables as well all right okay and that's all what I have for this lecture I hope you guys enjoyed it in the next lecture we're going to explore our save model and then gonna start serving our model using tensorflow serving please enjoy tensorflow 2.0 and happy learning hello everyone and welcome to this new lecture in this lecture we're going to explore our saved model and then we're gonna learn how to serve our model using tensorflow serving you will find some code here that doesn't make much sense okay but again all all the code lines in here they are pretty much standard so if you actually use them and deploy it with whatever model you have you should have no problem at all what you need to do basically is if you're gonna train let's say an algorithm to let's say classify for example at ciphered ten data set or let's say classify like smiling faces or classify whatever new data set what you need to do is just here when you save them all you just wanted to basically specify what's my directory we're gonna look like here okay what's my password look like and you need to just train the model basically all the steps that would have done beforehand and you need to save it and what's coming is pretty much the same consistent across every single basically step so here we said save my model I need to specify what's my object which is CNN inputs and CNN book outputs and that's where that's where all the maybe the differences might might might happen moving forward pretty much I'm gonna be pretty much the same so what I'm could do here is again explore our saved model so what I could do again I run that I'm gonna say clinician mark saved underscore model underscore CLI so saved underscore model underscore CLI will be used to explore or call it metagraph death the deaths and signature deaths basically I need to know what might what what is my model and I need to know what's the methods that I can apply to my model so basically when I say metagraph deaths that's basically my train model or my train weights and so on and the here all the methods that I can apply to it let's say I wanted to apply the predict method for example to make predictions based on my train model and basically what a meta graph contains it contains all the data flow graph all the actual architecture of the model all the associated variables and all the assets as well so let's run that so here I'm gonna say okay go to my saved underscore model CLI show I need to specify my directory so because here my export path is basically what I have here which is a directory of my model which is temp underscore version one is my apologies back slash version one that should show me my architecture basically of the model so as you guys can see here you'll find that here the shape of the input is 28 by 28 by 1 and that's make sense because I know that my model has been trained on the fashion M Ness data set which is again pretty pretty good because if you have any saved model and you don't know okay I don't know we're like you know who someone for example from a different team trained it and you want to know what to do mentions of the inputs and the outputs and so on you can just run this command that will show you basically the architecture of the model just the inputs and outputs from a very high level which is again pretty good here it will tell you the output shape is ten so I know that I have ten samples and the actual output is a soft max output or activation functions which again make sense all right let's go ahead and actually serve our models using tensorflow 2.0 serving so first we're going to add our tensor flow model server package to our list of packages so here we just wanted these kind lines of code again they look a little bit strange but simply what we're doing here is that we're trying to install tensor flow model server package that's all what it is these are the two lines of code so if you run it so we're gonna install it and then we're gonna update it as well to have the most up-to-date version so we are running it here and that should just install our model server package and that would be the first step okay the next step is they were going to install our tensor flow model servers so gonna say get install tensor flow model server so that should install my server and then after an install the server I should be able to actually run my tensor flow serving on the server so I'm gonna do here I'm gonna say ok to run the tensor flow serving what I need to specify I need to specify what's my port we're gonna look like what's my port number I need to specify what's my model name I need to give it a name okay and I need to specify what's my model base path what would going to be my path to my model if you guys recall from previous lectures that we saved our model already on that in that export path if you guys remember here we had our temp max - 10 pack / 1 if you guys remember so after we do that here's what we need to do is that we could first we are going to say that's my model directory that's where I actually saved my Mon and while I'm could do here I'm gonna say ok run my model server I need to specify what's my port number so here my port number is 8 501 I need to give it a name so here I called it fashion amnesty you can call whatever you want and then here I need to specify what's my path of my model we're gonna look like which is simply our model directory if you guys recall beforehand so if I run that it's telling me looks good so job zero has started that's a good thing that's me in the server is running alright so let's go ahead and run our server so first we need to specify our port so here we're gonna specify our port number will be 8 501 here I need to specify the model name so here I called it fashion underscore model you can call it whatever you want and then here I need to specify my model base path and here my model base path which is model directory which is what I specified before as temp backslash 1 if you guys recall we had our directory here which is export path temp 1 if you guys remember so let's go ahead and run that so what on that first they were on that second it's telling you starting the job number 5 in a separate thread and if we take a look at the log it will tell me that everything looks good so the model has been successfully if you check out here successfully know that server be version which is again that's very good so now we know that the model we know that the we know that the model version is 1 and now with the model is running on the server again this is a huge amazing accomplishment again it's it's very very simple but now we are actually having an actual AI model running on the server and now we're pretty much ready to send the request to that server and get feedback from it again pretty incredible so let's go ahead and start making requests using tensorflow serving so first I'm gonna create a function that function has called show it basically just plots our actual index here plots an actual image first and then on the title it will show me the actual I need to show the actual image the actual label to label for it along with the actual label for example let's let's let's let me show an example so here for example we had we selected a random number from my testing data and then here we show okay show me my index my actual M along with the actual label so I'm gonna say an example image is let's say sandal for example so here I relied on the class names which is simply my translator if you guys remember we had here our class names here this is our class names the guys remember so here I'm just going there with the index just saying okay show me it is the shirt it's a sneaker is it a bag is that whatever and that's how you do here all right let's go down down down down all right looks good and this is the most important part because now we're pretty much ready to send a requests okay so first I'm gonna create or call a JSON object and what I'm gonna do here they're going to make three inference requests so if I run this cell basically here I created a data and that data consists of three samples of the image I need X tests I'm just gonna select three samples so from zero one and two so I selected three images and now I'm gonna pip install Q requests and I'm going to import requests and this basically we're gonna enable me to start making server requests so let's see how can we make an actual server request so first here we're going to make a request I'm gonna specify the header so I'm gonna say my content type here I'm gonna specify my application and then JSON afterwards I'm gonna say okay let's make a request to my server but to make it a quest I need to specify what's my port number would look like which which model I'm gonna be running which version am I talking about and what's the name of the model and then I'm gonna pass along my data that's all what it is so I'm gonna say requests dot post and that's how you actually create a request and then I'm gonna say HTTP I'm gonna specify my localhost and that will be my 8501 which of you guys they call it 501 that was my port number when I actually define my server here 851 the name of the model was called fashion model if you guys do call as one so here I'm gonna specify first of all my version which is gonna be version 1 and then gasifier models I'm going to call this exact same name so they'll be my fashion underscore model and then I'm gonna specify : and then I'm gonna specify my method of calling that model what methods are gonna be referring to and here I'm gonna use a predict method to basically make predictions based on my model and then I'm gonna specify my data so here I passed along my data figure I remember that was my data that basically contains all the images here if you guys remember and then I'm gonna specify my headers which is why I specified here basically so here are gonna make a request to the server the server will respond back we'll give me the JSON response and then here this think of it as kind of a decoder it just decodes the message that's coming from the server and that would be my predictions that would be technically my output so let's run that now I've got predictions from the server so let's go ahead and decode my predictions and here we go amazing so now I call it my image and then I plotted them what I got from my server which is my predictions so here I said go ahead to my predictions or play my art max you guys remember I had a soft max function at the output layer so now I need to see which one has the highest probability of occurrence and I'm gonna go into my class names and then I'm gonna specify which one am i referring to out of the ten classes that I have so here I said the model thought that it was an ankle boot class 9 and it was action ankle boot again this is incredible now I'm actually running an actual AI model on the server giving it a request and our sending it a request and getting feedback and I'm being able to decode my feedback as well so what we're gonna do here we're gonna do the same thing as well but I'm gonna call the server basically three times I was just gonna call the server and make predictions and then I'm gonna create a for loop that can plot my images along with my server feedback along with my true label as well so when I run that and here we go you will find that here basically I had I got three images because I sent out the data containing three images I got three back back three requests and have been able to create a for loop every time I'm gonna plot my predictions of whatever index along with my actual why label which along with the actual true class so here the model thought that was an ankle boot class night and it was actually ankle boot that's good the model thought it was a pullover class too and was actually a pullover that's great too amethyst trouser class one and it's actually our that means my model is performing great performing perfectly and that's it that's all what I have for this project I hope you guys enjoyed it please enjoy tensorflow 2.0 and happy learning