Deploying TensorFlow Models in Production With TensorFlow Serving

[00:00:00][Music] Hi everybody. I'm Noah Fidel and I'm going to share with you today how you can serve your models in production with TensorFlow serving. So we're going to start with uh that image on the bottom right and really this is just showing how as an industry software engineering is quite mature and advanced and machine learning is still early on in the early days. So if you think about all the great things we're used to with uh software engineering and best practices that we've all come to understand things like source control and continuous builds we might take for granted in machine learning we don't even know what some of those things are yet and that motivates a lot of what we do with TensorFlow serving. So starting with a few stories uh who has taken a commercial airline in the last 10 years okay so half of us are awake and the other half probably have as well.

[00:00:46]Um, so going back about 12 years ago, I was working on a system that did the flight planning uh software for the majority of the world's commercial airflights. This included things you might care about like how much fuel to take on your plane to get somewhere. It didn't have source control when I joined. Uh, it did when I left. Um, and I was, you know, helping do that. But this kind of motivates that even though source control was an understood thing at the time, just because we have a best practice doesn't mean everybody uses it yet. Um, and that's something with tensor serving. We want to make uh best practices that you all can use and make them easy enough that you get them uh kind of by default. So in 2010 I was at a small startup doing uh photo sharing on on mobile apps. Um it was really great. We had some great engineering. We tried to do all the best practices. We had source control and continuous builds and so on. Um and one day we decided to add a machine learn model to do face detection and autocropping and it was really great. We loved it. Uh our users loved it. Our investors really loved it.

[00:01:37]Um but we had no continuous build for machine learning. Um, and at the time the tools were kind of ad hoc enough that we checked in the weights uh to the model in the execution environment. Uh, but you couldn't retrain the model, right? And now it's a kind of understood best practice. You want to be able to do that. So here we are today in 2017. We have lots and lots of great tools. Uh, but really just, you know, we're getting started and we have a long way to go for machine learning. Uh, and hopefully all of us together can build a great ecosystem around machine learning and best practices and so on and share those. So here's the agenda uh, for today's talk. Uh to answer the first question, what is TensorFlow serving?

[00:02:11]I'm actually going to introduce what is serving. So some of you probably know what it is, this might be kind of the basics for you. Uh but for those who aren't familiar with it yet, um if you ever want to deploy your machine learning uh into production, you'll want to become familiar with these concepts.

[00:02:24]So serving is uh how you apply a machine learn model after you've trained it. Um seems pretty simple and straightforward.

[00:02:31]It has some attributes though that might be unexpected or new to you.

[00:02:36]uh and you know taking another view on this uh hopefully everybody is familiar with the orange boxes on the left side of the slide. So you have some data you're going to use that uh to train a model in TensorFlow and then you have that model and you have your application on the right. Uh let's say you're showing video recommendations to users in your application. Um somehow you need to get that model such that it's usable inside your application. So the most common way to do this is using an RPC server. uh and TensorFlow serving can be used both as an RPC server and as a set of libraries inside your app on embedded devices or in mobile. So a key attribute for serving is that it's online and low latency. And this differs quite a bit from uh training and other big data pipelines you might run uh your users don't want to wait for the recommendations. They're not going to wait a minute. They're probably not going to wait 10 seconds.

[00:03:23]You really need to be fast and consistently fast. Um, another thing that's, you know, different from many, uh, big data tools out there is that you'll have many models in a single process. So, let's say you had a great production model, it's serving great recommendations to your users, and then you have a new experiment you want to run. Um, the most common pattern that people do is they want to load uh, say a second experimental model in their server uh, alongside their main production model. So, it's really common. Another thing is an emerging best practice that we're seeing a lot uh inside Google and we want to bring uh more to industry is that you'll have many versions of your model over time.

[00:03:57]Uh so the data that you trained on last week might not be as relevant to the data you gathered yesterday. And so you want to continually train your models and continually deploy them. Uh last but not least, uh I'm just going to mention the last bullet here. Uh is anybody familiar with mini batching? Okay, just a oh good number.

[00:04:15]So mini batching is where you take a bunch of examples at training time, maybe queue them up, and then you run them through your graph together to get more throughput and higher efficiency.

[00:04:24]Um, and this is really great. It lets you train models more quickly, uh, process a lot more training data. Um, and so the challenge here is how can you do that in a production setting where all of your requests arrive asynchronously and you want to keep a nice bound on latency and we have some good tools both at the library level and binaries for you to do that. All right, so moving on to what is TensorFlow serving? So it's a flexible high performance serving system for machine learn models and it's designed for your production environments. Tensflow serving has uh three major uh pillars. The first are C++ libraries. These are what we use internally to build our binaries. Um and they're you know all open source. Um and they're very low level. They include things like how to save and restore a TensorFlow model as well as how to load new model versions over time. uh we have binaries that incorporate all of the best practices we've learned thus far uh out of the box and for the emerging ones that we're not sure if it's a best uh practice yet, you know, flag so that you can enable and try them. Uh we also have uh Docker containers and tutorials and code labs that let you autoscale our binaries on uh Kubernetes and so on. And finally, we have a hosted service with Google Cloud ML uh as well as an internal version of that.

[00:05:36]Okay, so diving into our libraries. Um, our core platform is completely generic.

[00:05:42]Um, and what this means is let's say you have a current system, maybe you're serving on a legacy or in-housebuilt ML system and you want to adopt TensorFlow, but for some transition period of time, you might want to mix that in with your legacy system. So, our core platform lets you include any uh C++ class you want as a servable. Um, the components of the libraries are all alikart. So every component if you go look at our page um every single class we have is used by some customer inside Google on its own and in just about any combination that you can imagine uh they're also uh used and deployed and taking the last three bullets together the highlight here is that uh all of our APIs uh by and large allow you to plug in your own implementations uh so you can you know add support for different model storage maybe uh you're already an existing user of ML maybe you have hundreds of models in a database uh if We provide a way to get models off of a file system. You can easily extend that to support a custom database or custom data store. Okay, so I'm going to walk you through. Hopefully this is not too complex. Um, this is our libraries at a high level inside TensorFlow serving.

[00:06:47]Uh, and they're also arranged here exactly as they are inside our binaries that we pre-build for you. So the green boxes are our standard kind of uh, abstractions with APIs and the yellow boxes are plugins into those abstractions. So uh let's imagine that this is a server and you're doing video recommendations and you have version one of your model loaded and it's called my model. So you have a source and the source is responsible for identifying new models uh and that should be loaded.

[00:07:16]So we have a file system plugged into the source. It does exactly what it sounds like. It monitors a file system sees that there's a new version. and let's say aha version two arrived right and now we want to load version two uh because it has you know was trained on fresher data and it's going to provide uh better recommendations to users so it's going to create a loader of a tensorflow saved model and it's important to note that the loader of a saved model knows how to load the saved model and it knows how to estimate the resources such as RAM or GPU uh storage that will be used by the model it doesn't actually load it yet uh that's the job of the manager so this loader is emitted to the manager as an aspire ired version and it's actually up to the manager to figure out when it's ready uh when it has thread scheduling available uh and when it has enough resources to load that new model version it'll do so.

[00:08:02]And this is where one really key uh plugin comes into play which is the version policy. And it turns out that in almost all scenarios if you're serving something like video recommendations to users you want that serving system to always be available right you never want to have a downtime. Um on the other hand there are use cases out there where let's say you have an offline uh big batch pipeline that's maybe annotating these videos in batch right and it's not directly userf facing um and let's say that your model is very big you might uh prefer to um have a little bit of unavailability in that pipeline and save a bunch of memory right so instead of loading both versions of that model at once you can actually delete the old one and then load the new one and just have a little hiccup in your serving. Uh so the version policies let you preserve availability or preserve resources. So it's a a nice thing there. Okay. So on to some strengths of the libraries. Uh first and foremost is they're optimized for very high performance and robustness. So these are used in some of the largest uh serving systems at Google u at you know pretty large scale. Um and uh we do things like you know ref count um accesses to your models. So uh from the previous slide, let's imagine that version two of the model is loaded and this is kind of getting into the details, but uh version two is loaded. You can't actually immediately unload version one because there might still be requests pending to it. So you actually keep track uh via ref counting of the requests in flight on each version of your models and only after all the requests have quesed can we delete a version that's no longer needed. Um, and last but not least, just kind of re-emphasizing, uh, we have these nice plug-in APIs, so you can build your own sources, uh, to get models from a database, uh, an RPC layer, a pub subsystem, kind of whatever you like. Okay, so uh, last show of hands, who's used TensorFlow serving libraries so far? Great, a few of you. So, um, for anyone who's thinking of using this, you would just see this, uh, by default.

[00:10:01]This is in our tutorials, but for those of you who've used the current libraries, what we found internally and externally is that you had to uh setting up all of those libraries with the best practices took a bunch of boilerplate and kind of common code. So we made a new class called server core. And what this does is it lets you declare the set of models you want loaded and pass them to server core and say just give me a manager of these models please. I don't really care about the details. Give me all the best practices out of the box.

[00:10:25]Um and so this should let you delete about 200 to 300 lines of code. So give that a try. uh send comments and feature requests. Okay, so this slide is intentionally short. Uh the binaries are very very simple. They take all of the best practices from the libraries of TensorFlow serving and just wrap them in a gRPC layer um along with some you know flags and configuration and monitoring and other things you would need. Our binaries are based on gRPC which is Google's high performance uh open source universal RPC framework. Um and you can extend this as well. uh but this is what we provide out of the box. In terms of the specific APIs uh currently open source complete with uh the API and implementation we have a low-level tensor oriented predict API. Uh so this should be usable for any kind of modeling uh inference that you would need to do. Coming soon we have uh regression and classification interfaces. The APIs are already on GitHub and the implementation is out for review. So it should land very soon. All right. Uh now I'm going to move on to some challenges that we've seen uh along with the best practices for how to solve them. So the first one this was a great uh story around isolation and latency. So uh this is a graph of latency with um each of those spikes is approximately daily mountain view time at noon. Uh so this is for a large Google customer serving several hundred thousand queries per second. Uh and right around give or take noon there was this latency spike at the 99.9th percentile um that put that serving customer outside of their SLA. So we looked into this and it turned out that there was a fantastic engineer we've been working closely with uh really liked to push uh new experimental models right before after you went to lunch. Um who knew and it actually threw us for a loop because it it did move forward and backward by an hour or two every day.

[00:12:10]And so we we didn't find any automated jobs doing this. Um but you know we did find the problem. It was loading a new model could have a pretty big latency impact on your existing models running inference. So if you dive in a little bit deeper, it turns out that the default, and I think it's a very good default for TensorFlow, is to optimize for throughput. That's what most of us want in most situations. So you don't have to go set a flag that says, "Please give me throughput." Um, but a side effect of that is by default optimizing for throughput. Um, pretty much all the models and all the sessions on your task will get access to all of the threads.

[00:12:41]Um so again very good default but for the specific case of you have many models and many versions loading over time what you really want to do is isolate the loading threads away from your inference threads. So we added you know it's a pretty fancy slide but um we added just one flag you can set the number of loading threads and we typically set it to somewhere around one or two um and I'll show you the after slide. So uh you can see much more detail but the main uh point from here is that the y-axis dropped by 10x. So the latency spikes now are you know roughly 100 milliseconds. This is completely inside the customer's SLA.

[00:13:14]Customers happy. We have a best practice now that all of you can use. The next challenge and I hinted at this earlier in the talk is about batching and handling asynchronous requests and getting the efficiency of mini batching at serve time. So if you look at the uh little graphic on the right, you have the green, blue, and orange boxes. And these represent your user queries arriving at different times uh kind of falling down towards your server. And uh what you want to do ideally to get that mini batching like performance is kind of wait for some period of time, take a few requests, put them together, and run one graph computation. At the same time, you really want to keep a uh strict upper bound on the latency you're willing to to wait. Um, so we've done quite a bit of investment here. Uh, our batching, uh, is available as a library as well as via flag for the server. Um, and there are a lot of interesting nuances here that, um, we didn't even realize what we were getting to when we first started doing this. And as just one example, let's imagine that you had two models on the same machine. What you might want to actually do is schedule their batches back to back instead of having them overlap and contend for threads. Uh, so we have the concept of a shared batchuler. So each model has its own and they cooperate via shareduler. Uh as a teaser for later, uh sequence models make this really interesting uh given that they're uh typically not just a feed forward network. Uh and the sequences can be of varying length. So it gets very tricky and challenging for how to batch these together. And uh Eugene is up in a couple of talks. He's going to go into much more detail about batching and and sequences. A few areas of emerging tech.

[00:14:50]Uh so Jonathan was mentioning same model. uh we're uh quite excited to announce this and encourage everybody uh to adopt it. So saved model is the universal serialization format for TensorFlow models. It has uh it's included in TensorFlow 1.0 and also in TensorFlow serving as of right now live on the repo. Uh there are two marquee features of save model. So the first one is support for multiple metagraphs. Uh and for folks who have been you know mostly doing training this you might not know why you'd want this but once you uh here's some more details you'll see at serve time it's really important to have different metagraphs and the use case here is let's say you were training uh a model and you went to save it for serving. So you typically want to remove things that are training specific like input cues and dropout layers. You really don't want those uh in your production serving models. A few people have tried you get really bad results um or no results at all if they just hang on a cube. Uh other things you might want to do is you might want to transform your serving graph. So you might want to have a quantized graph for serving on a GPU or a TPU. And what the multiple metagraph support does is it lets you have as many metagraphs as you want and you can store and access them uh by simple tags. So you can tag one with serve, one with train, one with serve and GPU or TPU and so on. The next concept is signature defs. So a signature defaf defines the signature of a computation supported by a TensorFlow graph. And this is really important. So if you look at the slide, most people probably can right away, especially for humans, we're good at this. You can read input classes and scores and probably figure out what's going on with this graph. Um, but if you handed this graph without those labels to a serving system, it would probably have no idea what to do with it, right? With these graphs, you can feed or fetch from just about any node in most graphs. So how would you identify where your input goes and where your output goes? And that's what signature defs do. So in this case they uh specify that that middle node on the left is where you want to feed in your TensorFlow example and the top right node is your string classes and the bottom right is your uh floatingoint scores for say a classification. Building on a signature def uh we're adding support for multi-headed inference. So this is one of the top feature requests we've had from the community. Uh is anyone familiar with multi-headed inference? Okay, very few. Uh this is going to be really popular and more common as people productionize and deploy uh machine learning in their products. So uh it's really really common that you might start with one model, maybe a click-through model uh to do predictions on say your video recommendations. Um but over time you might decide, you know, we'd also like to train a model for conversions or maybe people who watch the whole video.

[00:17:21]Um and quite often people would go about this by actually training a second model. But for starters, you're training on almost all the same data. uh all of your data pre-processing. All of your um hidden layers quite often are the same.

[00:17:34]Um and then at uh serve time, you're doing a lot of redundant computation. So you're parsing your input, pre-processing it, doing the same, you know, hidden layer computation, and really only the output layers change. So our multi-headed inference support builds on signatures. It lets you specify multiple signatures together and say run these together uh in one request and it'll do that.

[00:17:55]uh performing multi-head inference will save you uh operational overhead and complexity of deploying multiple models.

[00:18:01]Uh it'll save you bandwidth, CPU, latency and so on. Uh so we're really excited and that should land sometime in the next week or two. All right, so wrapping up and again uh Eugene is going to go into much more depth on sequence models. I just wanted to highlight that there are many flavors of sequence models. Uh they are generally very expensive uh to serve both in compute cost and latency. And we're investigating specifically uh batching, padding, and unrolling strategies to make them more efficient and effective uh for all of you to serve. All right. So, uh for anybody who's interested, uh you know, we really warmly welcome collaboration, pull requests, uh feature requests, bugs, and and and so on. Uh we sync all of our code uh to GitHub weekly. uh we have a developer on call and it includes facilitating your pull requests, answering questions, uh comments and and ideas and so on. Uh to get in touch you can reach us at discussensorflow.org. So uh here are a bunch of links for how you can get started. Uh I'll let you read these offline and please do get in touch with your questions and feedback. All right, so thanks very much and up next is Ashish.

[00:19:11][Music]