Implementing Transformer Models with Hugging Face & Gradio

[00:00:00]hey everyone I'm Shaw and I'm back with the third video in the series on using large language models in practice in this video I'm going to be breaking down a hugging face Transformers Library which is a python library that makes working with open source large language models super easy I'll start by explaining some key Concepts before diving into some concrete example code and then at the end of the video we're going to see how we can spin up our very own chatbot UI using Transformers and gradio and with that let's get into the video so in the previous video of the series we were talking all about the open AI python API and what this API allows you to do is to programmatically interact with open ai's language models so you can build tools or if you want to build some kind of product or service however one obvious downside is that API calls cost money so in some situations where this cost might be too prohibitive we can turn to open source Solutions one way we can do this is via the hugging face Transformers Library which is what I'm going to talk about today so you're probably wondering what is hugging face well it's more than just an emoji on your phone hugging face is actually an AI company and they've become a major hub for open source machine learning in the past few years so there are three key elements to hugging faces ecosystem which is largely contributed to its recent popularity the first one is its models there are hundreds of thousands of pre-trained Open Source machine learning models freely available on hugging face second is their data sets repository so these are open access data sets that developers and practitioners can grab to train their own machine learning models or to fine-tune existing models and finally is hugging face spaces which is a platform that allows users to build and deploy machine learning applications so while these three aspects of hugging faces ecosystem have made developing machine learning models more accessible than ever before or there's still another key element of the ecosystem worth mentioning which is the Transformers Library so Transformers is a python Library developed by hugging face that makes downloading and training machine learning models super easy so while the library was originally developed specifically for natural language processing its current functionality spans all different domains from computer Vision Audio processing multimodal problems and more so just to give you a flavor of how easy it is to get started with the Transformers Library let's look at a concrete example suppose we want to do sentiment analysis you can imagine that there could be a lot of steps involved in doing sentiment analysis so first you need to find the model that is able to do this classification task then you'll need to take some raw text and convert it into a numerical representation that you can pass into the model and then you need to kind of decode the numerical output of the model to get a label for the text input and so while this might sound like many different steps and could be very complicated this can all be done in one line of code in the Transformers library and so this is possible with the pipeline function as you can see the syntax is super simple here so we have this pipeline function we just need to specify the task we want it to do so here we put sentiment analysis and then we pass to the pipeline function a text input so here I put love this and then if we run that the output of this line of code here is a label of positive and a score associated with that label so this makes sentiment analysis super easy but of course sentiment analysis is not the only thing we can do with the pipeline function you can also do summarization translation question answering feature extraction text generation and many many more if you want the full list it's available on hugging faces documentation this is the link down here and I'll also drop it in the description below going back to this one line example here it almost feels like magic because we didn't give it a model we just said hey do sentiment analysis and apply it to this text here we could have been a bit more explicit here and specified the model that we wanted to use to do sentiment analysis and so to do that it's very simple we just specify a model using the syntax here we're using distillber base uncased fine-tuned SST to English so this is actually the default model in the Transformers library that was used before so that's why we have the same exact output but what really makes Transformers powerful is we could have put any one of the thousands of text classification models available on hugging face and so to explore these models we could have gone to huggingface.co models which is a growing repository of pre-trained Open Source machine learning models for things such as natural language processing computer vision and much more so let's see what this looks like so we'll click on here and I'm going to zoom in a little bit we can see here there are currently 200 184 000 models on the platform and if we look on the left here these are for all different types of tasks they're these categories multimodal computer vision natural language processing audio tabular reinforcement learning and so just now we were doing sentiment analysis which is a type of text classification so we can see what other models we could have used here so just zooming in there are over 28 000 models we could have used for text classification but notice we can add additional filters to narrow down the models we want to pick so let's say we want to make sure we can use the model easily with the Transformers library in that case we can just click on this Transformers filter and then it'll narrow down even more so we went from 28 000 models to about 27 000 models so still a lot of different options we can also go further we can specify the data sets we want it to have been trained on so for example if we wanted to be trained on PubMed because we wanted to use this for a medical use case and you know so on languages licenses so I guess this is important if you have a specific use case in mind for example you want to develop a product for commercial use you want to make sure that the license kind of aligns with what you're trying to do and then there's some other filters as well so let's just say that we're fine with any text classification model that is compatible with the Transformers library then we can kind of explore from here right now it's sorted by trending but we can also sort by number of likes number of downloads recently updated so the number one most trending one looks like it's from the bloke and the model name is llama270b guanaco Q Laura fp16 so we can go ahead and click on that and if we click on that we get the model card just to kind of explore this a bit here's the organization or the individual that created the model this is the model name we can easily copy the model name if we want to paste it into our pipeline function we can do that very easily also we have all the different tags for the model here we have the text classification tag it's available in Transformers which is what we filtered on but also this model is compatible with pytorch this is an important point because the models on hugging phase aren't only for the Transformers Library they are also for many other popular machine learning Frameworks so if you're not using the Transformers library but using pi torch hugging faces models repository can still be a very helpful resource so another cool thing about the model card is that there are these like quick start buttons here so we can get a quick start for like fine-tuning the model on Amazon sagemaker this will give you some code to jumpstart your model fine tuning and let's say you don't want to just do text classification you want to do text generation or token classification this will give you some example code to get you started also there's deploy so you can deploy this model using Amazon sagemaker and then also you can use in Transformers so this is the same syntax we saw in the slides where you're just specifying the task you wanted to do and then just specify model and then coming out of that there's just a bunch of general information about the model laid out here now you can imagine that we have these model cards for hundreds of thousands of models on this platform if you're trying to build some kind of machine learning app it's really never been easier to get started you can just use Transformers to load in any of these state-of-the-art open source language models and just start building from there so hopefully you have a little bit of a sense of what you can do with the Transformers Library your next thought might be like how do I get this on my machine and start using it so the standard way to install the library is via PIV and hugging face has a great guide on their website on how to do this and so I'm not going to walk through the PIP installation steps here however I will walk through a conda installation specifically for the example code that we will see in the following slides there's two steps the first step is head over to the GitHub repository and download the HF Dash EnV yaml file so here's the GitHub this is linked in the description below here we have the HF Dash env.yaml file it's a yaml file listing out all the dependencies for the example code and here we of course have the Transformers Library along with all the other dependencies and so once you download that this next step is to execute the following two commands in your terminal or anaconda command prompt depending on the machine you're using two commands here one is you're going to change directories into wherever you have this HF env.yaml file saved and then you can create a new content environment using this command here so conda envcreate dash dash file HF env.yaml it'll probably take a few minutes to download all the dependencies but once that's done you should be good to go and you shouldn't get any kind of Errors running the following example code so while Transformers does more than just NLP these days and the example code here we're just going to focus on NLP tasks first we'll start with sentiment analysis so before we saw something like this where we use the pipeline function to do sentiment analysis and we specified the model but what's different here is instead of just doing it all in one line we use the pipeline function to create this classifier object and then we can take this object and pass in text to it and it'll generate an output like this so if we pass in the text hate this it'll spit out a label negative and a score associated with that label however you're not limited to just passing one input at a time into this classifier object you can actually pass in a list and it'll do a batch prediction for all the elements in that list so for example we have a text list here so this is great thanks for nothing you've got to work on your face your beautiful never change and so we pass these into the classifier and we get the following labels so it says the first one is positive since the second one is positive even though I sent some sarcasm there the third one is negative and then the fourth one is positive but of course there are more models than just this default one on hugging face that we could have used so one example is Roberta bass go emotions by Sam Lowe so the difference with this model and the default model is that the model here has several Target labels that it uses for text classification and so just using the same exact syntax we can create a classifier object using the pipeline function and then we can apply the classifier object to the first element in our text list defined before and so you can see now instead of just positive or negative there are tons of labels so there's admiration approval neutral excitement gratitude Joy curiosity so on and so forth and actually there are even more but I just cropped the image because there's just too many super easy to do sentiment analysis with Transformers and super easy to swap out this model with any other model you like on the hugging face platform another thing we can do of course is summarization so even though this is a completely different task the syntax is very similar so we use the pipeline function to specify the task and the model we want to use and create this summarizer object then we Define the text that we want as input into this object then we pass it in along with some other input parameters so here we're defining a minimum length the maximum length and then we do a couple things to retrieve just the summary text so this is text from the blog associated with this video it's just talking about hugging face and what it is and it takes this couple paragraphs of text and it reduces it to the following two sentences hugging face is an AI company that has become a major hub for open source machine learning they have three major Elements which allow users to access and share Machine learning resources and of course you can chain together multiple objects so for example you can bring together summarization and sentiment analysis by passing the summarize text into our classifier from before and we can generate all these different outputs and so really these become like Lego blocks and you can just piece together very easily very quickly these different NLP tools for whatever particular use case that you're working on another NLP task we can do is conversational text the syntax is slightly different because in a conversation like with a chatbot there's a bit of back and forth but we started with the same exact syntax so we can use the pipeline function to specify the model that we want to use to create this chat bot object and then we can use this conversation object to handle the back and forth between the user and the chat bot how we do this is we pass in the initial user prompt into this conversation object and then we save it as conversation then we can update this conversation object by passing it into this chat bot object and we can print the result and so what this looks like is the user says hi I'm Shaw how are you and then the chatbot says I'm doing well how are you doing this evening I just got home from work then to keep the conversation going we can use this add user input method so here we're adding a follow-up question where do you work and again we pass the conversation into the chat bot and then Auto magically the chat bot will generate a response and it'll update the conversation object and we can print it all out here so follow up where do you work then the chatbot says I work at a grocery store what about you what do you do for a living and all of this is just running locally on my machine no need for API calls no need for cloud resources these models are small enough that it's just running locally okay so while this is giving us very powerful functionality this is a very awkward way to interact with a chat bot so let's see how we can spin up a user interface to make this a bit more intuitive we can actually do this very easily using a library called gradio in just a few lines of code so first I'm initializing these two lists so one list is to store the user inputs and another list to store the chat bot responses and then we Define a function I call it vanilla chat bot here and it has two inputs it has the message which is essentially the user input and the history which is just the history of everything that's happened in this python script it is automatically generated and so in this vanilla chatbot function we use the same exact syntax we used in the previous slide so we have this conversation object we're passing in the message so this is the user input but also we can pass in the context of the conversation so the chat bot knows what the back and forth has been up until this point the way we do this is we pass in the message list and the response list so all this goes into conversation and we pass the conversation to the chatbot who will generate a response appendix response to the conversation and then we will return the latest generated response from the conversation object and then in basically one line of code we can spin up the chat interface with radio they have this chat interface base object that we can readily use so the first input of this object is the function we just defined the vanilla chat bot and then we can Define some other things like the title of the user interface and then a description for the user interface so all this gets stored into this demo chatbot object and then we simply can just launch it if we do that it'll start running locally at this URL here so if it doesn't automatically pop up in your browser you can just copy paste this into your browser also you can create a public link to this chat interface which is pretty cool you know you can spin this up and then you can send the link to someone and they can actually access the application that you made locally so let's see what this demo looks like here's the chat bot running if you run it in Jupiter lab it'll actually spin up the UI in the notebook or you can just click on this which will open a new tab and then you can start talking with the chatbot so let's just talk to the chat bot gotta response pretty quickly hello how are you doing today I just got home from a long day work so this chatbot for some reason always wants to throw it in our face that it had a long day at work and it seems like it wants me to ask what it does for a living so this one might take a bit more time but we'll see oh Works in a warehouse it's pretty boring but pays the bills how about you there's some other things here you can like have it redo its output you can undo you can clear it it gave a different response that's cool I'm gonna cashier to the grocery store isn't the most exciting job in the world so it doesn't really matter what this chatbot does for a living it just doesn't enjoy its work so that's pretty cool however we can take this one step further and instead of Hosting this chat bot locally we can host this chatbot on huggingface Via hugging face spaces essentially spaces are just git repositories that are hosted by hugging face and they allow you to create machine learning applications and so let's see how we can do this go to space's website you know we'll actually find a lot of existing applications so let's see this is a really popular one so this is the open llm leaderboard so we can see them ranked here with all these different metrics and you can filter as you like so that's pretty cool these are just open source machine learning applications that anyone can access so you can actually look at the source code by going over to files here and then you can see how they did it you can also clone the repository you can run with Docker so it makes it really easy to not only deploy your applications but find applications that are already on Spaces to use as a starting point but going back to spaces if we want to create a new space we just go here click create new space you can create a name we'll call it vanilla chat bot license doesn't matter we'll use radio as our SDK we'll make it public since this is hosting the application it needs computational resources so you have a few different options here you can have gpus or just CPUs of course A lot of these cost money so we'll just stick with the free version for this example we'll hit create space next you'll see something like this you it's giving us some nice instructions on how to upload our application to spaces so first it wants us to clone the repository create our app.pi file so this is where our application is going to go and then pushing our code to the repository and so an important note here is you need to add requirements basically what libraries are necessary to run your application when you do to make the push so let's see what this looks like we can clone the repository just copy paste this into terminal and we got it here so I already have the files ready to go in a different folder called my first face and so you can see that they're there if we look in the repo we can see it has a readme file already so all we need to do is add this app.py file and this requirements.txt file so I'll do that on my other screen we do that again we should see those files there okay so we clone the repo we added our app file and requirements file and now we just commit and push so here OBS decided to stop working so I lost audio but you can see me pushing the code to the git repository now I'm waiting around for the image and app to spin up then it finally did spin up and we see that we have this nice interface completely hosted on hugging face spaces the chatbot indeed does work but it was significantly slower here than it was when I was running it locally and since this is publicly available you can actually go and interact with this very same chat bot using the link on the screen and also in the description below so that's basically it I hope this video and demo has been helpful to you and given you a flavor of what's possible with the hugging face ecosystem while it does seem like we covered a ton of information and content we've only really scratched the surface of what's possible with the hugging face Transformers library and broader ecosystem so with that being said in future videos of this series we will explore more sophisticated use cases of the hugging face Transformers Library such as how to fine tune a pre-trained large language model as well as how to train a language model completely from scratch couple other things I'll call out is that there is a Blog associated with this video published in towards data science they're definitely details in there that I probably left out of this video and of course as always all the code that I covered in this video is available on the GitHub repository linked here and in the description below and if you enjoyed this content please consider liking subscribing and sharing with others and as always thank you so much for your time and thanks for watching