Data Science with Python: A Beginner's Tutorial

[00:00:03]hello and welcome to this session on data science with python so what are we going to do in this session we will talk about what is data science and some of the basics of python as you may be aware data science we use either python or r as some of the tools and programming languages so this session the focus is on python so we will talk about the basics of python why to learn python how to install python and then we will talk about some of the important libraries which are required for data analysis and then we will go into little bit of details about exploratory data analysis and we will take an example there of loan prediction and we will see a little bit about data wrangling using pandas which is one of the libraries of python and then we will end up with small predictive model or logistic regression model which is a part of machine learning and in case you need more details about machine learning in this session we will probably not go into the details of machine learning there is a separate session you might want to look at there is a separate video on machine learning you might want to take a look at that if you need more details here we are just giving a quick overview to see how we can use the scikit-learn library so we will not go into a lot of details about the basics of machine learning all right so with that let's move forward so what is data science data science is about finding insights from the data so if there is a lot of data if you have sufficient data how to analyze and find some insights out of it this is what is data science all about a couple of examples here customer prediction now let's say you have a customer base and you want to find out who are most likely to buy your product so you can use from your past behavior you can probably develop a model and try to predict who are the people out of the thousand leads or potential customers who will actually buy so there may be some patterns that you can use to predict similarly service planning so for example you're running a restaurant and you want to know how many people will be coming or how many customers will be visiting your restaurant on a given day now based on your historical data you can build a model to predict that as well so that there is no wastage of food and so on and so forth so these are very quick and easy examples of how data science can be used in business now let's talk about python for doing data science we need some kind of a programming language or a tool and so on so this session will be about python there are other tools like for example r and we will probably do a separate video on that but this session is on python and you must have already heard python is really becoming very popular everybody is talking about python not only data science in iot and ai and many other places so it's a very popular it's getting very popular so if you are not yet familiar with python this may be a good time to get started with it so why do we want to use python so basically python is used as a programming language because it is for data science because it has some rich tools from a mathematics and from a statistical perspective it has some rich tools so that is one of the reasons why we use python and if you see some of the trends if you're probably tracking some of the trends you will see that over the last few years python has become programming language of choice and especially for data science sas was earlier one of the most popular tools but now increasingly python is being used for doing data science and of course as well as r one of the reasons of course is that python and r are open source compared to sas which is a commercial product so that could definitely be one explanation but beyond that i think it is the ease of understanding this language the ease of using this language which is also making it very popular in addition to the availability of fantastic libraries for performing data science what are the other factors there are speed then there are availability of number of packages and then of course the design goal all right so what are each of these design goal primarily the syntax rules in python are relatively intuitive and easy to understand thereby it helps in building applications with con size and readable code base so with a few lines of code you can really achieve a lot of stuff and then there are a lot of packages that are available that have been developed by other people which can be reused so we don't have to reinvent the wheel and last but not least the speed so python is a relatively faster language of course it is not as fast as let's say crc plus plus but then relatively it is still faster so these are the three factors which make python the programming language of choice so if you want to get started with python the first thing obviously is to install python so there is some documentation and there are some steps that you need to follow so we will try to briefly touch upon that otherwise of course there is a lot of material available on how to install python and so on you can always look around but this is one of the again there are different ways in which you can also install python so we will use the anaconda path there is a packaging tool called anaconda so we will use that path you can also directly install python but in our session we will use the anaconda route so the first thing you need to do is download anaconda and this is the path for that and once you click on this you will come to a page somewhat like this and download you can do the corresponding download based on whether you have a windows or ubuntu there is a also a downloader possible for our package available for ubuntu if you are doing something on ubuntu so based on which operating system in fact this page will automatically detect which operating system you are having and it will actually suggest so for example you see here if you are running mac os then it will automatically detect that you have max and the corresponding installers will be displayed here similarly if you're on some flavor of linux like ubuntu or any other then you will get the corresponding download links here and then beyond that you can also select which version of python you want to install of course the latest version is in the three point x range at the time of recording this 3.6 is one of the latest versions but some of you may want to do or start with the earlier version which is python 2.7 to point x and you can download that as well if you don't have anything installed then my suggestion is start with python 3.6 all right so once you do that you will be able to install python and you will be able to run jupyter notebook okay so now that you know how to install python and if you have installed python let's take a look at what are the various libraries that are available so python is a very easy language to learn and there are some basic stuff that you can do for example adding or printing a hello world statement and so on without importing any specific libraries but if you want to perform data analysis you need to include or import some specific libraries so we are going to talk about those as we move forward so pandas for example is used for structured data operations so if you let's say are performing something on a csv file you import a csv file create a data frame and then you can do a lot of stuffs like data munching and data preparation before you do any other stuff like for example machine learning or so on so that's pandas scipy as the name suggests it is kind of it provides more scientific capabilities like for example it has a linear algebra it has fourier transform and so on and so forth then you have numpy which is a very powerful library for performing n-dimensional or creating n-dimensional arrays and it also has some of the stuff that is there in sci-fi like for example linear algebra and fourier transform and so on and so forth then you have matplotlib which is primarily for visualization purpose it has again very powerful features for visualizing your data for doing the initial what is known as exploratory data analysis for doing univariate analysis bivariate analysis so this is extremely useful for visualizing the data and then scikit-learn is used for performing all the machine learning activities if you want to do anything like linear regression classification or any of this stuff then the scikit-learn library will be extremely helpful in addition to that there are a few other libraries for example networks and igraph then of course a very important one is tensorflow so if you are interested in doing some deep learning or ai related stuff then it would be a good idea to learn about tensorflow and tensorflow is one of the libraries there is a separate video on tensorflow you can look for that and this is uh one of the libraries created by google open source library so once you're familiar with machine learning data analysis machine learning then that may be the next step to go to deep learning and ai so that's where tensorflow will be used then you have beautiful soup which is primarily used for web scraping and then you take the data and then analyze and so on then os library is a very common library as the name suggests it is for operating system so if you want to do something on creating directories or folders and things like that that's when you would use os all right so moving on let's talk in a little bit more detail about each of these libraries so scipy as the name suggests is a scientific library and it very specifically it has some special functions for integration and for ordinary differential equations so as you can see these are mathematical operations or mathematical functions so these are readily available in this library and it has linear algebra modules and it is built on top of numpy so we will see what is there in numpy so this is a again as the name suggests that num comes from numbers so it is a mathematical library and one of its key features is availability of an n-dimensional array object that is a very powerful object and we will see how to use this and then of course you can create other let's say objects and so on and it has tools for integrating with cc plus plus and also for tran code and then it of course also has linear algebra and fourier transformation and so on all these scientific capabilities okay what else pandas is another very powerful library primarily for data manipulation so if you're importing any files you will want to create it like a table so you will create what is known as data frames these are very powerful data structures that are used in python programming so pandas library provides uh this capability and once you import a data import the data into data frame you can pretty much do whatever you're doing like in a regular database so people who are coming from a database background or sql background would really like this because it is very they will feel very much at home because it feels like you're using you're viewing a table or using a table and you can do a lot of stuff using the pandas library now there are two important terms or components in pandas series and the data frame i was just talking about the data frame so let's take a look at what are series and what is a data frame so within pandas we have series and data frames so series is primarily some of you may also be knowing this as let's say an array so it's a one-dimensional structure data structure if you will so in some other languages we may call it as an array or maybe some others probably an equivalent of a list in r perhaps i'm not very sure on that aspect but yes so this is like a one-dimensional storage of information so that is what is series whereas data frame is like a table so you have a two-dimensional structure you have rows and you have columns and this is very people as i said who are familiar with sql and databases will be able to relate to this very quickly so you have like a table you have rows and columns and then you can manipulate the data so if you want to create a series this is how you would create a code snippet and as you can see the programming in python is very simple there are no major overheads you just need to import some libraries whichever essential and then start creating objects so you don't have to do additional declaration of variables and things like that so that is i think one key difference between python and other programming languages and what does this series contain it has to contain these numbers 6 3 4 6 and x is my object consisting of the series so if you display if you just say x it will display the contents of x and you will see here that it creates a default index then you have data frames so if you want to create a data frame as you can see the series is like a one-dimensional structure there is just like a row one row of items whereas a data frame looks somewhat like this it is a two-dimensional structure so you have columns in one dimension and then you have rows in the other dimension how do you create a data frame you need to create you need to rather import pandas and then you import in this case we are basically creating our own data so that's the reason we are importing numpy which is one of the libraries we just refer to a little bit before so we are using one of the functionalities within numpy to create some random numbers otherwise this is not really mandatory you probably will be importing the data from outside maybe some csv file and import into the data frame so that's what we are doing here so in this case we are creating our own test data that's the reason we are importing numpy as np and then i create a data frame saying pd dot data frame so this is the keyword here similarly here in this case while creating series we said pd.series and then you pass the values similarly here using pd.dataframe now in order to create the data frame it needs the values in each of these cells what are the values in the rows and what are the values in the column so that in our example we are providing using this random number generator so np dot random is like a class or a method that is available in numpy and then you are saying okay generate some random numbers in the form of a four by three matrix or four by three data frame the four here indicates the number of rows and the three here indicates the number of columns so these are the columns 0 1 2 are the columns and these are the rows here 0 this is 1 this is 2 this is 3 okay and once again it will when you display df it will give us a default index there are ways to omit that but at this point we will just keep it simple so it will display the default index and then the actual values in each of these rows and columns so this is the way you create a data frame so now that we have learned some of the basics of pandas let's take a quick look at how we use this in real life so let's assume we have a situation where we have some customer data and we want to kind of predict whether a customer's loan will be approved or not so we have some historical data about the loans and about the customers and using that we will try to come up with a way to maybe predict whether loan will be approved or not so let's see how we can do that so this is a part of exploratory analysis so we will first start with exploratory analysis we will try to see how the data is looking so what kind of data so we will of course i'll take you into the jupiter notebook and give you a quick live demo but before that let's quickly walk through some of the pieces of this program in slides and then i will take you actually into the actual code and do a demo of that so the python program structure looks somewhat like this the first step is to import your all the required libraries now of course it is not necessary that you have to import all your libraries right at the top of the code but it is a good practice so if you know you are going to need a certain set of libraries it may be a good idea to put from a readability perspective it's a good practice to put all the libraries that you're importing at the beginning of your code however it is not mandatory so in the middle of the code somewhere if you feel that you need a particular library you can import that library and then start using it in the middle of the code so that's also perfectly fine it will not give any errors or anything however as i said it's not such a good practice so we will import all the required libraries in this case we are importing pandas numpy and matplotlib and in addition if we include this piece of code percentage matplotlib inline what will happen is all the graphs that we are going to create the visualizations that we are going to create will be displayed within the notebook so if you want to have that kind of a provision you need to have this line so it's always a good idea when you're starting off i think it's a good idea to just include this line so that your graphs are shown in line okay so these are the four we will start with these four lines of code then the next step is to import your data so in our case there is a training data for loans by the name loan p underscore train dot csv and we are reading this data so in this case you see here unlike the previous example where we created a data frame with some data that we created ourselves here we are actually creating a data frame using some external data and it's the method is very very straightforward so you use the read underscore csv method and it is a very intuitive function name and you say pd dot read underscore csv and give the path of the file csv file that's about it and then that is read into the data frame df this can be any name we are calling it df you can call xyz anything there's a name just name of the object so head is one of the methods within the data frame and it will give us the first five so this is just to take a quick look now you have imported the data you want to initially have a quick look how your data is looking what are the values in some of the columns and so on and so forth right so typically you would do a head df dot head to get the sample of let's say the first few lines of your data so that's what has happened here so it displays the first few lines and then you can see what are the columns within that and what are the values in each of these cells and so on and so forth you can also typically you would like to see if there are any null values or are there any is the data for whatever reason is invalid or looking dirty for whatever reason some unnecessary character so this will give a quick view of that so in this case pretty much everything looks okay then the next step is to understand the data a little bit overall for each of the columns what is the information so the describe function will basically give us a summary of the data what else can we do pandas also allows us to visualize the data and this is more like a part of what we call it as univariate analysis that means each and every column you can take and do some plots and visualization to understand data in each of the columns so for example here the loan amount column we can take and then the hist basically hist method will create a histogram so you take all the values from one column which is loan amount and you create a histogram to see how the data is distributed right so that's what is happening here and as you can see there are some extreme values so this is again to identify do we have to do some data preparation because if the data is in a completely hazard way the analysis may be difficult so we these are the initial or exploratory data analysis is primarily done to understand that and see if we need to do some data preparation before we get into the other steps like machine learning and statistical modeling and so on so in this case we will see that here by plotting this histogram we see that there are some extreme values so there are some values a lot of it is around 100 range but there is also something of one or two observations in the 700 range so it's pretty scattered in that sense or they're not really scattered distributedly scattered but it is randomly scattered so the range is really huge so what can we do about this so there are some steps that we need to do normalization and so on so we'll see that in a bit so this is for one of the columns let's take another column which is applicant income similar kind of similar situation you have most of your observations in this range but there are also some which are far off from where most of the observations are so this is also pretty this also has some extreme values so we'll have to see what can be done credit history is the binary value so some people have a zero value and some will have credit history of one this is just like a flag so this basically is telling us how many people have a one and how many people have zero so it looks like majority of them have a value of 1 and a few about 100 of them have a value of 0. okay what else can we do so we now understood a little bit about the data so we need to do some data wrangling or data munching and see if we can some bring in some kind of normalization of all this data and we will kind of try to understand what is data wrangling and before we actually go into it okay so data wrangling is nothing but a process of cleaning the data if let's say there are there are multiple things that can happen in this particular example there were no missing values but typically when you get some data very often it will so happen that a lot of values are missing either there are null values or there are a lot of zeros now you cannot use such data as it is to perform some let's say predictive analysis or perform some machine learning activities and so on so that is one part of it so you need to clean the data the other is unifying the data now these ranges of this data are very huge some of them are going from some columns are going from 0 to 100 000 and some columns are just between 10 to 20 and so on these will affect the accuracy of the analysis so we need to do some kind of unifying the data and so on so that is what wrangling data wrangling is all about so before we actually perform any analysis we need to bring the data so to some kind of a shape so that we can perform additional analysis actual analysis on this and get some insights now how do we deal with missing values is a very common issue when we take data or when we get data from the business when a data scientist gets the data from the business so we should never assume that all our data will be clean and all the values filled up and so on because in real life very often there will be the data will be dirty so data wrangling is the process where you kind of clean up the data first of all identify whether the data is dirty and then clean up so how do we find some data is missing so there are a few ways you can write a small piece of code which will identify if for a given column or for given row any of the observations are null primarily so this line of code for example is doing that it is trying to identify how many null values or missing values are there for each of the columns so this is a lambda function and what we are saying is find out if a value is null and then you add all of them how many observations are there where this particular column is null so it does that for all the column so here you will see that for loan id obviously it's an id so there are no null values or missing values gender has about 13 observations where the values are missing similarly marital status as three and so on and so forth so we'll see here for example the loan amount has 21 observations where the values are missing loan amount term has 14 observations and so on so we'll see how to handle this missing values so there are multiple ways in which you can handle missing values if the number of observations are very small compared to the total number of observations then sometimes one of the easy ways is to completely remove that data so or delete that record exclude that record so that is one way of doing it so if there are let's say a million records and maybe 10 records are having missing values it may not be worth doing something to fill up those values it may be better off to get rid of those observations right so that is the missing values are proportionately very small but if there are relatively large number of missing values if you exclude those observations then your accuracy may not be that very good so the other way of doing it is we can take a mean value or for a particular column and fill up wherever there are missing values fill up those observations or cells with the mean value so that way what happens is you don't give some value which is too high or too low and it somehow fits within the range of the observations that we are seeing so this is one technique again there are it can be case to case and you may have to take a call based on your specific situation but these are some of the common methods if you see in the previous case loan amount had 21 and now we went ahead and filled all of those with the mean value so now there are zero with missing values okay so this is one part of a data wrangling activity what else you can do you can also check what are the types of the data so df dot d types will give us what are the various data types so all right so you can also perform some basic mathematical observations we have already seen that mean we found out so similarly if you do call the mean method for the data frame object it will actually perform or display or calculate the mean for pretty much all the numerical columns that are available in this right so for example here applicant income coeplicant income and all these are numerical values so it will display the main values of all of those now another thing that you can do is you can actually also combine data frames so let's say you import data from one csv file into one data frame and another csv file into another data frame and then you want to merge these because you want to do an analysis on the entire data okay one example could be that you have data in the form of csv files one csv file for each month of the year january february march each of these are in a different so you can import them into let's say 12 data frames and then you can merge them together as a single data frame and then you perform your analysis on the entire data frame or the entire data for the year so that is one example so how do we do that this is how we do again in this case we are not importing any data we are just creating some random values uh using some random values so let's assume i have a data frame which is by the name one and i assign some random values here which is a five by four format so there are five rows and four columns and this is how my data frame one looks and then i create another data frame which is data frame two again random numbers of the format five by four and i have something like this now i want to combine these two how do i combine these two i can use the concatenate or concat method and i can combine these two so pd dot concat and it takes the the data frames one and two if you have more of them you can provide them and it will just simply add all of them merge all of them or concatenate whatever you call whichever term you call it will so of course we have to make sure that the structure remains the same like i said this could be let's say sales data coming for 12 different months but each of the files has the same structure so now you can combine all of them merge all of them by using the concat method if we have let's say structure is not identical then what will happen let's say we have these two data frames one has a column by the name key and the second column is lval and a second data frame which has a column by the name key but the second column by the name are well not l val so you see here the structure is not identical so you can still combine them but then the way they get combined or merged is somewhat like this so it takes the key as a common parameter between them some common column has to be there otherwise this will not work and then we have to use merge instead of concatenate and when we do a merge then we get the result will be in this format what it does is it uses the key as the common thread between them and then it kind of populates the values accordingly so if you see here the first one had four and bar for key and then it had l values of 1 and 2 right so if we go back 4 and bar had 1 and 2 l values so that's what we see here 1 and 2. whereas in the right data frame we had foo bar and bar as a second time and then our values are three four and five so what it has done for foo it has put for the existing right for four is already existing because it has come from left so it will just put the value of r while here which is 3 similarly it will put 4 here because for bar if you go back for bar it is the value is 4 and since it has one more value of bar it will go and add this 5 as well the only thing here is that this one had for example left had only two values and only one value for bar but since we are appending or merging and there are two key values with the bar therefore it will kind of repeat the value of l val here so that's what we are seeing in this case right so l value appears twice the number 2 appears twice but that is because r value there are two of them okay all right so that is how when you don't have identical structure that's how you merge now we will talk a little bit about scikit-learn so scikit-learn is a library which is used for doing machine learning or work for performing machine learning activities so if you want to do linear regression logistic regression and so on there are easily usable apis that you can call and that's the advantage of scikit-learn and it provides a bunch of algorithms so i think that is the good part about this library so if you want to use scikit learn obviously you need to import these modules and also there are some sub modules you may have to import based on what you're trying to use like for example if we know if we want to use logistic regression again people who are probably not very familiar with machine learning there is a separate module for machine learning you may want to take a look at that but we will just touch upon the basics here so machine learning has some algorithms like linear regression logistic regression and random forest classification and so on so that is what we are talking about here so those algorithms are available and when if you want to use some of them you need to import them and from the scikit-learn library so scikit-learn is the top-level library which is basically sklearn right and then it has a kind of subparts in it you need to import those based on what exactly you will be or which algorithm you will be using so let's take an example as we move and we will see that whenever we perform some machine learning activity those of you who are familiar with machine learning will already know this we split our labeled data into two parts training and test now there are multiple ways of splitting this data how do we either some people do it like 50 50 some people do it 80 20 which is training is 80 and test it is 20 and so on so it is individual preference there are no hard and fast rules by and large we have seen that training data set is larger than the test data set and again we will probably not go into details of why do we do this at this point but that's one of the steps in machine learning so scikit learn offers a readily available method to do this which is strain test split all right so in this example let's say we are taking the values x and y are our values x is the independent variables and y is our dependent variable okay and we are using these two and then i want to split this into train and test data so what do we do we import the train test split sub module from within scikit learn which is sklearn right so within that we import drain test split and then you call the strain test split method or function or whatever you call it and pass the data so x is the all the values of the independent variables and y is our labels so you pass x and y and then you specify what should be your size of the test data so only one you need to specify so if you say test size is 0.25 it is understood that train size will be 0.75 so you're telling what should be the ratio of the split so technically it doesn't nothing prevents you from giving whatever you like here so you can give test as 80 and train as 20 so whichever way but then this normal practices you will have the training data set would be larger than the test data set and typically it would be 80 20 75 25 or 65 35 something like that right so that is the second parameter and this is just to say that you know the data has to be randomly split so it shouldn't be like you take the first 75 percent and put it in training and then the next 25 percent and put it in test so that so such a thing shouldn't happen so we first set the state random state so that the the splitting is done in a very random way so they randomly picked up the data and then put it into training and test and then this results in these four data frames so x train and x test and white rain and y test okay so that is basically the result it will now that the splitting is done let's see how to implement or execute logistic regression so in logistic regression what we try to do is try to develop a model which will classify the data logistic regression is an algorithm for supervised learning for performing classification so logistic regression is for classification and usually it is binary classification so binary classification means there are two classes so either like a yes no or for example customer will buy or will not buy so that is a binary classification so that's where we use logistic regression so let's take a look at the code how to implement something like that using psychic learn so the first thing is to import this logitech regression submodule or subclass whatever you call it and then create an instance of that so our object is classifier so we are creating an object by the name this is a name by the way you can give any name in our case we are saying classifier we say classifier is equal to logistic regression so we are creating an instance of the logistic regression variable or class or whatever okay and you can pass a variable or a parameter rather which is the random state is equal to 0 and once you create the object which in our case is named classifier you can then train the object by calling the method fit so this is important to note we don't call any there is no method like train here but we call what is known as there is a method called fit so you are basically by calling the fit method you are training this model and in order to train the model you need to pass the training data set so x underscore train is your independent variables the set of independent variables and y underscore train is your dependent variable or the label so you pass both of these and call the fit function or fit method which will actually result in the training of this model classifier now this is basically showing what are the possible parameters that can be passed or initiated when we are calling the logistic or the instance of logistic regression so this is but you can also look up the help file if you have installed python so some of these are very intuitive but some you may want to take a look at the details of what exactly they do all right so moving on once we train the model by calling fit then the next step is to test our model so this is where we will use the test data you need to pay attention here here i am calling so there are two things one is in order to test our data we have to actually call what is known as the method known as predict right so here this is where so the training is done now is the time for inference isn't it so we have the model now we want to check whether our model is working correctly or not so what do you do you have your test data remember we split it 25 percent of our data was stored here right we split it into test and training so that 25 of the data we pass to and call the method predict so that the model will now predict the values for y right so that's why here we are calling it as y underscore predict and if we display here as i said this is the logistic regression which is basically binary classification so it gives us the results like yes or no in this particular case and then you can so this is what the model has predicted our model has classified now but we also know we already have the labels for this so we need to compare with the existing labels with the known labels whether this classification is correct or not so that is where is the next step which is basically calculating the accuracy and so on will come into play okay so in this case the first thing most important thing to note is we do the prediction using predict and here we are passing x underscore test and not train right in this case we did x and y ten so again one more point to be noted here in case of training we will pass both the independent variables and also the dependent variables because the system has to internally it has to verify that is what is the training process so what it will do it will take the x values it will try to come up with the y value and compare with the actual y value right so that is what is the training method so that's why we have to pass both x as well as y whereas in case of predict we don't pass both we only pass because we are pretending as if this is the actual data so in actual data you will not have the labels isn't it so we are just passing the independent variables and the system will then come up with the y values which we will then okay remember we also know the actual value so we will compare this with the actual values and we will find out whether how accurate the model is so how do we do that we use what is known as a confusion matrix so this is also readily available in the python library so we import this confusion matrix and some of you who already know machine learning will find this familiar but those who are new to machine learning this confusion matrix is nothing but this matrix this kind of a matrix which basically tells how many of them are correctly predicted and how many of them are incorrectly predicted so the some of the characteristics let's quickly spend some time on this confusion matrix itself this the total numbers out here these are just the numbers these are like number of observations then the accuracy is considered to be highest when the the numbers or the sum of the numbers across the diagonals is maximum okay and the numbers outside of the diagonal should be minimum so which means that if this model was 100 percent accurate then the sum of these two there would have been only numbers in these two along the diagonal this would have been zero and this would have been zero okay so that is like a hundred percent accurate model that is very rare but just that you are aware so just to give an idea okay all right so once you have the confusion matrix you then try to calculate the accuracy which is in a percentage so there are two things that we can do from a confusion matrix or that we can calculate from a confusion matrix one is the accuracy and the other is the precision what is the accuracy accuracy is basically a measure of how many of the observations have been correctly predicted okay so let's say this is a little bit more detailed view of the confusion matrix it looks very similar like as we saw in this case right so this is a two by two matrix that's what we're seeing here 18 27 2 1 0 3 so 18 27 2 1 0 3. now but what are these values that is what is kind of the labels are shown here in this so there are altogether 150 observations so as i said the sum of all these 4 right 18 plus 27 plus 1 0 3 plus 2 is equal to 150 that's the first thing we have to observe the sum of all these values will be equal to the sum of test observations number of test observations we have 150 test observations because remember we had about 500 we split that into 2575 so that is why we have 150 here and i think 350 in the training data set okay so that we get the numbers correct so that's the first thing now this next thing is let's take a look at the actual values this view is the actual view so there are actually right in the actual data we have labels yes and no so as per the actual data there are 45 observations tagged as no and similarly there are 105 observations that are tagged as yes or labeled as yes okay now i know for the first time when you're seeing this it may be a little confusing but just stay with me okay so this is the actual part of it and this side tells us the predicted part of it so our model is predicted and it has totally predicted 20 of them as no right so that is what this is totally 20 of them it has predicted as no and it has predicted 130 of them as yes okay i hope this part is clear so before we go into the middle part let us first understand what exactly are these numbers so actually tagged as no there are 45 total actually tagged as yes there are 105 and predicted no there are 20 predicted as yes there are 130. this is the result from our model okay this is the result from our model and this is the actual value which we already know because this is our label data that's the first thing now now let us take a look at each of these individually okay now what are the options we have once again okay so now what is happening here let us look at these these values so this 18 says that these are actually tagged as no and the model is also predicted as no which means this is what is known as a true positive right or true negative sorry right which means that our model has predicted is it correctly it is negative because it says no so and it is also predicted no so it is known as what is known as true negative okay now let's come to this side of it that way we are talking about the diagonal remember i said most of the values should be in the diagonal okay so that means these 18 are correctly tagged they are labeled as no and our model is predicted as no so these are correctly tagged and these are known as true negative okay similarly if we come diagonally down there are 103 observations which are labeled as yes actual value is s and our model is also predicted as yes and these are known as true positive values positive because of this yes okay right so what is important is this is true this is also true so we have to make sure that the maximum number of values are in the true section okay true positive and true negative that's the reason i said the sum along the diagonal should be maximum now let's see if our model was 100 accurate this sum in this case it is only 103 plus 103 plus 18 which is 121 but if our model was accurate the sum of these two would have been 150 that means it's a perfect model okay all right now what else since we covered these two let's also cover these two so here this says that 27 of them were actually labeled no but our model is predicted as yes that means this is wrong right similarly these are two of them where the actual value is yes but our model is predicted as no that means it's a wrong prediction so you get the point so therefore along the diagonals are the correct values whereas in other places it is all wrong values or wrong predictions okay now how do we calculate accuracy from this information so the way to calculate accuracy is so we say okay there are total observations are 150 and what are the correctly predicted values these are the correctly predicted values which is 18 plus 103 so this will give us our accuracy so 103 plus 18 which is 121 divided by our total observations which is 150 is our accuracy which is 0.8 or we can say it is 80 percent okay now there is another concept called precision so precision is given by the formula true positives divided by the predicted positives totally predicted positives okay what do we mean by that which are the true positives here remember which are the true positives we just recall we just talked in the previous slide which are the true positives you see here so this 103 are the true positives which means that the value is positive actual value is positive predicted value is also positive so that's why it's called a true positive so 103 divided by so that is our true positive 103 divided by totally predicted as yes now what is totally predicted is yes remember 130 of them have altogether been predicted as yes not that they are correctly predicted only 103 have been correctly predicted but 130 of them have been predicted as yes so precision is basically the ratio of these two out of the totally predicted how many of them are actually true that ratio so 103 by 130 which is again about 80 percent is the precision that's how you calculate precision so this is just a simple formula and the term that you need to remember so accuracy is you need to take total of true positive and true negative divided by the total number of observations whereas precision is true positives divided by the totally predicted positives okay so that is our accuracy and precision now what we did the accuracy calculation was manual but we can also use some libraries which are already existing and the functions within that library so a scikit-learn provides uh one such method so for example accuracy underscore score is one such method so if you use that and pass your test and predicted values only the y you need to pass right the dependent variable values so if you pass that it will calculate it for you so in this case again as you can see it still calculates the same which is 80 which we have seen here as well okay so this can be done using the method great so that's pretty much what we have done here before we conclude let me take you into the code and show you how it actually looks okay so this is our code let me run it okay one by one we have already seen most of the steps in the slides so i will but i will run this in the actual jupyter notebook some of you if you are not yet familiar with jupiter notebook again there are other videos we created on how to install jupyter notebook and how to set up jupyter notebook and so on in this tutorial also we there was one slide on how to install python and jupyter notebook if you have not yet done please do that so that then you can actually walk through this code while you're watching this okay so what are we doing here we are importing the libraries required libraries recall here we have pandas we have numpy and for visualization we have matplotlib and this line is basically reading the csv file so we have the csv file locally on our local drive and this is where i'm checking the data just so i'm starting with my exploratory analysis how the data is looking so it looks good there are no major missing values or anything like that so it will display all the columns and it will show me the first five rows if when i'm using this head function and then i want to see a kind of a summary of all the each of the numerical columns so that's what i'm doing here so these are the numerical columns and it gives a summary like how many observations are there what is the mean standard deviation minimum maximum and so on and so forth for each of them and then you can do some visualization so this is the visualization for this okay the next step is to view the data data visualization and we will do that using a histogram for a couple of these columns so in this case i'm taking a look at the loan amount and if i create the histogram it displays the data here in the form of a histogram one thing that we gather from this as i mentioned in the slides as well is how the data is kind of scattered so while most of the values are in this range 0 to 300 range there are a few extreme values around the 700 range so that is one information we get from this histogram similarly for the applicant income if we draw a histogram something similar we can see that while most of the values are in this range 0 to 20 000 range there are a few in the range of 80 000 and probably 65 000 and so on okay so the next step is to perform data wrangling where we will check if any data is missing and how to fill those missing values and so on so in this case we will just check for all the columns how many data or how many entries are there with missing values so this is the results the loan id has all the columns or all the cells filled gender has 13 missing values marital status has three missing values and so on and so forth loan amount has 21 and this is what we are going to show you how to remove these missing values so when you have missing values as i mentioned in the during the slides there are a couple of ways of handling that one is you can completely remove those or you fill in with some meaningful value so in this case we will fill the missing values with the mean value of the loan amount so let's go ahead and do that and now if we check here now loan amount number of missing values is 0 because what we did was for all these 21 cells where the values were missing we filled with the mean value of the loan amount so now there are no more missing values for loan amount we can do this for other columns as well but this was just one example so we have shown it here okay so we will run this for credit history and loan amount term as well and then if we calculate the mean of pretty much all the numerical columns that's the method call so df.mean will give us the mean of all the numerical values and another thing that we can do is we want to find out what are the data types of each of these columns so you can call df.d types and get the data types of course it may not be that very useful most of the cases is an object but for example this one it shows as int 64 and there are float64 and so on and so forth now in addition to doing the exploratory data analysis we can do some machine learning activity as well so in this case we are going to do logistic regression so this is the example that i have shown you in the slides as well this is the actual code for that all right so the first step here is to import the libraries and then the next step is to separate the independent variables and the dependent variables so x is our independent variable and y is our dependent variable so we separate the data into two parts and this will be our target as well right so that's how we separate it now we have to split the data into training and test data sets as i mentioned in the during the slides we use the train test split method and when we call this and pass the independent variables and the dependent variables and we specify the test size to be 0.25 which means the training size will be 0.75 which is nothing but you split the data into training data set which is 75 percent and test data set in which is 25 okay so once you split that you will have all your independent variables data in x strain the training data which is 75 percent of it similarly independent variables for test will be in x underscore test and dependent variable strain will be in underscore train and dependent variable test will be y underscore test once we do this we have to do a small exercise for scaling remember we had some data which was kind of very scattered there were some extreme values and so on so this will take care of that so that the data is normalized so that before we pass to our algorithm the data is normalized so that the performance will be much better the next step is to create the instance of logistic regression object so that's what we are doing here so classifier is our logistic regression instance right classifier is equal to logistic regression we are saying so one instance of logistic regression is created and then we call the training method the name of the method actually is fit but what it is doing is it is taking the training data x is the training data the independent variables and y is the dependent variable so we are taking both of these and the model gets trained so the method for calling the training is fit okay so it gives us the output and then once we are done with the training we do the testing and once again just to recall in the slides when i was showing you the slides also i mentioned we don't pass y here while we are testing well for training we do pass y but right so for fit we are passing x and y but for test we are only passing x something you need to observe because y will be calculated by the model and we will then compare that with the known value of y to measure the accuracy so that's what we will do here and the method that is called here is predict so this will basically create or predict the values of y now we have in this case a binary classification so the outputs are yes or no y indicates yes and n indicates no so y or n is the output now how do we measure the accuracy as we have seen earlier i described how confusion matrix works and how we can use confusion matrix for calculating the accuracy that's what we are seeing here so this is the confusion matrix and then you want to do the measure the accuracy you can directly use this method and we find that it is 80 so we in the slides we have seen when we calculate manually as well we get an accuracy of 80 okay let's go back to our slides and do a summary so what we have done in the session we talked about what is data science and [Music] why python is being used why it is becoming so popular how to install python and we talked about the various libraries in python like panda scipy numpy and so on and then we took a couple of examples and wrote the code and demonstrated the code for performing exploratory analysis and performing data wrangling or data manipulation and then we at the end we did one example of machine learning using scikit-learn library and perform the logistic regression example so with that we come to the end of the session if you have any queries comments please put them below and you can also give your email id so that we can get back to you if you have any questions i hope you enjoyed the session thank you very much hi there if you like this video subscribe to the simply learn youtube channel and click here to watch similar videos turn it up and get certified click here