Data Science for Beginners: Full Course with Python Tutorial

[00:00:03]are you one of the many who dreams of becoming a data scientist keep watching this video if you're passionate about data science because we will tell you how does it really work under the hood emma is a data scientist let's see how a day in her life goes while she's working on data science project well it is very important to understand the business problem first in our meeting with the clients emma asks relevant questions understands and defines objectives for the problem that needs to be tackled she is a curious soul who asks a lot of eyes one of the many traits of a good data scientist now she gears up for data acquisition to gather and scrape data from multiple sources like web servers logs databases apis and online repositories oh it seems like finding the right data takes both time and effort after the data is gathered comes data preparation this step involves data cleaning and data transformation data cleaning is the most time consuming process as it involves handling many complex scenarios here emma deals with inconsistent data types misspelled attributes missing values duplicate values and what not then in data transformation she modifies the data based on defined mapping rules in a project etl tools like talent and informatica are used to perform complex transformations that helps the team to understand the data structure better then understanding what you actually can do with your data is very crucial for that emma does exploratory data analysis with the help of eda she defines and refines the selection of feature variables that will be used in the model development but what if emma skips this step she might end up choosing the wrong variables which will produce an inaccurate model thus exploratory data analysis becomes the most important step now she proceeds to the core activity of a data science project which is data modeling she repetitively applies diverse machine learning techniques like k n decision tree knife base to the data to identify the model that best fits the business requirement she trains the models on the training data set and test them to select the best performing model emma prefers python for modeling the data however it can also be done using r and sas well the trickiest part is not yet over visualization and communication emma meets the clients again to communicate the business findings in a simple and effective manner to convince the stakeholders she uses tools like tableau power bi and click view that can help her in creating powerful reports and dashboards and then finally she deploys and maintains the model she tests the selected model in a pre-production environment before deploying it in the production environment which is the best practice right after successfully deploying it she uses reports and dashboards to get real-time analytics further she also monitors and maintains the project's performance well that's how emma completes the data science project we have seen the daily routine of a data scientist is a whole lot of fun has a lot of interesting aspects and comes with its own share of challenges now let's see how data science is changing the world data science techniques along with genomic data provides a deeper understanding of genetic issues in reaction to particular drugs and diseases logistic companies like dhl fedex have discovered the best rules to ship the best suited time to deliver the best mode of transport to choose thus leading to cost efficiency with data science it is possible to not only predict employee attrition but to also understand the key variables that influence employee turnover also the airline companies can now easily predict flight delay and notify the passengers beforehand to enhance their travel experience well if you're wondering there are various roles offered to a data scientist like data analyst machine learning engineer deep learning engineer data engineer and of course data scientist the median base salaries of a data scientist can range from 95 000 to 165 000 so that was about the data science are you ready to be a data scientist if yes then start today the world of data needs you hello and welcome to the session on data science with python so what are we going to do in this session we will talk about what is data science and some of the basics of python as you may be aware data science we use either python or r as some of the tools and programming languages so this session the focus is on python so we will talk about the basics of python why to learn python how to install python and then we will talk about some of the important libraries which are required for data analysis and then we will go into a little bit of details about exploratory data analysis and we will take an example there of loan prediction and we will see a little bit about data wrangling using pandas which is one of the libraries of python and then we will end up with small predictive model or logistic regression model which is a part of machine learning and in case you need more details about machine learning in this session we will probably not go into the details of machine learning there is a separate session you might want to look at there is a separate video on machine learning you might want to take a look at that if you need more details here we are just giving a quick overview to see how we can use the scikit-learn library so we will not go into a lot of details about the basics of machine learning all right so with that let's move forward so what is data science data science is about finding insights from the data so if there is a lot of data if you have sufficient data how to analyze and find some insights out of it this is what is data science all about a couple of examples here customer prediction now let's say you have a customer base and you want to find out who are most likely to buy your product so you can use from your past behavior you can probably develop a model and try to predict who are the people out of the thousand leads or potential customers who will actually buy so there may be some patterns that you can use to predict similarly service planning so for example you're running a restaurant and you want to know how many people will be coming or how many customers will be visiting your restaurant on a given day now based on your historical data you can build a model to predict that as well so that there is no wastage of food and so on and so forth so these are very quick and easy examples of how data science can be used in business now let's uh talk about python for doing data science we need some kind of a programming language or a tool and so on so this session will be about python there are other tools like for example r and we will probably do a separate video on that but this session is on python and you must have already heard python is really becoming very popular everybody is talking about python not only data science in iot and ai and many other places so it's a very popular it's getting very popular so if you are not yet familiar with python this may be a good time to get started with it so why do we want to use python so basically python is used as a programming language because it is for data science because it has some rich tools from a mathematics and from a statistical perspective it has some rich tools so that is one of the reasons why we use python and if you see some of the trends if you're probably tracking some of the trends you will see that over the last few years python has become programming language of choice and especially for data science sas was earlier one of the most popular tools but now increasingly python is being used for doing data science and of course as well as r one of the reasons of course is that python and r are open source compared to sas which is a commercial product so that could definitely be one explanation but beyond that i think it is the ease of understanding this language the ease of using this language which is also making it very popular in addition to the availability of fantastic libraries for performing data science what are the other factors there are speed then there are availability of number of packages and then of course the design goal alright so what are each of these design goal primarily the syntax rules in python are relatively intuitive and easy to understand thereby it helps in building applications with the con size and readable code base so with the few lines of code you can really achieve a lot of stuff and then there are a lot of packages that are available that have been developed by other people which can be reused so we don't have to reinvent the wheel and last but not least the speed so python is relatively faster language of course it is not as fast as let's say crc plus plus but then relatively it is still faster so these are the three factors which make python the programming language of choice so if you want to get started with python the first thing obviously is to install python so there is some documentation there are some steps that you need to follow so we will try to briefly touch upon that otherwise of course there is a lot of material available on how to install python and so on you can always look around but this is one of the again there are different ways in which you can also install python so we will use the anaconda path there is a packaging tool called anaconda so we will use that path you can also directly install python but in our session we will use the anaconda route so the first thing you need to do is download anaconda and this is the path for that and once you click on this you will come to a page somewhat like this and download you can do the corresponding download based on whether you have a windows or ubuntu there is a also a download possible for our package available for ubuntu if you are doing something on ubuntu so based on which operating system in fact this page will automatically detect which operating system you are having and it will actually suggest so for example you see here if you're running mac os then it will automatically detect that you have max and the corresponding installers will be displayed here similarly if you're on some flavor of linux like ubuntu or any other then you will get the corresponding download links here and then beyond that you can also select which version of python you want to install of course the latest version is in the three point x range at the time of recording this 3.6 is one of the latest versions but some of you may want to do or start with the earlier version which is python 2.7 to point x and you can download that as well if you don't have anything installed then my suggestion is start with python 3.6 alright so once you do that you will be able to install python and you will be able to run jupyter notebook okay so now that you know how to install python and if you have installed python let's take a look at what are the various libraries that are available so python is a very easy language to learn and there are some basic stuff that you can do for example adding or printing hello world statement and so on without importing any specific libraries but if you want to perform data analysis you need to include or import some specific libraries so we are going to talk about those as we move forward so pandas for example is used for structured data operations so if you let's say are performing something on a csv file you import a csv file create a data frame and then you can do a lot of stuffs like data munching and data preparation before you do any other stuff like for example machine learning or so on so that's pandas scipy as the name suggests it is kind of it provides more scientific capabilities like for example it has linear algebra it has fourier transform and so on and so forth then you have numpy which is a very powerful library for performing n-dimensional or creating n-dimensional arrays and it also has some of the stuff that is there in sci-fi like for example linear algebra and fourier transform and so on and so forth then you have matplotlib which is primarily for visualization purpose it has again very powerful features for visualizing your data for doing the initial what is known as exploratory data analysis for doing univariate analysis bivariate analysis so this is extremely useful for visualizing the data and then psychic learn is used for performing all the machine learning activities if you want to do anything like linear regression classification or any of this stuff then the scikit-learn library will be extremely helpful in addition to that there are a few other libraries for example networks and igraph then of course a very important one is tensorflow so if you are interested in doing some deep learning or ai related stuff then it would be a good idea to learn about tensorflow and tensorflow is one of the libraries there is a separate video on tensorflow you can look for that and this is a one of the libraries created by google open source library so once you're familiar with machine learning data analysis machine learning then that may be the next step to go to deep learning and ai so that's where tensorflow will be used then you have beautiful soup which is primarily used for web scraping and then you take the data and then analyze and so on then os library is a very common library as the name suggests it is for operating system so if you want to do something on creating directories or folders and things like that that's when you would use os all right so moving on let's talk in a little bit more detail about each of these libraries so scipy as the name suggests is a scientific library and it very specifically it has some special functions for integration and for ordinary differential equations so as you can see these are mathematical operations or mathematical functions so these are readily available in this library and it has linear algebra modules and it is built on top of numpy so we will see what is there in numpy so this is a again as the name suggests that num comes from numbers so it is a mathematical library and one of its key features is availability of a n-dimensional array object that is a very powerful object and we will see how to use this and then of course you can create other let's say objects and so on and it has tools for integrating with cc plus plus and also for tran code and then it of course also has linear algebra and fourier transformation and so on all these scientific capabilities okay what else pandas is another very powerful library primarily for data manipulation so if you're importing any files you will want to create it like a table so you will create what is known as data frames these are very powerful data structures that are used in python programming so pandas library provides this capability and once you import a data import the data into data frame you can pretty much do whatever you're doing like in a regular database so people who are coming from a database background or sql background would really like this because it is very they will feel very much at home because it feels like you're using you're viewing a table or using a table and you can do a lot of stuff using the pandas library now there are two important terms or components in pandas series and the data frame i was just talking about the data frame so let's take a look at what are series and what is a data frame so within pandas we have series and data frames so series is primarily some of you may also be knowing this as let's say an array so it's a one-dimensional structure data structure if you will so in some other languages we may call it as an array or maybe some others probably an equivalent of a list in r perhaps i'm not very sure on that aspect but yes so this is like a one dimensional storage of information so that is what is series whereas data frame is like a table so you have a two-dimensional structure you have rows and you have columns and this is very people as i said who are familiar with sql and databases will be able to relate to this very quickly so you have like a table you have rows and columns and then you can manipulate the data so if you want to create a series this is how you would create a code snippet and as you can see the programming in python is very simple there are no major overheads you just need to import some libraries whichever essential and then start creating objects so you don't have to do additional declaration of variables and things like that so that is i think one key difference between python and other programming languages and what does this series contain it has to contain these numbers 6 3 4 6 and x is my object consisting of the series so if you display if you just say x it will display the contents of x and you will see here that it creates a default index then you have data frames so if you want to create a data frame as you can see the series is like a one-dimensional structure there is just like a row one row of items whereas a data frame looks somewhat like this it is a two-dimensional structure so you have columns in one dimension and then you have rows in the other dimension how do you create a data frame you need to create you need to rather import pandas and then you import in this case we are basically creating our own data so that's the reason we are importing numpy which is one of the libraries we just referred to a little bit before so we are using one of the functionalities within numpy to create some random numbers otherwise this is not really mandatory you probably will be importing the data from outside maybe some csv file and import into the data frame so that's what we are doing here so in this case we are creating our own test data that's the reason we are importing numpy as np and then i create a data frame saying pd dot data frame so this is the keyword here similarly here in this case while creating series we said pd dot series and then you pass the values similarly here using pd.dataframe now in order to create the data frame it needs the values in each of these cells what are the values in the rows and what are the values in the column so that in our example we are providing using this random number generator so np dot random is like a class or a method that is available in numpy and then you are saying okay generate some random numbers in the form of a four by three matrix or four by three data frame the four here indicates the number of rows and the 3 here indicates the number of columns so these are the columns 0 1 2 are the columns and these are the rows here 0 this is 1 this is 2 this is 3 okay and once again it will when you display df it will give us a default index there are ways to omit that but at this point we will just keep it simple so it will display the default index and then the actual values in each of these rows and columns so this is the way you create a data frame so now that we have learned some of the basics of pandas let's take a quick look at how we use this in real life so let's assume we have a situation where we have some customer data and we want to kind of predict whether the customer's loan will be approved or not so we have some historical data about the loans and about the customers and using that we will try to come up with a way to maybe predict whether loan will be approved or not so let's see how we can do that so this is a part of exploratory analysis so we will first start with exploratory analysis we will try to see how the data is looking so what kind of data so we will of course i'll take you into the jupiter notebook and give you a quick live demo but before that let's quickly walk through some of the pieces of this program in slides and then i will take you actually into the actual code and do a demo of that so the python program structure looks somewhat like this the first step is to import your all the required libraries now of course it is not necessary that you have to import all your libraries right at the top of the code but it is a good practice so if you know you are going to need a certain set of libraries it may be a good idea to put from a readability perspective it's a good practice to put all the libraries that you're importing at the beginning of your call however it is not mandatory so in the middle of the code somewhere if you feel that you need a particular library you can import that library and then start using it in the middle of the code so that's also perfectly fine it will not give any errors or anything however as i said it's not such a good practice so we will import all the required libraries in this case we are importing pandas numpy and matplotlib and in addition if we include this piece of code percentage matplotlib inline what will happen is all the graphs that we are going to create the visualizations that we are going to create will be displayed within the notebook so if you want to have that kind of a provision you need to have this line so it's always a good idea when you're starting off i think it's a good idea to just include this line so that your graphs are shown in line okay so these are the four we will start with these four lines of code then the next step is to import your data so in our case there is a training data for loans by the name loan p underscore train dot csv and we are reading this data so in this case you see here unlike the previous example where we created a data frame with some data that we created ourselves here we are actually creating a data frame using some external data and it's the method is very very straightforward so you use the read underscore csv method and it is a very intuitive function name and you say pd dot read underscore csv and give the path of the file csv file that's about it and then that is read into the data frame df this can be any name we are calling it df you can call xyz anything there's a name just name of the object so head is one of the methods within the data frame and it will give us the first five so this is just to take a quick look now you have imported the data you want to initially have a quick look how your data is looking what are the values in some of the columns and so on and so forth right so typically you would do a head df dot head to get the sample of let's say the first few lines of your data so that's what has happened here so it displays the first few lines and then you can see what are the columns within that and what are the values in each of these cells and so on and so forth you can also typically you would like to see if there are any null values or are there any is the data for whatever reason is invalid or looking dirty for whatever reason some unnecessary character so this will give a quick view of that so in this case pretty much everything looks okay then the next step is to understand the data a little bit overall for each of the columns what is the information so the describe function will basically give us a summary of the data what else can we do pandas also allows us to visualize the data and this is more like a part of what we call it as univariate analysis that means each and every column you can take and do some plots and visualization to understand data in each of the columns so for example here the loan amount column we can take and then the hist basically hist method will create a histogram so you take all the values from one column which is loan amount and you create a histogram to see how the data is distributed right so that's what is happening here and as you can see there are some extreme values so this is again to identify do we have to do some data preparation because if the data is in a completely hazard way the analysis may be difficult so we these are the initial or exploratory data analysis is primarily done to understand that and see if we need to do some data preparation before we get into the other steps like machine learning and statistical modeling and so on so in this case we will see that here by plotting this histogram we see that there are some extreme values so there are some values a lot of it is around 100 range but there is also something of one or two observations in the 700 range so it's pretty scattered in that sense or they're not really scattered distributedly scattered but it is randomly scattered so the range is really huge so what can we do about this so there are some steps that we need to do normalization and so on so we'll see that in a bit so this is for one of the columns let's take another column which is applicant income similar kind of similar situation you have most of your observations in this range but there are also some which are far off from where most of the observations are so this is also pretty this also has some extreme values so we'll have to see what can be done credit history is the binary value so some people have a zero value and some will have credit history of one this is just like a flag so this basically is telling us how many people have one and how many people have zero so it looks like majority of them have a value of one and a few about hundred of them have a value of zero okay what else can we do so we now understood a little bit about the data so we need to do some data wrangling or data munching and see if we can some bring in some kind of normalization of all this data and we will kind of try to understand what is data wrangling and before we actually go into it okay so data wrangling is nothing but a process of cleaning the data if let's say there are there are multiple things that can happen in this particular example there were no missing values but typically when you get some data very often it will so happen that a lot of values are missing either there are null values or there are a lot of zeros now you cannot use such data as it is to perform some let's say predictive analysis or perform some machine learning activities and so on so that is one part of it so you need to clean the data the other is unifying the data now these ranges of this data are very huge some of them are going from some columns are going from 0 to 100 000 and some columns are just between 10 to 20 and so on these will affect the accuracy of the analysis so we need to do some kind of unifying the data and so on so that is what wrangling data wrangling is all about so before we actually perform any analysis we need to bring the data so to some kind of a shape so that we can perform additional analysis actual analysis on this and get some insights now how do we deal with missing values is a very common issue when we take data or when we get data from the business when a data scientist gets the data from the business so we should never assume that all our data will be clean and all the values filled up and so on because in real life very often there will be the data will be dirty so data wrangling is the process where you kind of clean up the data first of all identify whether the data is dirty and then clean up so how do we find some data is missing so there are a few ways you can write a small piece of code which will identify if for a given column or for given row any of the observations are null primarily so this line of code for example is doing that it is trying to identify how many null values or missing values are there for each of the columns so this is a lambda function and what we are saying is find out if a value is null and then you add all of them how many observations are there where this particular column is null so it does that for all the columns so here you will see that for loan id obviously it's an id so there are no null values or missing values gender has about 13 observations where the values are missing similarly marital status has three and so on and so forth so we'll see here for example loan amount has 21 observations where the values are missing loan amount term has 14 observations and so on so we'll see how to handle this missing values so there are multiple ways in which you can handle missing values if the number of observations are very small compared to the total number of observations then sometimes one of the easy ways is to completely remove that data so or delete that record exclude that record so that is one way of doing it so if there are let's say a million records and maybe 10 records are having missing values it may not be worth doing something to fill up those values it may be better off to get rid of those observations right so that is the missing values are proportionately very small but if there are relatively large number of missing values if you exclude those observations then your accuracy may not be that very good so the other way of doing it is we can take a mean value or for a particular column and fill up wherever there are missing values fill up those observations or cells with the mean value so that way what happens is you don't give some value which is too high or too low and it somehow fits within the range of the observations that we are seeing so this is one technique again there are it can be case to case and you may have to take a call based on your specific situation but these are some of the common method if you see in the previous case loan amount had 21 and now we went ahead and filled all of those with the mean value so now there are zero with missing values okay so this is one part of a data wrangling activity what else you can do you can also check what are the types of the data so df.d types will give us what are the various data types so all right so you can also perform some basic mathematical observations we have already seen that mean we found out so similarly if you do call the mean method for the data frame object it will actually perform or display or calculate the mean for pretty much all the numerical columns that are available in this right so for example here applicant income co applicant income and all these are numerical values so it will display the main values of all of those now another thing that you can do is you can actually also combine data frames so let's say you import data from one csv file into one data frame and another csv file into another data frame and then you want to merge these because you want to do an analysis on the entire data okay one example could be that you have data in the form of csv files one csv file for each month of the year january february march each of these are in a different so you can import them into let's say 12 data frames and then you can merge them together as a single data frame and then you perform your analysis on the entire data frame or the entire data for the year so that is one example so how do we do that this is how we do again in this case we are not importing any data we are just creating some random values or using some random values so let's assume i have a data frame which is by the name one and i assign some random values here which is a five by four format so there are five rows and four columns and this is how my data frame one looks and then i create another data frame which is data frame two again random numbers of the format five by four and i have something like this now i want to combine these two how do i combine these two i can use the concatenate or concat method and i can combine these two so pd dot concat and it takes the the data frames one and two if you have more of them you can provide them and it will just simply add all of them merge all of them or concatenate whatever you call whichever term you call it will so of course we have to make sure that the structure remains the same like i said this could be let's say sales data coming for 12 different months but each of the files has the same structure so now you can combine all of them merge all of them by using the concat method if we have let's say structure is not identical then what will happen let's say we have these two data frames one has a column by the name key and the second column is lval and a second data frame which has a column by the name key but the second column by the name arval not l val so you see here the structure is not identical so you can still combine them but then the way they get combined or merged is somewhat like this so it takes the key as a common parameter between them some common column has to be there otherwise this will not work and then we have to use merge instead of concatenate and when we do a merge then we get the result will be in this format what it does is it uses the key as a common thread between them and then it kind of populates the values accordingly so if you see here the first one had fu and bar for key and then it had l values of one and two right so if we go back foo and bar had one and two l values so that's what we see here one and two whereas in the right data frame we had foo bar and bar as a second time and then our values are three 4 and 5. so what it has done for foo it has put for the existing right for 4 is already existing because it has come from left so it will just put the value of r while here which is 3.

[00:34:03]similarly it will put 4 here because for bar if you go back for bar it is the value is 4 and since it has one more value of bar it will go and add this 5 as well the only thing here is that this one had for example left had only two values and only one value for bar but since we are appending or merging and there are two key values with the bar therefore it will kind of repeat the value of l val here so that's what we are seeing in this case right so l value appears twice the number 2 appears twice but that is because r value there are two of them okay all right so that is how when you don't have identical structure that's how you merge now we will talk a little bit about scikit-learn so scikit-learn is a library which is used for doing machine learning over for performing machine learning activities so if you want to do linear regression logistic regression and so on there are easily usable apis that you can call and that's the advantage of scikit-learn and it provides a bunch of algorithms so i think that is the good part about this library so if you want to use scikit-learn obviously you need to import these modules and also there are some sub-modules you may have to import based on what you're trying to use like for example if we know if we want to use logistic regression again people who are probably not very familiar with machine learning there is a separate module for machine learning you may want to take a look at that but we will just touch upon the basics here so machine learning has some algorithms like linear regression logistic regression and random forest classification and so on so that is what we are talking about here so those algorithms are available and when if you want to use some of them you need to import them and from the scikit-learn library so scikit-learn is the top-level library which is basically sklearn right and then it has a kind of subparts in it you need to import those based on what exactly you will be or which algorithm you will be using so let's take an example as we move and we will see that whenever we perform some machine learning activity those of you who are familiar with machine learning will already know this we split our labeled data into two parts training and test now there are multiple ways of splitting this data how do we either some people do it like 50 50 some people do it 80 20 which is training is 80 and test it is 20 and so on so it is individual preference there are no hard and fast rules by and large we have seen that training data set is larger than the test data set and again we will probably not go into details of why do we do this at this point but that's one of the steps in machine learning so psychic scikit-learn offers a readily available method to do this which is drain test split all right so in this example let's say we are taking the values x and y are our values x is the independent variables and y is our dependent variable okay and we are using these two and then i want to split this into train and test data so what do we do we import the train test split sub module from within scikit learn which is sklearn right so within that we import drain test split and then you call the strain test split method or function or whatever you call it and pass the data so x is the all the values of the independent variables and y is our labels so you pass x and y and then you specify what should be your size of the test data so only one you need to specify so if you say test size is 0.25 it is understood that train size will be 0.75 so you're telling what should be the ratio of the split so technically it doesn't nothing prevents you from giving whatever you like here so you can give test as 80 and train as 20 so whichever way but then this normal practices you will have the training data set would be larger than the test data set and typically it would be 80 20 75 25 or 65 35 something like that right so that is the second parameter and this is just to say that you know the data has to be randomly split so it shouldn't be like you take the first 75 percent and put it in training and then the next 25 percent and put it in test so that so such a thing shouldn't happen so we first set the state random state so that the the splitting is done in a very random way so they randomly picked up the data and then put it into training and test and then this results in these four data frames so x train and x test and y train and y test okay so that is basically the result it will now that the splitting is done let's see how to implement or execute logistic regression so in logistic regression what we try to do is try to develop a model which will classify the data logistic regression is an algorithm for supervised learning for performing classification so logistic regression is for classification and usually it is binary classification so binary classification means there are two classes so either like a yes no or for example customer will buy or will not buy so that is a binary classification so that's where we use logistic regression so let's take a look at the code how to implement something like that using scikit learn so the first thing is to import this logitech regression sub module or subclass whatever you call it and then create an instance of that so our object is classifier so we are creating an object by the name this is a name by the way you can give any name in our case we are saying classifier we say classifier is equal to logistic regression so we are creating an instance of the logistic regression variable or class or whatever okay and you can pass a variable or a parameter rather which is the random state is equal to 0 and once you create the object which in our case is named classifier you can then train the object by calling the method fit so this is important to note we don't call any there is no method like train here but we call what is known as there is a method called fit so you are basically by calling the fit method you are training this model and in order to train the model you need to pass the training data set so x underscore train is your independent variables the set of independent variables and y underscore train is your dependent variable or the label so you pass both of these and call the fit function or fit method which will actually result in the training of this model classifier now this is basically showing what are the possible parameters that can be passed or initiated when we are calling the logistic or the instance of logistic regression so this is but you can also look up the help file if you have installed python so some of these are very intuitive but some you may want to take a look at the details of what exactly they do all right so moving on once we train the model by calling fit then the next step is to test our model so this is where we will use the test data you need to pay attention here here i am calling so there are two things one is in order to test our data we have to actually call what is known as the method known as predict right so here this is where so the training is done now is the time for inference isn't it so we have the model now we want to check whether our model is working correctly or not so what do you do you have your test data remember we split it 25 percent of our data was stored here right we split it into test and training so that 25 of the data we pass to and call the method predict so that the model will now predict the values for y right so that's why here we are calling it as y underscore predict and um if we display here as i said this is the logistic regression which is basically binary classification so it gives us the results like yes or no in this particular case and then you can so this is what the model has predicted our model has classified now but we also know we already have the labels for this so we need to compare with the existing labels with the known labels whether this classification is correct or not so that is where is the next step which is basically calculating the accuracy and so on will come into play okay so in this case the first thing most important thing to note is we do the prediction using predict and here we are passing x underscore test and not train right in this case we did x and y ten so again one more point to be noted here in case of training we will pass both the independent variables and also the dependent variables because the system has to internally it has to verify that is what is the training process so what it will do it will take the x values it will try to come up with the y value and compare with the actual y value right so that is what is the training method so that's why we have to pass both x as well as y whereas in case of predict we don't pass both we only pass because we are pretending as if this is the actual data so in actual data you will not have the labels isn't it so we are just passing the independent variables and the system will then come up with the y values which we will then okay remember we also know the actual value so we will compare this with the actual values and we will find out whether how accurate the model is so how do we do that we use what is known as a confusion matrix so this is also readily available in the python library so we import this confusion matrix and some of you who already know machine learning will find this familiar but those who are new to machine learning this confusion matrix is nothing but this matrix this kind of a matrix which basically tells how many of them are correctly predicted and how many of them are incorrectly predicted so the some of the characteristics let's quickly spend some time on this confusion matrix itself this the total numbers out here these are just the numbers these are like number of observations then the accuracy is considered to be highest when the the numbers or the sum of the numbers across the diagonals is maximum okay and the numbers outside of the diagonal should be minimum so which means that if this model was 100 accurate then the sum of these two there would have been only numbers in these two along the diagonal this would have been zero and this would have been zero okay so that is like a hundred percent accurate model that is very rare but just that you are aware so just to give an idea okay all right so once you have the confusion matrix you then try to calculate the accuracy which is in a percentage so there are two things that we can do from a confusion matrix or that we can calculate from a confusion matrix one is the accuracy and the other is the precision what is the accuracy accuracy is basically a measure of how many of the observations have been correctly predicted okay so let's say this is a little bit more detailed view of the confusion matrix it looks very similar like as we saw in this case right so this is a two by two matrix that's what we're seeing here 18 27 2 1 0 3 so 18 27 2 1 0 3. now but what are these values that is what is kind of the labels are shown here in this so there are altogether 150 observations so as i said the sum of all these four right 18 plus 27 plus 103 plus 2 is equal to 150 that's the first thing we have to observe the sum of all these values will be equal to the sum of test observations number of test observations we have 150 test observations because remember we had about 500 we split that into 2575 so that is why we have 150 here and i think 350 in the training data set okay so that we get the numbers correct so that's the first thing now this next thing is let's take a look at the actual values this view is the actual view so there are actually right in the actual data we have labels yes and no so as per the actual data there are 45 observations tagged as no and similarly there are 105 observations that are tagged as yes or labeled as yes okay now i know for the first time when you're seeing this it may be a little confusing but just stay with me okay so this is the actual part of it and this side tells us the predicted part of it so our model is predicted and it has totally predicted 20 of them as no right so that is what this is totally 20 of them it has predicted as no and it has predicted 130 of them as yes okay i hope this part is clear so before we go into the middle part let us first understand what exactly are these numbers so actually tagged as no there are 45 total actually tagged as yes there are 105 and predicted no there are 20 predicted as yes there are 130. this is the result from our model okay this is the result from our model and this is the actual value which we already know because this is our label data that's the first thing now now let us take a look at each of these individually okay now what are the options we have once again okay so now what is happening here let us look at these these values so this 18 says that these are actually tagged as no and the model is also predicted as no which means this is what is known as a true positive right or true negative sorry right which means that our model as predicted is it correctly it is negative because it says no so and it is also predicted no so it is known as what is known as true negative okay now let's come to this side of it that way we are talking about the diagonal remember i said most of the values should be in the diagonal okay so that means these 18 are correctly tagged they are labeled as no and our model is predicted as no so these are correctly tagged and these are known as true negative okay similarly if we come diagonally down there are 103 observations which are labeled as yes actual value is s and our model is also predicted as yes and these are known as true positive values positive because of this yes okay right so what is important is this is true this is also true so we have to make sure that the maximum number of values are in the true section okay true positive and true negative that's the reason i said the sum along the diagonal should be maximum now let's see if our model was 100 accurate this sum in this case it is only 103 plus 103 plus 18 which is 121 but if our model was accurate the sum of these two would have been 150 that means it's a perfect model okay all right now what else since we covered these two let's also cover these two so here this says that 27 of them were actually labeled no but our model is predicted as yes that means this is wrong right similarly these are two of them where the actual value is yes but our model is predicted as no that means it's a wrong prediction so you get the point so therefore along the diagonals are the correct values whereas in other places it is all wrong values or wrong predictions okay now how do we calculate accuracy from this information so the way to calculate accuracy is so we'd say okay there are total observations are 150 and what are the correctly predicted values these are the correctly predicted values which is 18 plus 103. so this will give us our accuracy so 103 plus 18 which is 121 divided by our total observations which is 150 is our accuracy which is 0.8 or we can say it is 80 percent okay now there is another concept called precision so appreciation is given by the formula true positives divided by the predicted positives totally predicted positives okay what do we mean by that which are the true positives here remember which are the true positives we just recall we just talked in the previous slide which are the true positives you see here so this 103 are the true positives which means that the value is positive actual value is positive predicted value is also positive so that's why it's called a true positive so 103 divided by so that is our true positive 103 divided by totally predicted as yes now what is totally predicted is yes remember 130 of them have altogether been predicted as yes not that they are correctly predicted only 103 have been correctly predicted but 130 of them have been predicted as yes so precision is basically the ratio of these two out of the totally predicted how many of them are actually true that ratio so 103 by 130 which is again about 80 percent is the precision that's how you calculate precision so this is just a simple formula and the term that you need to remember so accuracy is you need to take total of true positive and true negative divided by the total number of observations whereas precision is true positives divided by the totally predicted positives okay so that is our accuracy and precision now what we did the accuracy calculation was manual but we can also use some libraries which are already existing and the functions within that library so a scikit-learn provides one such method so for example accuracy underscore score is one such method so if you use that and pass your test and predicted values only the y you need to pass right the dependent variable values so if you pass that it will calculate it for you so in this case again as you can see it still calculates the same which is 80 which we have seen here as well okay so this can be done using the method great so that's pretty much what we have done here before we conclude let me take you into the code and show you how it actually looks okay so this is our code let me run it okay one by one we have already seen most of the steps in the slides so i will but i will run this in the actual jupyter notebook some of you if you are not yet familiar with jupiter notebook again there are other videos we created on how to install jupiter notebook and how to set up jupyter notebook and so on in this tutorial also we there was one slide on how to install python and jupyter notebook if you have not yet done please do that so that then you can actually walk through this code while you're watching this okay so what are we doing here we are importing the libraries required libraries recall here we have pandas we have numpy and for visualization we have matplotlib and this line is basically reading the csv file so we have the csv file locally on our local drive and this is where i'm checking the data just so that i'm starting with my exploratory analysis how the data is looking so it looks good there are no major missing values or anything like that so it will display all the columns and it will show me the first five rows if when i'm using this head function and then i want to see a kind of a summary of all the each of the numerical columns so that's what i'm doing here so these are the numerical columns and it gives a summary like how many observations are there what is the mean standard deviation minimum maximum and so on and so forth for each of them and then you can do some visualization so this is the visualization for this okay the next step is to view the data data visualization and we will do that using a histogram for a couple of these columns so in this case i'm taking a look at the loan amount and if i create a histogram it displays the data here in the form of a histogram one thing that we gather from this as i mentioned in the slides as well as how the data is kind of scattered so while most of the values are in this range 0 to 300 range there are a few extreme values around the 700 range so that is one information we get from this histogram similarly for the applicant income if we draw a histogram something similar we can see that while most of the values are in this range 0 to 20 000 range there are a few in the range of 80 000 and probably 65 000 and so on okay so the next step is to perform data wrangling where we will check if any data is missing and how to fill those missing values and so on so in this case we will just check for all the columns how many data or how many entries are there with missing values so this is the results loan id has all the columns or all the cells filled gender has 13 missing values marital status has three missing values and so on and so forth loan amount has 21 and this is what we are going to show you how to remove these missing values so when you have missing values as i mentioned in the during the slides there are a couple of ways of handling that one is you can completely remove those or you fill in with some meaningful values so in this case we will fill the missing values with the mean value of the loan amount so let's go ahead and do that and now if we check here now loan amount number of missing values is 0 because what we did was for all these 21 cells where the values were missing we filled with the mean value of the loan amount so now there are no more missing values for loan amount we can do this for other columns as well but this was just one example so we have shown it here okay so we will run this for credit history and loan amount term as well and then if we calculate the mean of pretty much all the numerical columns that's the method call so df.mean will give us the mean of all the numerical values and another thing that we can do is we want to find out what are the data types of each of these columns so you can call df.d types and get the data types of course it may not be that very useful most of the cases is an object but for example this one it shows as int64 and there are float64 and so on and so forth now in addition to doing the exploratory data analysis we can do some machine learning activity as well so in this case we are going to do logistic regression so this is the example that i have shown you in the slides as well this is the actual code for that all right so the first step here is to import the libraries and then the next step is to separate the independent variables and the dependent variables so x is our independent variable and y is our dependent variable so we separate the data into two parts and this will be our target as well right so that's how we separate it now we have to split the data into training and test data sets as i mentioned in the during the slides we use the train test split method and when we call this and pass the independent variables and the dependent variables and we specify the test size to be 0.25 which means the training size will be 0.75 which is nothing but you split the data into training data set which is 75 percent and test data set in which is 25 okay so once you split that you will have all your independent variables data in x strain the training data which is 75 percent of it similarly independent variables for test will be in x underscore test and dependent variable strain will be in underscore train and dependent variable test will be y underscore test once we do this we have to do a small exercise for scaling remember we had some data which was kind of very scattered there were some extreme values and so on so this will take care of that so that the data is normalized so that before we pass to our algorithm the data is normalized so that the performance will be much better the next step is to create the instance of logistic regression object so that's what we are doing here so classifier is our logistic regression instance right classifier is equal to logistic regression we are saying so one instance of logic regression is created and then we call the training method the name of the method actually is fit but what it is doing is it is taking the training data x is the training data or the independent variables and y is the dependent variable so we are taking both of these and the model gets trained so the method for calling the training is fit okay so it gives us the output and then once we are done with the training we do the testing and once again just to recall in the slides when i was showing you the slides also i mentioned we don't pass y here while we are testing while for training we do pass y but right so for fit we are passing x and y but for test we are only passing x something you need to observe because y will be calculated by the model and we will then compare that with the known value of y to measure the accuracy so that's what we will do here and the method that is called here is predict so this will basically create or predict the values of y now we have in this case a binary classification so the outputs are yes or no y indicates yes and n indicates no so y or n is the output now how do we measure the accuracy as we have seen earlier i described how confusion matrix works and how we can use confusion matrix for calculating the accuracy that's what we are seeing here so this is the confusion matrix and then you want to do the measure the accuracy you can directly use this method and we find that it is 80 so we in the slides we have seen when we calculate manually as well we get an accuracy of 80 okay let's go back to our slides and do a summary so what we have done in the session we talked about what is data science and why python is being used why it is becoming so popular how to install python and we talked about the various libraries in python like pandas scipy numpy and so on and then we took a couple of examples and wrote the code and demonstrated the code for performing exploratory analysis and performing data wrangling or data manipulation and then we at the end we did one example of machine learning using scikit-learn library and perform the logistic regression example welcome to data science interview questions my name is richard kirschner with the simply learn team that's www.simplylearn.com get certified get ahead before we dive in and start going through the questions one at a time we're going to start with some of the logical kind of concept that's enters in a lot of interviews and this one you have two buckets one of three leaders and the other five leaders you're expected to measure exactly four leaders how will you complete the task and no you only have the two buckets you don't have a third bucket or anything like that just the two buckets and the object of the question like this is to see how well you are thinking outside the box in this case you're in a larger box you have two buckets and also the pattern which you go on and what that means is if you look at the two buckets and we'll show you their answer in just a second you have a bucket with three liters and a bucket with five liters and the first thought is what happens if you go from left to right so we have a direction and what happens if you pour the three liters into the five liter bucket well if you pour the three liters into five liter bucket you have an empty bucket of three liters and what's really important here is i was thinking outside the box you realize that you have a five liter bucket that has three liters in it and two empty leaders so you have two additional leaders you can fill up if we continue that process we can pour from the left to right from the small bucket to the large bucket you can now measure in two additional liters into the five liter bucket and three minus two is one and you can keep doing that you can empty the five liter bucket in pour those three liters in that one liter in and then you can pour three liters in what's cool about these questions as you explore them is you realize there's multiple ways usually to solve i went from small bucket to big bucket the simply learned team their solution that they pulled out was you fill the five liter bucket and empty it into the three liter bucket now you're left with two liters in the five liter bucket so that's great we can empty the three liter bucket so now we're going from large to small remember we went from small to large so you can go both either way but you have to go one way or the other it turns out and you can empty the three liter bucket and pour the contents of the five liter bucket in it so the three liter bucket now has two liters and if it has two liters that means it has an empty one liter and by now you probably have guessed that if you have an empty space you can start using that empty space of one liter as a measuring so we fill the five liter bucket again and we pour the water in the three liter bucket it already has the two liters and so we're only pouring one liter in there and five minus one is four so interview questions they break up into all kinds of different patterns we have logic like this one which is a lot of fun we have questions that come up that are more vocabulary list the difference between supervised and unsupervised learning probably one of the fundamental breakdowns in data science and supervised learning uses known and labeled data as input supervised learning has a feedback mechanism most commonly used supervised learning algorithms are decision tree logistic regression support vector machine and you should know that those are probably the most common use right now and there certainly are so many coming out so that's a very evolving thing and be aware of a lot of the different algorithms that are out there outside of the deep learning because a lot of these work faster on raw data numbers than they do than a deep neural network would unsupervised learning uses unlabeled data as input unsupervised learning has no feedback mechanism most commonly used unsupervised learning algorithms are k-means clustering hierarchical clustering the appropriate algorithm and there certainly are more i'm going to say k-means definitely is at the top of the list and the hierarchical clustering those two are used so many times so really important to understand what those are and how they're used and most important is understand that supervised learning is you have your data set where you have training data and you have all those different pieces moving around but you you're able to train it you know the answers and unsupervised we're just grouping things together that look like they go together how is logistic regression done logistic regression measures the relationship between the dependent variable our label what we want to predict and the one or more independent variables are features by estimating probability using its underlying logistic function sigmoid and whenever i draw these charts i always end up drawing them the right hand side first because you want to know what your output is what is you want out of here and the left hand side what do you have going in so you have your in and out you can see we have a nice labeled image here to help you remember this we have our inputs we have our linear model we have our probabilities what are the probabilities of it being a certain way based on these features coming in the sigmoid function and it's important to note that the sigmoid function is maybe the most commonly used but it's only one of a number of functions that are out there and the sigmoid function turns our probabilities into a value between 0 and 1 or very close to 0 very close to 1 between 0.1 and 0.009 and based on that we generate an answer in this case is zero or one how is logistic regression done so last time we talked about the sigmoid function generally depending on what your interview and level of math and what expertise you're going in for the market you'll have to understand that formula of the probability equals 1 over 1 plus e to the negative y and that's e to the base 2. so you have your probability function or your sigmoid function which pushes it as you can see we have a nice visual of that that helps a lot to have that visual on the sigmoid function you definitely should know your y equals m times x plus c your base euclidean geometry of forming a line in the slope plus the intercept the y intercept and then you have your natural log and the natural log is to the e as opposed to a base two or base ten so your natural log to the e of the probability over one minus a probability equals your m times x plus c or your euclidean line that helps a lot as far as the graphing and understanding the sigmoid function so we'll just keep pushing on to question number three explain the steps in making a decision tree and i noticed last time we brought up the decision tree in the forest a lot of questions came up what is the difference so let's go through that when you make a decision tree you're going to take the entire data set as input you're going to calculate entropy of the target variable as well as the predictor attributes i remember entropy is just how chaotic is it so if you have like you know banana and grapes and oranges if you're mixing in fruit and that's your data coming in you have all these different objects that are so separate from each other and the more they become uniform the lower the entropy and we call that information gain so we gain information on sorting different objects from each other so you have your entropy you have to calculate your information gain of all attributes and then you choose the attribute with the highest information gain as the root node so if you can separate your group and each group chaos in each group is lowered whichever split lowers the chaos the most that's where you split it and that's your root node at that point you repeat the same procedure on every branch till the decision node of each branch finalized so understanding that setup is pretty important as far as decision trees and you can see here we have a nice visual of a decision tree for example if you want to build a decision tree to decide whether we should accept or decline a job offer since these are interview questions that's a good one to ask and just as a tip you should be pretty aware of the formula for entropy and information gain so you need to look those up if you don't remember those and the salary if it's greater than 50 000 no decline the offer yes it's got a good salary the commute is greater than an hour yes decline the offer no offers incentives yes except the offer no incentives decline the offer so we use decision tree pretty much for everything if you want and if you have a decision tree then you also should understand how do you build a random forest model and remember that a random forest is built up of a number of decision trees so if you split your date up into a lot of different packages and you do a decision tree on each of those different groups of data the random forest is bringing all those trees together so how do you build a random forest model randomly select k features from a total of m features where k is less than m among the k features calculate the node d using the best split point split the node into daughter nodes using the best split repeat steps 2 and 3 steps until leaf nodes are finalized build for us by repeating steps one to four for end number times to create in number of trees so you can see it's got the same build pattern as the tree but instead you're building a number of different trees little small trees so it all have an n leaf node random forest has a vote at the end and whoever gets the most votes wins that's the answer how can you avoid overfitting of your model very important question in any kind of mathematical scientific data science setup in any of them there are three main methods to avoid overfitting and you should really understand overfitting overfitting means that your model is only set for a very small amount of data and ignores the bigger picture keep the model simple take into account fewer variables thereby removing some of the noise in the training data good advice for any programming at all use cross-validation techniques such as k-folds cross-validation use regularization techniques such as lasso that penalize certain model parameters if they're likely to cause overfitting and you should also be well aware that your cross validation techniques that's like a pre data or your lasso and your regularization techniques are usually during the process so when you're prepping your data that's when you're gonna do a cross validation such as like splitting your data into three groups and you train it on two groups and test it on one and then switch which two groups you tested on that kind of thing so can you solve another one of these i love these things there are nine balls out of which one ball is heavy in weight and the rest are of the same weight and how many minimum weighings will you find the heavier ball and when we say weighing think of a scale where you can put objects on one side and the other and you can see which side is heavier and you want to minimize that you want to split the balls up in such a way that you're going to do as few measurements as you can you will need to perform two wangs so you can get it down to just two wings and i always think if there's nine balls and divide them into three groups of three place three balls on each side so you can just randomly pick six of the balls and three on one side three on the other and if they balance out both sides are equal then you know the heavy weight isn't in any of those so out of the remaining three balls from step one take two balls and place one ball on each side a little tricky there because i always want to put all three i want to put two on one side and one on the other but no just take randomly pick two of those put one on each side if they balance out then the left out ball the one you didn't measure will be the heavier one otherwise you'll see it in the balance you'll see which one's heavier because it'll take one of the balls down now we go to scenario b where they did not balance out so now we know which side has a heavier ball in it and it's just very similar to what we did before if the balls in step one do not balance out then take those three balls that have the heavier side on them and reproduce step two to find out the heavier ball difference between univariate bivariate and multivariate analysis and hopefully if you know a little latin you'll kick in there that you have uni and you have bi and you have multi because the answer is in the words themselves so the first one this type of data contains only one variable so let's see univariate purpose of the univariate analysis is to describe the data and find patterns that exist within it so when you only see one one variable coming in in this case we're using height of students you're limited as far as what you can do with that data so you can come up and draw different patterns and conclusions from those patterns using the means the median the mode dispersion range minimum maximum so we're describing the data so all those words would describe the data and that's about all you can do with them data like that there's no correlation there's nothing to go beyond that as far as guessing or predicting anything so we move into bivariate you know uni means one by means two bivariate this type of data involves two different variables the analysis of this type of data deals with causes and relationships and the analysis is done to find out the relationship among the two variables and this is always a favorite one because everybody loves ice cream in the summer when it's hot and very few people go for ice cream in the winter when it's really cold so it's easy to see the correlation in the data the temperature and ice cream cells in summer season and you can see here where the temperature goes from 20 to 35 and as the temperature goes up so does the sales of ice cream it goes from 2 000 i'm not sure 2 000 what's i'm guessing it's a very large chain because if they're selling 2 000 ice cream cones and they have a lot of business good for them a little vendor on the corner selling 2 000 ice cream cones a day and 3 100 the next day here the relationship is visible from the table that temperature and cells are directly proportional to each other so the hotter the temperature we can predict an increase in cells so the word prediction should come up so we have description and prediction when the data involves three or more variables it is categorized under multivariate it is similar to bivariate but contains more than one dependent variable in this example another really common one the data for house price prediction the patterns can be studied by drawing conclusions using mean median and mode dispersion or range minimum maximum etc and so you can start describing the data that's what all that was and then using that description to guess what the price is going to be so this is very good if you're in the market and you have already looked at the area and you already know that a two bedroom zero floor 900 square foot house is usually runs about 40 000 you can guess what the next one that looks similar to it is and i'll just throw in another word in there i don't see very often unless you're really a hardcore data science we talked about describing the data descriptive we talked about predictive and there's also post postscriptive postscriptive means we're going to change the variables to try to guess what the outcome is if we change what's going on so that would be the next step but that usually doesn't show up unless you're dealing with some really hardcore data science groups what are the feature selection methods to select the right variables there are two main methods for feature selection there's filter methods and wrapper methods and when you're filtering your before we discuss the two methods real quick the best analogy for selecting features is bad data in bad answer out so when we're limiting or selecting our features it's all about cleaning up the data coming in so it's cleaner and is more representative of what we're trying to predict filter method filter methods as they come in we have linear discrimination analysis anova chi squared chi square is probably the most common one and these are all part of pre-processing we're taking out all the outliers all the things that have a difference that is very different from the data we're looking at the odd ones and sometimes you take the odd ones out and then you analyze them separately to see why they're odd but remember your filter methods you want to pull all that weird stuff out wrapper methods on the other hand our forward selection backward selection recursive feature elimination and one of the most important things remember about wrapper methods is they're very labor-intensive you have to have some pretty high-end computers if you're doing a lot of data analysis with the wrapper method and just quickly forward selection means you have all your different features they're off to the side and we test just one feature at a time we keep adding them in until we get a good fit backwards we have all the features and we start we run a test on that to see how well it does and then we start removing features to see what works and recursive which is the most processing hungry algorithm out there goes through and just recursively looks through all the different features and how they pair together but again we have filter method and wrapper method and it's important to understand that we're sorting the data out and finding out which features are going to represent the data the best and which ones are not going to really add any value to our models let's jump number eight in your choice of language write a program that prints the numbers from 1 to 50. but for multiples of 3 print fizz instead of the number and for the multiples of 5 print abuzz for numbers which are multiples of both 3 and 5 print fizz buzz and this really is testing your knowledge and iterating over data very important my sister who runs at the university the data science team is in charge of their department the first question she asks in her interview of anybody who comes in is how do they iterate through data so if this question comes up a lot and it's very important you have an understanding and there's actually a slight error on this code which i'll point out in just a second the concept is we have fizz buzz in range you have range 51 which in this case goes from 0 to 51 and i'm gonna challenge you to see if you can catch the error and i'll tell you at the end of the code where the error is what that means is that we're going to go through all the numbers 0 1 2 3 4 and we're going to process through this loop if the remainder of fizzbuzz divided by three equals zero and fizzbuzz divided by five also equals zero then print fizzbuzz continue and else if fizzbuzz divided by three equals zero then print fizz print fizz continue else if is buzz divided by five equals zero print buzz continue and print fizzbuzz you fit the print the answer in this case fizzbuzz is either gonna be the number we generated which is zero or it'll be the fizz buzz fizz or buzz that's a mouthful now if you didn't catch the error in the code which is always a fun game find the error it has to do with the range and it's important to remember the range here says range to 51 that's 0 to 51 which is is correct we want to go to 51 because it stops it gets to 50 and it stops so that's 0 to 50. but if you remember the question asked from 1 to 50. so the range should be 1 comma 51 not just 51 which does 0 to 51. in this particular script in python you could leave out the continue but the continue in this script skips the next lsf so it doesn't keep processing it going down and in programming a lot of scripts you don't need the continuance this would depend on what script you chose and there's probably some other ways to do this it's a lot of fun and you can see here from the output we end up with fizz buzz for zero which shouldn't be there one two fizz four buzz fizz seven eight fizz buzz eleven fizz and so on sounds like a drinking game from my college days so long ago many decades ago you are given a data set consisting of variables having more than 30 percent missing values how will you deal with them oh the joy of messy data coming in ways to handle missing data values data set is huge we can just simply remove the rows with missing data values there's the quickest way i.e we use the rest of the data to predict the values you just go in there and say any row of our data that has a n a in it get rid of it that doesn't work with smaller data so a smaller data you start running into problems because you lose a lot of data and so we can substitute missing values with the mean or average of the rest of the data using pandas data frame in python there's different ways to do this obviously in different languages and even in python there's different ways to do this but in python it's real easy you can do the df.mean so you get the mean value so if you set mean equal to that then you can do a df.fill in a with the mean value very easy to do in a python panda script and if you're using python you should really know pandas and numpy number python and pandas data frames for the given points how will you calculate the euclidean distance in python so back to our basic algebra from high school euclidean distance is the line on the triangle and so if we're given the points plot one equals one comma three plot two equals two comma five we know that from this we can take the difference of each one of those points square them and then take the square root of everything so the euclidean distance equals the square root of plot 1 0 minus plot 2 is 0 squared plus plus 1 of 1 minus plot 2 of 1 squared mouthful there and you can remember if you have multiple dimensions that go past two dimensions you could have plot three can simply be the distance from plot one you only need to do one side of that or plot two you can do either way and square that and take the square root of that another mind bender how to calculate some how to figure out the solution to something what is the angle between the hour and minute hands of a clock when the time is halfed past six so you want to kind of imagine that clock where the large hand is pointed down to the 30 and the other half is gonna be right between the six and the seven because it's half past six there's actually a couple ways to solve this but let's take a look and see how they did it note a clock is a complete circle having 360 degrees in one hour the hour hand covers 360 over 12 so it equals 30 degrees for each hour in one minute the minute hand covers 360 degrees over 60 minutes or 6 degrees per minute the minute hand has traveled for 30 minutes so it has covered 30 times 6 which equals 180 degrees so we know that's 180 degrees from the 12. the hour hand has traveled for 6.5 hours 6 and a half 6.5 so it's covered 6.5 times 30 which equals 195 degrees the difference between the two will give the angle between the two hands thus the required angle equals 195 minus 180 equals 15 degrees and this is nice the way they solved it because you can now punch in any kind of time within reason the hard part is on the hours is you have to be able to convert the hours into decimals explain dimensionality reduction and list its benefits dimension reduction refers to the process of converting a set of data having vast dimensions into data with lesser dimensions fills to convey similar information concisely it helps in data compressing and reducing the storage space it reduces computation time as less dimensions lead to less computing it removes redundant features for example there's no point in storing a value in two different unit units meters and inches and i certainly ran into a lot with this with text analysis i've been known to run a text analysis over a series of documents ends up with over 1.4 million different features that's a lot of different words being used and if you do what they call buy connect them you connect two words together now you're up to 4.8 million different features and you start having to figure ways to bring that down what can we get rid of that kind of thing so you can see where that can get really high in on processing and learning how to reduce the list dimensions is very important how will you calculate eigenvalues and eigenvectors of a three by three matrix and what they're really looking for here is when you write it out for the eigen is that you know that you're going to use the lambda that's the most common one obviously you can use any symbol you want but lambda is usually what they use and that you do it down the middle diagonal and so when you take that matrix and you take the characteristic equation you end up with the determinant and that's the minus two minus lambda minus four two minus two one minus lambda two four two five minus lambda and that's what they're looking for and you know that's equal to zero so when you're doing a matrix in the eigen setup with the eigenvectors that's all going to come out equal to zero and then you can go ahead and write the whole equation out so we can expand the indeterminant as you can see right here the minus 2 minus lambda times it's a mouthful i'll leave it up here for a second so you can look at it when you break it down into the algebraic functions you end up with minus lambda cubed plus 4 lambda squared plus 27 lambda minus 90 equals zero so now we have a nice algebraic equation built from the eigenvectors and always remember you can hit the pause button and you can also send a note send a note to simplylearn if you have more questions on vectors or on this definitely have that resource available to you or post down in below on the youtube video comments and so when we calculate the eigenvalues and eigenvectors of a 3x3 matrix as we continue on down the math of this and to be honest i really don't like working with matrixes like this it's important to understand the math behind it and it's important to know the code just enough so that you're not lost when someone's explaining it or it comes up when i'm working on different data science models of course if you're dealing with the high end math side of it then you better know this first is by hitting trial so you try in different variables to solve for zero and you can come in here and you'll find that if we put in the three in there we end up with a zero at the end and substitute the three hence we end up with lambda minus three is one of the factors and you can do the math going out on that where we have lambda cubed minus four lambda squared minus 27 lambda plus 90 equals lambda minus 3 times lambda squared minus lambda minus 30. so eigenvalues based on that one are three minus five and six and then from there we can calculate the eigenvector for lambda equals three and you can see here where the matrix as we write it out is the minus five minus four two minus two minus two minus two four two two that's from the beginning put in the x y and z equals zero zero zero and so when we put in those numbers and we calculate them out we have for x equals one we have the minus five minus four y plus two z equals zero minus two minus two y plus two z equals zero and subtracting the two equations we just had we get three plus two y equals zero y equals minus three over two and z equals a minus one over two that's going back to the first equation and similarly we can calculate the eigenvectors for minus five and six how should you maintain your deployed model distribution time my favorite i spent ten years in software distribution so first thing and this is true not just of your data science model but of any computer code going out there this basic setup can work although usually there's a little added steps in there first we're going to monitor it so we have a constant monitoring of all the model is needed to determine the performance accuracy of the models so yeah we want to just keep an eye on it we want to make sure they're accurate we want to make sure that whatever they're supposed to predict or i threw in that bonus word post script where you change something and you want to figure out how your changes are going to affect things we need to monitor it and make sure it's doing what it's supposed to do evaluation metrics of the current model is calculated to determine if new algorithm is needed and then we compare it the new models are compared against each other to determine which model performs the best and then we do a rebuild the best performing model is rebuilt on the current state of data this is interesting i found this out just recently if you're in weather prediction the really big weather areas have about seven or eight different models depending on what's going on and so you actually have almost a little force going on there where they're like which model is going to fit best and this is what we're going to use to predict the weather with so not only do you don't necessarily get rid of the models but you figure out which models fit data of what's going on or the current state of data what are recommender systems most commonly used nowadays in marketing so very big industry understanding recommender systems predicts the rating or preference a user would give to a product and they they're split into two different areas one is collaborative filtering and a good example of that is the last.fm recommends tracks that are often played by other users with similar interests so people who if you're on amazon people who bought this also bought that this got me a few times and then there's contact based filtering and we're looking at content instead of looking at who else is listening to the music leave the example pandora which uses the properties of a song to recommend music with similar properties so you have collaborative filtering and content based filtering how to find rmse and msc in linear regression model hopefully you remember what the two acronyms mean because that is like half the answer we have the root mean square error and the mean square error in linear regression model so we're looking for error the rmse and the msc are the two of the most common measures of accuracy for a linear regression model and you can see here we have the root mean square error rmse equals and this is the square root of the sum of the predicted minus the actual squared over the total number so we're just looking for the average mean so we're looking for the average over the end and the reason you need to know about the difference between rmse versus mse is when you're doing a lot of these models and you're building your own model why do you need to take the square root of it it doesn't change the value as far as the way you're using it because you're looking as to see whether the error is greater or less than so why add that extra computation in so a lot of models use the msc which indicates the mean square error or the average error and it's the same formula minus the square root at the end or across the whole thing oh another riddle to solve if it rains on saturday with a probability of 0.6 and it rains on sunday with a probability 0.2 what is the probability that it rains this weekend and the trick in probabilities on this case is we're not we need to know what is the probability of it not raining what is it not what's the chance of it not raining on saturday and if it doesn't rain on saturday we want to take that and combine that with the chance of it not raining on sunday the total probability which in this case we're just going to use one minus the probability that it will not rain on saturday so it's 1 minus 0.6 we're going to take that as a union which we simply just multiply them together of the probability they will not rain on sunday and it's important to recognize the union here or the and you can see by the formula down here we end up with 0.68 or 68 chance that it will rain on the weekend and there are a couple other ways to solve this but this is probably the most traditional way of doing that how can you select k for k means so first you better understand that what k means is and that k is the number of different groupings and most commonly we use is the elbow method to select k for k means the idea of the elbow method is to run k-means clustering on the data set where k is the number of clusters within the sum of squares wss is defined as the sum of the squared distance between each member of the cluster and its centroid and you should know all the terms for your k means on there and with the elbow point and again here's our iteration in our code we talked about that earlier you iterate starting with um usually you don't start right at one but you might start with two three or four and you just see where it comes out and you can see the nice elbow there which is easy to see graphically where the number of k clusters and the wss value drops and then it just kind of flattens out and there's no reason to take the k means any further what is the significance of p value oh good one especially if you're dealing with r because that's the first thing that pops up p-value typically less than or equal to 0.05 indicates a strong evidence against the null hypothesis and you should know what a difference why we use null hypothesis instead of the hypothesis so you reject the null hypothesis very important that term null hypothesis in any scientific setup and also in data science it doesn't mean that it's true it means that there's a high correlation that it's true so if your null hypothesis means it's not true your hypothesis is has a high correlation that it's probably true and if the p-value is typically greater than.05 it indicates a weak evidence against the null hypothesis so you fail to reject the whole null hypothesis and if you reject that then your actual hypothesis is probably not true the correlation of your data with what you think it's saying is is probably incorrect and if you're right at the cutoff of 0.05 it's considered to be marginal could go either way and again you can use that p value on different features to decide whether you're going to include your features as far as something worth exploring in your data science model how can outlier values be treated oh good one you can drop outliers only if it is a garbage value so sometimes you end up with like one outlier that just is probably someone's measurements way off height of an adult equals abc feet this cannot be true as height cannot be a string value in this case outliers can be removed if the outliers have extreme values they can be removed for example if all the data points are clustered between 0 to 10 but one point lies at 100 then we can remove this point and again sometimes you just look for the outliers so you can see what's going on if there's something unusual there so maybe the equipment's not calibrated correctly if you cannot drop outliers you can try the following try a different model data detected as outliers by linear model can be fit by nonlinear model so we be sure you are choosing the right model so if it has like more of a curved look to it instead of a straight line you might need to use something other than just a straight line linear model try normalizing the data this way the extreme data points are pulled to a similar range if you can use algorithms which are less affected by outliers example random forest so there is another solution is you can come up with the random force which a lot of times completely bypasses your outliers how can you say that a time series data is stationary oh that's an interesting term stationary meaning it's not moving but it's a time series we can say that a time series is stationary when the variance and mean of the series is constant with time and this graphic example is very easy to see we have our the variance is constant with time so we have our first variable y and x and x being the time factor and y being the variable as you can see goes through the same values all the time it's not changing in the long period of time so that's stationary and then you can see in the second example the waves get bigger and bigger so that's non-stationary here the variance is changing with time again we have y which stays constant so that if you look at the bigger picture it's the same wave over and over again and then of course we have where the wave is growing in size going up it can also go down so it'd also be non-stationary how can you calculate accuracy using confusion matrix oh great one confusion matrixes are so useful when you're taking that first look at data and also when you're showing the shareholders and you want to ask them for money how can you calculate accuracy using confusion matrix so you have your total data that we're looking at is 650 and you have your predicted values and your actual values and your predicted p and your actual p and so we look at this you'll note that if the predicted p and the actual p are 262 but our predicted p also had 15 that weren't correct so you can see there's a false positive there of 15 and the same thing with the n you can see where n predicts n and it has a false negative of 26 out of the total number of n values in there and so we can do an accuracy on there the true positive plus the true negative is our total observations so you have a total of 0.93 accuracy or 93 percent and just a quick note on this this is so important because it's one thing if someone is being diagnosed with say cancer you know this is life death or is my nuclear reactor going to blow up suddenly if the p is the probability of it blowing up and this let's say you have 15 that's a lot less than say the 26 chances of it blowing up you know so the actual domain of your data is very important so if you're non-positive you don't really care about the predicted value having non-positive as positive because they're going to do a biopsy on the cancer or whatever anyway but you're very interested if you have a positive an actual positive value which is looked at as negative a false negative that's really important in that domain depending on what domain you're in write the equation and calculate precision and recall rate and so continuing with our confusion matrix i was just talking about the different domains we have the precision equals 262 over 277 so your precision is the true positive over the true positive plus false positive and the recall rate is your true positive over the total positive plus false negative and you can see here we have that 262 over 277 equals a 94 and the recall over here is the 262 over 280 which equals 9.9 or 90 and oh good we're going to take a pause for another brain teaser if a drawer contains 12 red socks 16 blue socks and 20 white socks how many must pull out to be sure of having a matching pair the last time i went through these kind of brain teaser things was like 20 years ago and i had six people sitting across the table waiting for my answer that's kind of mind-numbing when you're in an interview like that hopefully you're not stuck in an interview like that but uh on this you need to ask yourself how many different colors of socks are there so they've thrown a lot of extra data in here that you don't need to solve the answer the answer is four an example your first pick is white your second pick is red third pick is blue so no pairs yet and that means when you get to the fourth pick there's a hundred percent chance you're going to have a match so the most is going to be four that you ever have to pull out of your drawer if it was four colors the answer would be five and so on it doesn't matter how many white socks you have or how many red socks or blue socks different pairs you have it's the different colors a number of different colors people who bought this also bought recommendations seen on amazon as a result of which algorithm we covered this earlier recommendation engine is done with collaborative filtering collaborative filtering exploits the behavior of other users in their purchase history in terms of ratings selection etc it makes predictions on what you might interest a person based on the preference of many other users and this algorithm features of the items are not known and we have a nice example here where they took a snapshot of a sales page it says for example suppose x number people buy a new phone and then also buy tempered glass with it next time when a person buys a phone he'll be recommended to buy tempered glass along with it and if you remember the vocabulary words we covered earlier this is the recommendation this is collaborative the other word was content based so looking at things with similar content versus collaborative which is similar people remember you know you're not going to know every vocabulary word but it also doesn't hurt to get your three by five cards out and make yourself a vocabulary stack of cards buy an app on your phone for it sql query i remember back in the 90s it was so important to know sql query and only a few people got it nowadays it's just part of your kit you have to know some basic sql so write a basic sql query to list all orders with customer information and you can kind of make up your own name for the database and you can pause it here if you want to write that down on a paper and let's go ahead and look at this we have to list all orders with customer information and so usually you have an order table and a customer table and you have an order id a customer id order number total amount and then from your customer table you have id first name last name city county and so if we're going to write in sql with this we're going to select keyword there for sql selecting order number total amount first name last name city country so that's the columns we're going to look at we're going to do that from our order where we're going to join it with our customer and we're going to join it on the order customer id equals the customer id so very basic sql query that's going to return a table of data for us you are given a data set on cancer detection you've built a classification model and achieved an accuracy of 96 percent whoo 96 why shouldn't you be happy with your model performance what can you do about it that's an interesting one because this comes up that's one of the standard data sets on there is for cancer detection cancer detection results in imbalanced data in an imbalanced data set accuracy should not be based as a measure of performance because it is important to focus on the remaining four percent which are the people who were wrongly diagnosed we talked a bit about this earlier you have to know your domain you know this is the medical cancer domain versus weather domain you know whether channel they get by with 50 wrong in cancer you don't want four percent of the people being wrongly diagnosed wrong diagnosis is of a major concern because there can be people who have cancer but we're not predicted so in an imbalanced data set accuracy should not be used as a measurement performance which of the following machine learning algorithm can be used for inputting missing values of both categorical and continuous variables so we have a couple choices here we have k means clustering we have linear regression we have the k in in nearest neighbor and decision tree and which of the following machine learning algorithms can be used for inputting missing values of both categorical and continuous variables now certainly you can use some preprocessing to do some of that but you should have gone with the k nearest neighbor because it can compute the nearest neighbor and if it doesn't have the value it just computes the nearest neighbor based on all the other features where when you're dealing with k-means clustering or linear regression you need to do that in your pre-processing otherwise it'll crash decision trees also although there's some variants on that too can you solve another riddle always fun ones given a box of matches and two ropes not necessarily identical measure a period of 45 minutes and in this particular setup the ropes are not uniform in nature and the rope takes exactly 60 minutes to completely burn out so each rope takes up to 60 minutes to burn out and there's actually a couple different solutions to this but let me go ahead and one of the things is they're not uniform in nature so even though they take 60 minutes anyways let's go ahead and see what they did to solve it and then we can also look at different options we have two ropes a and b light a from both ends and b from one end okay when a is finished burning we know that 30 minutes have elapsed and b has 30 minutes remaining now like the other end of b also so that the remaining part of b will burn taking 15 minutes to burn this we have gotten 30 plus 15 equals 45 minutes excellent solution mine which i like was to take one rope fold it in two so we know it's a half hour take the other rope fold it in four places so we know that that one's 15 minutes and then you can just connect the two and burn it straight across i think they're trying to cover that by saying they're not regular the ropes are have some irregularities maybe that's what they meant by that you couldn't do something like that that's my solution below are the eight actual values of target variable in the train file so we have a training file not to be confused with the train on the tracks we have zero zero zero one one one one one what is the entropy of the target variable we mentioned earlier that you should know your entropy and how to calculate the entropy what is the entropy of the target variable so we have a couple options here we have minus five over eight logarithm of five over eight plus three over eight logarithm of three over eight okay let's just see where they got those numbers from we have uh one which is going to be five ones and three zeros and then we have a total of eight okay and then we have the option of five which is number of ones five eight logarithm of five eighths plus three eighths logarithm of three eighths and we also have three eighths logarithm of five eighths plus five eight logarithm of three eighths and then we kind of reverse those numbers around and let's see what you're going to get here which one did you think it was you should check the first one so what is the entropy of the target variable the key there is a target variable so we're looking at the target in this case is going to be one usually that's what you're looking for and so the entropy of that one we want to subtract out the entropy of the non-targeted variable whoops i had that backwards we want to we're looking at zero so we want to subtract out the 5 8 from there so 5 8 logarithm 5 8 or negative 5 8 logarithm 5 8 plus 3 8 logarithm 3 8. and they have the hint on the bottom entropy equals i of p of n so we have a negative p plus p n times the logarithm base 2 of p over p plus n minus the n over p plus n times logarithm 2 of n over p plus n we want to predict the probability of death from heart disease based on three risk factors age gender and blood cholesterol level what is the most appropriate algorithm for this case so we have three features and we want to know the predictability of death okay a little morbid there choose a right algorithm do we want to use logistic regression for this linear regression k means clustering or the aproria algorithm and if you selected logistic regression then you've probably got the right answer linear regression remember deals with like you take your line and draw a line through the data of course you don't necessarily have to use a straight line there's other means for that but you're dealing with a lot of numbers and k-means means we're just going to cluster objects together with the logistic regression though you can mix those things together in buckets so really the logistic regression is what you want to use in that model would be the most apt fit after studying the behavior of a population you have identified four specific individual types who are valuable to your study you would like to find all users who are most similar to each individual type which algorithm is most appropriate for this study certainly identifying census and just about a lot of different markets is common so maybe they have a census or whatever it means but let's take a look at some of the different algorithms we might use on this we have k-means clustering linear regression association rules and decision trees and i'll give you a hint we're looking for grouping people together by similarities and by four different similarities so very specific they gave you one of the values specifically the k value so k means clustering would be great for this particular problem you have run the association rules algorithm on your data set and the two rules banana apple is associated with grape and apple orange is associated with grape have been found to be relevant what else must be true so this would change you to understand association rules you could picture in this particular one you're going shopping and uh you almost always see somebody who has bananas they usually have grapes in their bag also and somebody who has apples usually has grapes in their bags and then apples and oranges is also associated with grapes and let's go and take a look at that and we have a couple different options here first one is banana apple and grape orange must be a frequent item set not so much banana apples oranges must be relevant rule grape is common with banana apple must be a relevant rule and how about grape apple must be a frequent item set let's go back and take a look at that and we notice that we have bananas apples to grapes we have apple orange to grape boy there's a lot of grapes and a lot of apples in there and so if you said the last one grape and apple must be a frequent item set then you got it correct your organization has a website where visitors randomly receive one of two coupons it is also possible that visitors to the website will not receive a coupon you have been asked to determine if offering a coupon to visitors to your website has any impact on their purchase decision which analysis method should you use and so let's go ahead and start by giving you another hint and give you some limiting your selection we have a one way anova k means clustering association rules and student t test so obviously you should know what each one of these means but let's take a look at the question again so you want to know which method should you use to see if the coupons are valid for their purchase well we're not clustering and we're not associating things together we want to know the end result student t-test also drawing that little t in boxes and switch them around there's really only one answer that works in here and that's a one-way anova so that draws us to an end i want to thank you for joining us today for more information visit www.simplylearn.com get certified get ahead feel free to visit our website and ask any additional questions you have you can also post questions down below in the youtube video and we'll try to get that one of our experts will try to reach out to you and answer those questions again thank you for joining us and happy learning and since these are interview questions good luck on your next interview or if you're the interviewer looking for questions to ask people good luck on your next interview hi there if you like this video subscribe to the simply learn youtube channel and click here to watch similar videos turn it up and get certified click here you