Building Your First Neural Network with Python and Keras

[00:00:00]Now it's time to build the first ever neural network.

[00:00:04]So as I said, we're using us.

[00:00:07]Um, we, as you see here, what we do is very, very simple.

[00:00:13]Actually, we're just creating a model and we are adding layers, layers to it.

[00:00:18]So basically model add a Cara Slayer model, add a Carol Slayer model, add a Carol Slayer.

[00:00:25]So it's quite simple.

[00:00:27]Um, In theory, but of course you can go as deep as you want.

[00:00:32]So let's start with the first line we have model model equals two care us models.

[00:00:37]Sequential.

[00:00:38]There are three ways how you can create models with Kara.

[00:00:41]So you have this quite sequential class, which is very simple.

[00:00:45]You just choose the predefined layers and then put layers on top of.

[00:00:49]Uh, there is a functional class, which is a little bit more detailed.

[00:00:53]You can create more complex, uh, neural networks with that one.

[00:00:58]Uh, but we don't need the functionality now.

[00:01:00]So we're just using the simplest one.

[00:01:01]There was also a model, uh, models library inside.

[00:01:06]And before that one, you basically make everything from scratch manually.

[00:01:10]And that is more for if you're doing research and if you're trying to achieve the highest possible a curiosi, uh, and that's not really something that we need here.

[00:01:20]All right.

[00:01:21]The second line here is a layer that we're.

[00:01:25]And the first layer that we're adding is basically the input layer.

[00:01:28]What is done here.

[00:01:30]And the flattened layer is because we have a 28 to 28 matrix.

[00:01:34]We want to make it one big string of 784 pixels.

[00:01:39]That is 28 times 28, 784.

[00:01:42]And I am saying the input shape is this much.

[00:01:45]Please flatten it.

[00:01:46]And flattening.

[00:01:46]It means that adding them one after another, making it one long line, and that is a number of inputs.

[00:01:53]Topic going.

[00:01:55]The next line we are creating actually the next two lines we are creating hidden.

[00:02:01]Uh, hidden layers or cold dense layers here.

[00:02:04]Well, not specifically hidden layers, but older layers are called dense layers.

[00:02:08]That means they are connected with everything that is before them or after them in a dense way.

[00:02:13]So you might see here this word and other places do either, do you have dense and sparse dense means every neuron is connected to every other.

[00:02:22]Sports means not every neuron is connected to every other neuron.

[00:02:25]So there are some missing links in between, and sometimes you need that too, but we will talk about that in the future lessons.

[00:02:30]So right now, all we need is dense layers for the second hidden or the first hidden layer.

[00:02:35]I have a 300 neuron layer and I'm going to keep the activation very low because as we talked about, as we mentioned briefly in the previous lessons about the building blocks of neural networks, value is one of the best ones that we can.

[00:02:48]Especially for hidden layers.

[00:02:50]Same for the second hidden layer.

[00:02:52]We have hundred neurons and we are setting the activation to be with.

[00:02:57]And the last one is our output layer.

[00:02:59]Actually we have 10 output layers.

[00:03:02]And why is that?

[00:03:03]It's because there are 10 classes and you want to classify our images, the input images into 10 different classes.

[00:03:10]So that starts from zero to 1, 2, 3, all the way to nine.

[00:03:13]And that makes 10 different classes.

[00:03:15]And we want our model to make a prediction for each of these classes.

[00:03:19]So every time they give it, an image is going to tell us how likely it is that it says zero.

[00:03:24]And then how likely it is that.

[00:03:26]And so on and so forth and also how likely it is that it's a nine and at the end we will have, uh, the possibilities.

[00:03:33]Oh, which number this image is here.

[00:03:37]Uh, you might realize that we were using softmax softmax, I'll just quickly show you what it looks like.

[00:03:46]So it looks like this, it goes from zero to one.

[00:03:51]And in that case, it looks very similar to six.

[00:03:54]Uh, but then you can ask, why are we using soft max and not signaling because in this exercise and this problem specifically the classic.

[00:04:03]Uh, I not related to each other, so they cannot, one image can not be half, one and half the other.

[00:04:10]They are completely in the, in the independent.

[00:04:13]Yes.

[00:04:14]And, uh, what Sigma does is it creates possibility that creates probabilities that are independent from each other.

[00:04:22]Same thing can have 0.5 possibility to be two different things.

[00:04:26]Whereas for softmax all the possibilities of summation of all the possibilities and.

[00:04:30]Lead to one.

[00:04:31]So they sum to one.

[00:04:33]So that means that if something possibility of an image being two is higher, that means possibility of it being three needs to be lover.

[00:04:41]Uh, and that's, that's the kind of thing we need because the images we give can only be one thing.

[00:04:45]They cannot be multiple.

[00:04:48]Uh, and that's why we're using softmax.

[00:04:50]So once we build our model, we can look at a summary of our model.

[00:04:55]Um, so this is a nice little function that tells you what your model looks like.

[00:05:00]Uh, let's see.

[00:05:02]So, as I said, the flattened layer, the input layer has 784.

[00:05:07]The dense layer has 300 neurons.

[00:05:10]Uh, the first hidden layer, the dense too, you know, they cannot cope be called the same thing.

[00:05:15]So that's why they have, they underscore one underscore to the first hidden layer.

[00:05:19]It has 300 neurons.

[00:05:21]The second hidden layer has hundred neurons and the output layer has 10 neurons.

[00:05:26]And for each of these, we also have the number of.

[00:05:29]And also we have a total number of parameters.

[00:05:31]And what are these parameters?

[00:05:33]As I said, for every connection between two neurons, we have the weights and for every neuron itself, we had the bias.

[00:05:38]So in total we have a lot of parameters and what our model is going to do is update.

[00:05:44]Uh, weights and biases to give us the best security possible.

[00:05:50]And if you want to do it, you can also take a look at what your layers look like.

[00:05:57]So Kara is basically has a lot of, um, nice functions or commands for you to look into the model that you built.

[00:06:04]Of course, it's not always going to be the simple, you know, you're going to build something more complex sometimes, and you might need to look at some summaries or look into the layers and see what your.

[00:06:15]The next thing that we want to do is to compile this model.

[00:06:18]So it was.

[00:06:20]What we're doing is we're basically giving it some hyper parameters here and hyper-parameters are things that you have to set yourself beforehand.

[00:06:28]They can not be learned.

[00:06:29]So parameters, as you see here are things that going to be learned is also says trainable parameters.

[00:06:35]These are all going to be learned.

[00:06:36]So they're going to either be initialized to zero, thus for.

[00:06:40]Or they're going to be initialized, randomly that's for weights.

[00:06:43]Um, but on top of that, while you're compiling your model, you also give it some, uh hyper-parameters.

[00:06:50]So these are things that you basically hard-code, and that are not going to be changed.

[00:06:54]Um, all of these options.

[00:06:57]You can actually see for all of these things, but options that you have.

[00:07:01]So for example, for loss, for using sparse categorical cross entropy, and that means that is this thing, spar, sparse, categorical cross entropy is how we're going to measure how good our model is doing.

[00:07:14]So, uh, you have some other ones too, so let's go here.

[00:07:18]I'll, I'll show you some other options.

[00:07:20]So for everything basically on Cara's, you can.

[00:07:24]Uh, see what options you have.

[00:07:27]So it's says poor, realistic losses, aggression losses.

[00:07:30]After a while of working with deep learning or generally machine learning, you'll probably have an idea which one to use.

[00:07:36]Uh, but generally, if you don't have an idea, you can Google for it or you can come and look here at the, uh, options.

[00:07:43]So let's go and find the one that I'm using right now.

[00:07:47]Sparse, categorical, cross entropy.

[00:07:50]And it says, use this cross entropy loss function when there are two or more labeled classes.

[00:07:55]And that's what I have.

[00:07:56]I have more than two label classes right now.

[00:07:58]And that's why I'm using suppress categorical cross entropy.

[00:08:01]Uh, but as I said, based on your problem, if it's a classification regression problem, or what do you want to get out of it and the type of output that you're getting, you're going to want to change this.

[00:08:11]Of course.

[00:08:12]Uh, but yeah, we will talk more about these in the future.

[00:08:16]Uh, optimizer.

[00:08:17]We are using a stochastic gradient descent.

[00:08:20]Again, you can find the available optimizers here.

[00:08:24]Let's take a look quickly.

[00:08:26]So we have this one, the gradient descent, one AIDEM ADA, grad, ADA, max, Natum all of these things.

[00:08:33]Um, of course these are not only made randomly these choices.

[00:08:38]So right now we are using gradient descent because that's kind of like the basis one that you use.

[00:08:43]But if you want to make your model faster, for example, you can use some other things.

[00:08:46]But as I said, we will go take a more systematic look on which one to choose.

[00:08:51]And when, uh, in the coming.

[00:08:54]The last one is the metrics.

[00:08:57]This is basically what you want your model to report on while it's training.

[00:09:01]So you don't only want to see the loss of reported.

[00:09:04]If you want to see some other things reported, you can also add them here and then your model will report it for you.

[00:09:09]So let's compile this model and, uh, I don't need this one.

[00:09:16]No, yes.

[00:09:18]So the last thing that we are going to do is basically.

[00:09:21]The model around our data.

[00:09:22]And that's why here we have the training, uh, X and Y values for the training.

[00:09:27]And we have some other things I'm going to explain.

[00:09:30]Um, the one thing that you might want to know again, here is a website where you can see.

[00:09:40]What are the options of things that you want to set?

[00:09:43]So here you're seeing I edit the datasets, the X and Y value for the dataset, what the model is going to fit on.

[00:09:50]But on top of that, I am setting the airbox validation data and that size, these are not the only things you can set.

[00:09:57]Of course you can set all of these things.

[00:09:59]So right now I'm giving X and Y I'm giving size.

[00:10:02]I'm giving ebox, I'm also giving validation data, and these are.

[00:10:06]All the other things that you can still set now, we don't need to do them right now, but you can come here and read what they all mean.

[00:10:13]So for example, my bows would mean, give me more information about what's happening while the, uh, the model is training.

[00:10:21]Um, or for example, instead of giving the validation data here, specifically, we can say validation split and say like this validation spit, uh, equals 0.1.

[00:10:33]So.

[00:10:34]The model for set asides is a 0.1, 0.1 of the whole training set aside to use as validation data.

[00:10:43]So you don't have to do the split beforehand.

[00:10:46]Um, so these are all the options that you can have things that you need.

[00:10:50]You don't need to use them right now, but you know, just in the future, if you want to check it out and also these values that you see here as equal are the default values.

[00:10:58]So if you don't specify it, this is what it's going to be like.

[00:11:01]So if you don't specify the epoch, it's just going to be.

[00:11:04]Um, all right.

[00:11:07]So here, what we do, as I said, we give the, uh, the training input and the training, true values for the output.

[00:11:15]Uh, epoch means how many times I want this model to be run on the whole dataset.

[00:11:21]And bedside means how many, um, data points.

[00:11:26]Are going to be given to the model at each time.

[00:11:28]And these things are kind of hard to compare.

[00:11:30]That's why I also written a note here.

[00:11:32]Uh, you have access to the notebooks.

[00:11:34]So while you're working on it, you can also read these little notes and use these little links.

[00:11:38]Um, but so here's how they're different.

[00:11:41]Bite-size means every time the model is running.

[00:11:46]So if you remember back for vacation, what happens is we calculate old values and backpropagation looks at the.

[00:11:53]And then changes the Bates and the biases of the model.

[00:11:56]Uh, what happens here is if you do not have a backside that is smaller than the number of instances that you already have, uh, what's what's going to happen is the model is going to be trained on all data points and then it's going to be updated, but that's kind of slow.

[00:12:15]That's why we sometimes use mini batches and mini batches are, can be as small as 1316.

[00:12:20]Even what you do is you train the model on only 32 as stated here, tota two examples, update all the weights and biases, and then you take the next 33 and do the same thing.

[00:12:32]And the next two to three and do the same thing.

[00:12:34]And once you're done with all the training examples, you have done one epoch and you do this 30 times.

[00:12:43]In this training and that means you've done 30 E-box, but you trained it on a lot of different batches.

[00:12:50]So once we run the model, maybe it will make more sense for you.

[00:12:53]Let's see.

[00:12:55]So as you see here, we have the epoch number.

[00:12:59]It says one out of 30.

[00:13:01]This is, this was the first ebook.

[00:13:02]The first time the model saw all the examples.

[00:13:07]Uh, here, we see how fast it's going with, see how the loss is changing for the training, how the courtesy is changing, uh, here and after it's done it also calculates these things for the validation set.

[00:13:20]So we've already given it a validation, right?

[00:13:22]So as you see here, the validation loss is getting lower and over and validation.

[00:13:28]Accuracy will be here.

[00:13:30]It's also getting higher and higher.

[00:13:34]And basically this training loss and accuracy.

[00:13:39]These are the training loss and accuracy, I guess I can't always tell like these ones, but anyways, these two at the training loss and accuracy, and we have validation, loss and accuracy.

[00:13:50]And if you see that your models lost training losses, getting lower and lower, whereas the validation loss starts getting higher at some point, that means your model is starting to overfit to your training.

[00:14:02]And we don't want that.

[00:14:03]So this is one way of looking at it.

[00:14:05]Um, what else?

[00:14:07]So we have some other numbers here.

[00:14:09]This is the amount of batches that we have.

[00:14:12]So we said we have 32.

[00:14:14]Uh, we want to put 32 examples inside the model, every time to train it and to do the backpropagation and the gradient descent step.

[00:14:24]And that means that our data, our model runs on 1,719, uh, So it's basically, you know, let's remember how many training examples that we had.

[00:14:39]Of course this needs to finish first.

[00:14:42]It's nearly down, so let's wait for it.

[00:14:47]All right.

[00:14:48]So it's done.

[00:14:50]Uh, just want the shape.

[00:14:53]So we have 5,000 now, 55,000 exempt.

[00:14:59]And if I divide them into two to two chunks, what we have is 1,719.

[00:15:05]Well, we have to round it up, uh, different groups.

[00:15:09]So it was like 32, 32 quarter to quarter to 1,719 of them.

[00:15:14]And when it does all of these passes, then we have, we have done a whole pass on a whole dataset and then in the whole data set, and then it goes to the next that you park.

[00:15:23]So let's look at some things here.

[00:15:27]Uh, the loss as we see the training loss, I see that it keeps getting lover and lover and lover.

[00:15:34]It's a good thing.

[00:15:34]It means that we are fitting the data set very nicely.

[00:15:38]Um, the accuracy is also getting higher and higher.

[00:15:42]Let's look at the validation loss.

[00:15:45]It gets low over here.

[00:15:49]Nice that it gets lover and lover.

[00:15:52]Well, now that I see that gets lower and lower, maybe I could have increased the number of E-box even, maybe I could have trained this data, said this network for longer.

[00:16:02]And I also want to look at the training validation, accuracy.

[00:16:07]It also looks like it's getting higher, so that's perfect.

[00:16:13]As you see here, sometimes it got a bit lover.

[00:16:16]Uh, oh, no, it didn't.

[00:16:18]Yeah, I guess here that it was point 98 and then it went back to point 97 94.

[00:16:23]Um, so I guess after a while or a curious, he didn't really increase much anymore, so it's good to know that there is a limit to how much we can, uh, train the state.

[00:16:37]So the best size for the batch sizes is I've read somewhere that it's actually best if you set it to something like the increment of two.

[00:16:47]So it's, you know, 16 30, 2, 6428, 256 sort of thing.

[00:16:52]It's because basically how computers are built and.

[00:16:57]Network wild training, of course, is using your, the processor of your computer.

[00:17:01]And that is based on the bedside system.

[00:17:04]Right.

[00:17:04]And that's always in increments of two.

[00:17:06]So they say it makes it faster to use increments of two.

[00:17:11]So I can of course change the batch size to something else if I want to.

[00:17:15]And if I can run it again, but it's going to run a little bit slower that's because.

[00:17:19]Now the model is calculating, uh, the partial derivatives that we talked about and doing backpropagation and gradient descent, every 10 examples.

[00:17:29]And that's going to make it a little bit slower because of all the calculation that needs to be done.

[00:17:34]And as you can see, now, we have more, um, steps to go through.

[00:17:38]So we have 5,500 to example, uh, groups of 10 examples together before we can calculate, uh, the partial derivatives and do gradient descent.