Mutual Information Explained: A Clear Step-by-Step Guide

[00:00:00][Music] Mutual ination it's really cool going to check it out now stack Quest hello I'm Josh starmer and welcome to stack Quest today we're going to talk about Mutual information and it's going to be clearly explained I don't want to spend a lot of time scaling up my stuff to work in the cloud I would rather spend my time working on my stuff cuz that's the fun part light in this stat Quest is also sponsored by the letters a b and c a always b b c curious always be curious imagine we had a data set with a lot of variables which are also called features and because we want to simplify the amount of time we spend collecting data we wanted to remove moove some of them thus we want to know how much each variable can tell us about the thing we want to predict in this case we want to predict if someone loves the movie Troll 2 and we want to know which of these variables likes popcorn height Etc play the biggest role in making good predictions in theory we could use something like R squar to see if a specific variable is related to love Troll 2 except r s only works with continuous data and Love's Troll 2 is yes or no and so is likes PopCorn by the way if you're not familiar with r squar you probably should be so check out the quest so when we have a mixture of continuous and discrete variables how do we quantify their relationship to the thing we want to predict well one way to quantify how each variable is related to love Troll 2 is to use Mutual information like R squ Mutual information is a numeric value that gives us a sense of how closely related two variables are bam the equation for calculating Mutual information looks kind of nasty but don't worry we'll go through it one step at a time in a nutshell these two sigmas Greek characters that stand for summation tell us that we're going to do a lot of addition and we're going to be adding up joint probabilities and dividing some of those joint probabilities by marginal probabilities um what are joint and marginal probabilities joint probabilities are just the probability of two things occurring at the same time for example given our data set we can calculate the probability that someone likes popcorn and loves troll too in this case the probability that someone likes popcorn and loves troll too is 3 divided 5 because three of the five people in the data set like popcorn and love Troll 2 in contrast marginal probabilities are just the probability of one thing occurring for example if we just focus on likes popcorn we can calculate the probability that someone does not like popcorn in this case the prob ability that someone does not like popcorn is 2 ided 5 because two of the five people in the data set do not like popcorn likewise we can calculate the marginal probability that someone does not love Troll 2 the probability that someone does not love Troll 2 is 1 divided 5 because only one of the five people in the data set does not love Troll 2 now for any pair of variables like likes popcorn and loves Troll 2 we can keep track of their joint and marginal probabilities in a table in this table the first two columns represent whether or not someone likes popcorn and the first two rows represent whether or not someone loves Troll 2 The Joint probability that someone likes popcorn and loves Troll 2 which we calculated earlier goes in the upper leftand corner of the table table likewise we can calculate the joint probability that someone does not like popcorn and loves Troll 2 and put that in the top of the second column now if we want to know the marginal probability that someone loves Troll 2 we can either calculate it like we did earlier by dividing the number of people that love troll to by the total number of people or we can simply add the joint probability that someone likes popcorn and loves Troll 2 to the Joint probability that someone does not like popcorn and loves troll too and when we do the math we get four / 5 so either way we get the same marginal probability 4 / 5 likewise we can solve for the remaining joint probabilities and plug them into the table and we can solve for the marginal probability that some some one does not love Troll 2 either directly from the data or we can add up the two joint probabilities in the second row in the table either way we get the same result then we can calculate the marginal probability that someone likes popcorn either directly or by adding the values in the column together then we can calculate the marginal probability that someone does not like popcorn now we have a t table filled out with all of the joint and marginal probabilities associated with likes popcorn and loves Troll 2 bam note the marginal probabilities are all in the margins of the table so that's where the name comes from small bam anyway now that we have this table of joint and marginal probabilities for likes popcorn and loves Troll 2 we can calculate their Mutual information by plugging the joint and marginal probabilities into this equation The Joint probabilities go here and here and the marginal probabilities both go here note the two summations ensure that we include all possible combinations of the variables likes popcorn yes and no with L's Troll 2 yes and no for example we start by plugging in the joint and marginal probabilities for where both popcorn and L's Troll 2 equal yes then we add a term where popcorn is yes and lov's Troll 2 is no then we add a term where popcorn is no and L Troll 2 is yes and lastly a term where popcorn is no and L's Troll 2 is no now that we have expanded this double summation by adding terms for for all possible combinations of popcorn and L's Troll 2 we plug in the joint and marginal probabilities that we calculated earlier for example The Joint probability that someone likes popcorn and troll to is 3 / 5 so we plug that in and then we plug in the marginal probability that someone likes popcorn 3 divided 5 and the marginal probability that someone loves troll two for divided 5 then we just plug in the joint and marginal probabilities for all of the other terms note before we move on I want to point out that the log function that we use in this equation can be any base however the default log function for most machine learning and most programming languages is the natural log so that's what we use here anyway when we do the math the whole thing is equal to 0.22 so the mutual information between likes popcorn and L's Troll 2 is 0.22 we can then compare this Mutual information value 0.22 to the mutual information values for other variables to decide which ones are the most useful double bam oh no it's a super technical note if you look carefully at this second term in the equation you'll see that if we divide 0 by 5 then we'll get a zero here and we'll get a zero in this numerator which means the whole term is 0 * the log of 0 and technically the log of 0 is not defined the good news is that even though the log of 0 is not defined as X gets close to zero x * the log of x equals 0 so the second term is just equal to zero tiny bam now let's go back to the raw data and see what happens when we change it so that likes popcorn is always yes in other words let's see what the mutual information is now that likes popcorn is always yes and never changes so just like before let's fill out the table of the joint and marginal probabilities and now let's use the table to calculate the mutual information first let's expand the summation so that we have a term for all possible combinations of likes popcorn and loves troll to then we plug numbers into each term and when we do the math we get zero in other words when likes popcorn never changes it can't tell us anything about what's happening in L's Troll 2 and in general when at least one of the two columns never changes then the mutual information will be zero because something that never changes can't tell us about something that does now let's go back to the raw data and see what happens when we change it so that likes popcorn is yes when L's Troll 2 is yes and likes popcorn is no when L's Troll 2 is no in other words now both columns change but they change in the exact same ways so let's fill out the table of the joint and marginal probabilities and use the table to calculate the mutual information ploo and when we do the math we get something close to but not exactly 0.5 so given this data set where the two columns change and they change in the exact same ways the mutual information is close to 0.5 and this value is larger than 0.22 which is what we got when we calculated the mutual information with the original data and remember both columns changed in the original data but not in exactly the same ways so when both columns change and the changes in one tell us more about what's going on in the other then the mutual information value is larger now let's go back to the raw data again and and see what happens when we change it so that likes popcorn is yes when Love's Troll 2 is no and likes popcorn is no when Love's Troll 2 is yes in other words let's see what the mutual information is when both columns change but in the exact opposite ways so we fill out the table of the joint and marginal [Music] probabilities and plug the numbers into the equation for Mutual information and when we do the math just like when we had both columns change in the exact same ways we get something close to but not exactly 0.5 in other words when both columns change it doesn't matter if they change in the exact same or exact opposite ways they both give us the same Mutual information and the changes in one column can tell us exactly what is changing in the other dou bam so far we've seen how to calculate Mutual information for two discrete variables likes popcorn and loves troll to how do we calculate the mutual information when we have a continuous variable like height when we want to calculate the mutual information and one or more of the variables as continuous then we simply create a histogram of the continuous values and then we can use each Ben in the histogram as a discrete category and use the discrete categories to calculate the joint and marginal probabilities note because we have three bins in the histogram we have three columns of joint probabilities in the table and that means that when we expand the equation for Mutual information we end up with six terms because we have Troll 2 equals yes and Troll 2 equals no for each of the three bins then we just do the math and get 0.22 and since we got 0.22 for both likes popcorn and height we can use either one triple bam one last note if you're familiar with how entropy is used in data science and if not check out the quest then you may have noticed that the equation for Mutual information has a lot in common with the equation for entropy they are both sums of probabilities times logs these equations are similar because Mutual information can be derived from entropy and in a sense Mutual information tells us how on average the surprise or change we see in one variable is related to the surprise or change in another for example do you remember when we calculated the mutual information and likes popcorn never changed well when we have something that never changes then the surprise is always zero so it's no surprise that the mutual information is also zero because the changes in L's Troll 2 have nothing to be related to in contrast when the surprise is greater than zero for both variables and the changes are related then the mutual information will also be greater than zero and reflect how much the changes are related bam now it's time for some Shameless self-promotion if you want to review statistics and machine learning offline check out the stat Quest PDF study guides and my book the stat Quest Illustrated guide to machine learning at stack quest.org there's something for everyone hooray we've made it to the end of another exciting stack Quest if you like this stat Quest and want to see more please subscribe and if you want to support stat Quest consider contrib to my patreon campaign becoming a channel member buying one or two of my original songs or a t-shirt or a hoodie or just donate the links are in the description below all right until next time Quest on