Shannon's Information Entropy Explained with Yes/No Questions

Cross-entropy measures the average number of bits needed to identify messages from distribution Q when the true distribution is P. It equals the entropy of P plus the Kullback-Leibler divergence (KL divergence) between P and Q. KL divergence quantifies the extra information required when using an incorrect distribution. In machine learning, minimizing cross-entropy trains models to better match true data distributions. Deep learning systems estimate probability distributions over possible outcomes, and cross-entropy loss guides training by measuring how well predicted distributions match actual observations. This information-theoretic framework provides natural metrics for evaluating model performance in probabilistic prediction tasks.

Cross entropy measures the difference between two distributions P and Q: H(P,Q) = Σ P(x)log(1/Q(x)). It equals entropy when P=Q, and increases as distributions diverge. KL Divergence (D_KL(P||Q)) measures the excess entropy introduced by approximating P with Q: D_KL = H(P,Q) - H(P). It quantifies how much information is lost when using Q instead of P. Both concepts are fundamental in machine learning, with cross entropy being the second most important loss function after mean squared error. KL Divergence is non-negative and zero only when distributions are identical.

Cross-entropy compares two distributions P (reality) and Q (model): H(P,Q) = H(P) + D_KL(P||Q). KL divergence measures 'information loss' when using Q instead of P. Cross-entropy equals true entropy plus this penalty. This framework explains why cross-entropy is optimal for classification - it directly measures how well the model approximates reality.

Cross entropy between P and Q is defined as the negative expectation with respect to P of log Q(x). When minimizing KL divergence, the term E_P[log P(x)] (negative entropy of P) is constant with respect to θ and can be dropped. The remaining term is exactly the cross entropy, which is the relevant quantity to minimize for model optimization. Cross entropy can be interpreted as the expected negative log likelihood of Q when data is drawn from P. This provides an intuitive understanding: cross entropy measures how well the model Q predicts data generated by P, with lower values indicating better predictions.

KL divergence (Kullback-Leibler divergence) measures the difference between two probability distributions as D_KL(P||Q) = H(P,Q) - H(P) = Σ P_i × log(P_i/Q_i). In compression terms, it represents wasted bits per symbol from using a poorly optimized code. In machine learning, it acts as an asymmetric distance measure—zero when distributions are identical, growing as they differ. The compact form hides the fact that it's the difference between cross-entropy and entropy. When distilling models, using KL divergence instead of cross-entropy would measure the distance between distributions rather than the weighted average loss.

Mutual information I(X;Y) quantifies the amount of information shared between random variables X and Y. It can be expressed as I(X;Y) = H(X) - H(X|Y) = H(Y) - H(Y|X) = D(P(X,Y)||P(X)P(Y)). Mutual information is symmetric, non-negative, and equals zero when X and Y are independent.

Mutual information quantifies the amount of shared information between two random variables. When generating dependent random variables together, you can achieve savings compared to generating them independently. For example, if X and Y have mutual information of 1 bit, you can save 1 bit by exploiting their dependence. The mutual information equals the minimum number of bits that must be communicated to describe the relationship between X and Y.

Mutual information I(X;Y) measures the amount of information shared between two random variables X and Y. It is defined as the KL divergence between the joint distribution P(X,Y) and the product of marginals P(X)P(Y): I(X;Y) = D(P(X,Y)||P(X)P(Y)). This measures how much knowing one variable tells us about the other. If X and Y are independent, mutual information is zero.

Mutual information I(X;Y) measures shared information between variables: I(X;Y) = H(X) - H(X|Y) = H(Y) - H(Y|X). It equals E[log(P(X,Y)/(P(X)P(Y)))] and is symmetric. When independent, I(X;Y)=0. Mutual information quantifies how much knowing one variable reduces uncertainty about another. It satisfies I(X;Y) ≥ 0 and represents the overlap in information content between variables.

Mutual information I(X;Y) represents the shared information between two random variables X and Y. It is the information that X and Y have in common. It can be calculated as I(X;Y) = H(X) + H(Y) - H(X,Y), or equivalently as I(X;Y) = H(X) - H(X|Y) = H(Y) - H(Y|X).

Shannon's Noisy Coding Theorem establishes that reliable communication over a noisy channel is possible only if the transmission rate is below the channel capacity, defined as 1 minus the binary entropy function H(p) = -p log p - (1-p) log(1-p). For the binary symmetric channel with bit-flip probability p, if the rate R < 1-H(p), there exist encoding-decoding schemes where the probability of decoding error approaches zero as the message length increases; however, if R > 1-H(p), the probability of error approaches one regardless of the coding scheme. This theorem demonstrates that random codebook constructions with appropriate rates can achieve exponentially small error probabilities, vastly outperforming simple repetition codes which require the rate to approach zero.

Shannon's fundamental result states that reliable communication is possible if and only if the information rate R is less than channel capacity C (R < C). When R < C, there exist error-correcting codes enabling arbitrarily low error probability as block length increases. When R > C, reliable communication becomes impossible regardless of coding scheme. The proof demonstrates that random coding achieves this bound in the limit of long blocks. This separates communication into two optimal stages: first compress data to remove unnecessary redundancy, then add controlled redundancy through error-correcting codes to combat channel noise. The theorem guarantees theoretical limits can be approached, though practical implementations may fall short.

This section presents Shannon's most audacious claim: reliable communication at channel capacity with near-zero errors is possible with proper coding. By reducing source rate R below channel capacity C, the number of source sequences grows slower than channel output sequences, providing ample room to spread messages apart. Shannon showed that with sufficiently long block lengths, the probability a received sequence could be reasonably interpreted as more than one message approaches zero. Finally, Shannon extended entropy to continuous signals, unifying Nyquist, Hartley, and his own insights into the Hartley-Shannon theorem: C = B log₂(1 + S/N), defining the ultimate speed limit for reliable communication over any noisy channel constrained by power and bandwidth.

Shannon's Noisy Channel Coding Theorem states that for any communication channel, reliable communication at any rate below the channel capacity is possible with arbitrarily small error probability, provided sufficiently long codewords are used; this is proven for the binary symmetric channel by showing that random linear codes achieve this performance when averaged over all possible parity check matrices.

The thermodynamic entropy (Boltzmann's S = k × ln(W)) and the information-theoretic entropy (S = -Σ P_i × ln(P_i)) are mathematically equivalent. The number of microstates W corresponds to the probability distribution of microstates, and the logarithm of W relates to the information content. This connection shows that thermodynamic entropy and information entropy are different manifestations of the same fundamental concept. The units of entropy depend on the context and the base of the logarithm used - in thermodynamics, units are energy per temperature (J/K); in information theory, using natural logarithm gives 'nats' and base-2 logarithm gives 'bits'.

Entropy S measures the number of states with non-zero probability in a system. For discrete systems with M equally probable states, S = log M. For continuous systems, entropy equals the logarithm of the probability blob's volume in phase space. The general entropy formula S = -ΣP_i log P_i applies to any probability distribution. For N coins with complete ignorance, all 2^N states are equally probable, giving S = N × log 2 (entropy in bits). If exactly one coin is tails but unknown which, S = log N. Correlation occurs when measuring one part provides information about another, reducing total entropy below the sum of individual entropies. Uncorrelated subsystems have additive entropies. The relationship between information-theoretic entropy and thermodynamic entropy is essentially identical, differing only by Boltzmann's constant k_B. This unifies statistical mechanics with thermodynamics, showing how macroscopic irreversibility emerges from microscopic reversibility through our limited ability to track all system details.

Entropy in Boltzmann statistical mechanics is closely related to Shannon information theory. Both are ensemble theories of information. Entropy measures the organization of information - something with small entropy has less information content than something with high entropy.

Entropy in statistical mechanics and information theory share a deep mathematical connection, as both measure uncertainty—statistical mechanics counts the number of microscopic arrangements (microstates) that produce the same macroscopic state, while information theory quantifies uncertainty about outcomes; the principle of maximum entropy provides a framework for making unbiased predictions by selecting the probability distribution with the highest entropy given limited information, revealing that entropy fundamentally measures both physical disorder and informational uncertainty.

The Gibbs entropy formula in statistical mechanics and Shannon's information entropy are fundamentally the same concept, differing only in units (joules/Kelvin vs. bits). Shannon's noiseless coding theorem states that the most efficient code for communicating random experiment results requires an average of S_info = -ΣP(x)log₂P(x) bits, which equals the information entropy. This reveals that thermodynamic entropy measures the amount of missing information about a system's microscopic state. The second law of thermodynamics, which states that entropy always increases in isolated systems, can be interpreted as information loss: as systems evolve toward equilibrium, we lose access to information about their microscopic configurations, making the equilibrium state the point of maximum information loss.