Information Theory's Most Surprising Result Explained

The Information Capacity Theorem, also known as the Shannon-Hartley Theorem, states that the maximum error-free data rate (channel capacity) for a continuous communication channel with bandwidth B Hz and additive white Gaussian noise with power spectral density n₀/2 is given by C = B × log₂(1 + S/N), where S is the signal power and N is the noise power; this theorem establishes that if the information rate R is less than or equal to the channel capacity C, there exists a coding technique that allows faithful transmission with arbitrarily small probability of error.

The Shannon-Hartley Theorem states that the channel capacity C (maximum data rate in bits per second) is given by the formula C = B × log₂(1 + S/N), where B is the channel bandwidth in Hz, and S/N is the signal-to-noise ratio (linear power ratio, not in dB). This theorem establishes the fundamental limit for reliable communication over noisy channels, showing that to increase channel capacity, one must either increase the bandwidth or improve the signal-to-noise ratio.

The Shannon-Hartley theorem establishes that the maximum error-free data rate (channel capacity) for a communication channel is given by C = B × log₂(1 + S/N), where B is the channel bandwidth in Hz, S is the average received signal power, and N is the average noise power; this formula demonstrates that increasing bandwidth or signal-to-noise ratio logarithmically increases the maximum achievable data rate, providing a fundamental limit for reliable communication over noisy channels.

This comprehensive lesson covers Shannon's Third Law (Shannon-Hartley Law), which determines the maximum information capacity of a band-limited channel with additive white Gaussian noise. The theorem states C = B log₂(1 + S/N) bits per symbol, where B is bandwidth, S is signal power, and N is noise power. The proof begins with capacity defined as maximum mutual information I(X;Y) = H(Y) - H(Y|X). Through joint entropy analysis at the channel and receiver, it is shown that H(N) = H(Y|X). Substituting the entropy formulas for AWGN (H(N) = B log₂(2πeN) and H(Y) = B log₂(2πe(S + N))) and applying logarithm properties yields the final capacity formula. This fundamental result establishes that channel capacity depends on bandwidth and signal-to-noise ratio.

The Shannon-Hartley theorem defines maximum channel capacity: C = B × log2(1 + S/N), where B is bandwidth and S/N is signal-to-noise ratio. Bandwidth changes have a linear effect on capacity, while signal-to-noise ratio changes have a logarithmic effect. For example, a 20 MHz bandwidth with -90 dBm noise floor and -60 dBm signal yields 86 Mbps capacity. Doubling bandwidth doubles capacity, while improving S/N provides diminishing returns. This theorem establishes fundamental limits on reliable data transmission over noisy channels.

The source coding theorem states that for a discrete memoryless source, the average codeword length L must satisfy L ≥ H, where H is the source entropy, and code efficiency is given by η = H/L × 100%. Shannon-Fano coding is a practical implementation of this theorem that assigns shorter codes to more probable symbols and longer codes to less probable ones through a systematic partitioning process: arrange symbols in decreasing order of probability, partition them into two groups with nearly equal total probability, assign 0 to the upper group and 1 to the lower group, and repeat recursively until each group contains only one symbol; the code for each symbol is read horizontally from the partition table, and the average codeword length is calculated as the weighted sum of probabilities multiplied by their respective code lengths.

Shannon's Noiseless Coding Theorem, also known as the Source Coding Theorem, establishes that for a discrete memoryless source, the average codeword length L of any binary code must satisfy H(X) ≤ log₂(L) < H(X) + 1, where H(X) is the entropy of the source; this theorem demonstrates that entropy represents the fundamental limit on data compression efficiency, with the average codeword length being bounded below by the source's entropy and above by entropy plus one.

This section presents the formal proof of the source coding theorem, which establishes entropy as the fundamental lower bound on compression efficiency. The theorem states that for any uniquely decodable D-source code, H_D(X) ≤ E[L], with equality if and only if codeword lengths exactly equal -log_D(P(x)) and are integers. The proof constructs a probability distribution Q(x) = D^(-l_x)/Z and applies Gibbs' inequality to show that the difference between expected code length and entropy equals the divergence between P and Q plus log(Z), which is non-negative. Equality requires both the Kraft-McMillan sum equals 1 and P=Q. The section also proves that optimal codes always exist and can be chosen as prefix codes, though achieving equality with entropy requires source probabilities to be exact powers of D, which rarely occurs in practice.

The Source Coding Theorem states that the minimum average number of bits needed to encode a source equals its entropy. This video proves the theorem using a lottery game analogy: with 1000 coin flips where each flip has 90% probability of being 0 and 10% probability of being 1, the most likely outcome is all zeros. To achieve 99% probability of winning, one must buy all tickets containing at most 123 ones, which requires approximately 2^530 tickets—demonstrating that the minimum bits needed equals the entropy of the source.

This section covers the mathematical foundations of data compression. A discrete memoryless source generates independent, identically distributed symbols with fixed probabilities. As message length increases, random messages concentrate around 'typical' sequences matching expected symbol proportions. The Source Coding Theorem proves that optimal compression requires at least N×H bits for N-symbol messages, where H is entropy—the average information content per symbol. Entropy measures uncertainty reduction, with more probable outcomes providing less information. This theoretical framework establishes the lower bound for all compression algorithms.

This lecture presents four essential inequalities used throughout network information theory: (1) Jensen's inequality, which states that for a concave function f and random variable X, E[f(X)] ≤ f(E[X]), with the reverse for convex functions; (2) Log-sum inequality, which states that for non-negative numbers a_i and b_i, Σa_i log(a_i/b_i) ≥ (Σa_i) log((Σa_i)/(Σb_i)); (3) Data processing lemma, which states that if X, Y, Z form a Markov chain, then I(X;Z) ≤ I(X;Y) and I(X;Z) ≤ I(Y;Z), showing that information processing cannot increase mutual information; (4) Fano's lemma, which relates probability of error to uncertainty, stating that H(P_e) + P_e log(L-1) ≥ H(U|Ũ), where P_e is the probability of error, L is the number of possible values, and H(U|Ũ) is the conditional entropy. These inequalities are fundamental tools for proving converse theorems in information theory.

This talk introduces a reductionist paradigm for studying network information theory, where different communication problems (such as network coding, index coding, and interference alignment) are connected through reductions—showing that solving one problem enables solving others. The central open problem discussed is the 'edge removal' question: if a network can communicate at rate R with a certain edge, how much does removing that edge affect achievable rates? This problem connects to seemingly unrelated topics like zero-error versus epsilon-error communication, secure communication against adversarial jamming, and the relationship between dependent versus independent encoders. The reductionist approach reveals that many problems in network information theory are fundamentally interrelated, suggesting a path toward unifying the field.

Information theory began with Shannon's 1948 work on the simplest network: a single transmitter and receiver with noise. The field expanded to study three-node networks including broadcast channels, multiple access channels, relay channels, and interference channels, extensively researched in the 1970s-1980s. However, three major challenges impede progress: (1) Real-world networks have intermediate node counts (neither small nor asymptotically large), (2) Capacities don't compose - individual component capacities don't predict larger network capacities, and (3) Sequential investigation of individual network types doesn't reveal generalizable principles. These challenges explain why even three-node networks remain incompletely solved despite decades of research.

Algebraic network information theory explores using linear/lattice codebooks instead of random iid codebooks. Key applications include distributed compression, physical-layer network coding, dirty paper coding, and interference alignment. The goal is developing a general theory analogous to El-Gamal and Kim's work but exploiting linearity. Paddock et al. (2013) showed nested linear codes can achieve joint typicality performance using auxiliary indexing to simulate desired input distributions.

Network information theory provides fundamental principles for understanding the limits of information propagation in distributed networks. The key insight is that traditional blockchains built on full replication (where every node downloads all data) are severely limited by individual node bandwidth. The theoretical limit suggests that with N nodes each having bandwidth C, the system should scale at N*C, not just C. This represents a million-fold improvement potential for systems like Ethereum with 1 million validators. The speaker learned this principle from their PhD work on wireless networks and applied it to blockchain scalability, demonstrating how fundamental research can lead to transformative applications.