Introduction to Information Theory
摘要
The chapter covers information theoretic definitions and properties that will be required in further chapters. It is hence not a comprehensive IT text. It introduces Shannon’s information measures, including Entropy and Differential Entropy, Conditional Entropies (Equivocation and Irrelevance), Mutual Information and Channel Capacity, the Chain Rule, Typical Sequences and the Asymptotic Equipartition Property. Channel Capacities of typical channels are derived: the BSC, the BEC, AWGN with discrete or continuous input, and the bandlimited Gaussian channel. The chapter also introduces the Bhattacharyya bound for maximum likelihood decoding and some Rate-Distortion Theory.