Information and Communication Theory

STEFAN HÖST is a Senior Lecturer in the Department of Electrical and Information Technology at Lund University, where he is active in the Broadband Communication research group. As a teacher he has been responsible for several courses within communication technology at the master's level. His main research area is within access networks seen from the physical layer. He is a frequent contributor to IEEE journals and to papers presented at multiple IEEE conferences.

• Preface ixChapter 1 Introduction 1Chapter 2 Probability Theory 52.1 Probabilities 52.2 Random Variable 72.3 Expectation and Variance 92.4 The Law of Large Numbers 172.5 Jensen’s Inequality 212.6 Random Processes 252.7 Markov Process 28Problems 33Chapter 3 Information Measures 373.1 Information 373.2 Entropy 413.3 Mutual Information 483.4 Entropy of Sequences 58Problems 63Chapter 4 Optimal Source Coding 694.1 Source Coding 694.2 Kraft Inequality 714.3 Optimal Codeword Length 804.4 Huffman Coding 844.5 Arithmetic Coding 95Problems 101Chapter 5 Adaptive Source Coding 1055.1 The Problem with Unknown Source Statistics 1055.2 Adaptive Huffman Coding 1065.3 The Lempel–Ziv Algorithms 1125.4 Applications of Source Coding 125Problems 129Chapter 6 Asymptotic Equipartition Property and Channel Capacity 1336.1 Asymptotic Equipartition Property 1336.2 Source Coding Theorem 1386.3 Channel Coding 1416.4 Channel Coding Theorem 1446.5 Derivation of Channel Capacity for DMC 155Problems 164Chapter 7 Channel Coding 1697.1 Error-Correcting Block Codes 1707.2 Convolutional Code 1887.3 Error-Detecting Codes 203Problems 210Chapter 8 Information Measures For Continuous Variables 2138.1 Differential Entropy and Mutual Information 2138.2 Gaussian Distribution 224Problems 232Chapter 9 Gaussian Channel 2379.1 Gaussian Channel 2379.2 Parallel Gaussian Channels 2449.3 Fundamental Shannon Limit 256Problems 260Chapter 10 Discrete Input Gaussian Channel 26510.1 M-PAM Signaling 26510.2 A Note on Dimensionality 27110.3 Shaping Gain 27610.4 SNR Gap 281Problems 285Chapter 11 Information Theory and Distortion 28911.1 Rate-Distortion Function 28911.2 Limit For Fix Pb 30011.3 Quantization 30211.4 Transform Coding 306Problems 319Appendix A Probability Distributions 323A.1 Discrete Distributions 323A.2 Continuous Distributions 327Appendix B Sampling Theorem 337B.1 The Sampling Theorem 337Bibliography 343Index 347