Probability, Random Variables, Statistics, and Random Processes

Ali Grami is a founding faculty member at the University of Ontario Institute of Technology (UOIT), Canada. He holds B.Sc., M.Eng., and Ph.D. degrees in Electrical Engineering from the University of Manitoba, McGill University and the University of Toronto, respectively. Before joining academia, he was with the high-tech industry for many years, where he??was the principal designer of the first North-American broadband access satellite system. He has taught at the University of Ottawa and Concordia University. At UOIT, he has also led the development of programs toward bachelor's, master's, and doctoral degrees in Electrical and Computer Engineering.

• Preface xiiiAcknowledgments xvAbout the Companion Website xviiPart I Probability 11 Basic Concepts of Probability Theory 31.1 Statistical Regularity and Relative Frequency 31.2 Set Theory and Its Applications to Probability 51.3 The Axioms and Corollaries of Probability 121.4 Joint Probability and Conditional Probability 181.5 Statistically Independent Events and Mutually Exclusive Events 211.6 Law of Total Probability and Bayes’ Theorem 281.7 Summary 32Problems 322 Applications in Probability 372.1 Odds and Risk 372.2 Gambler’s Ruin Problem 412.3 Systems Reliability 432.4 Medical Diagnostic Testing 472.5 Bayesian Spam Filtering 502.6 Monty Hall Problem 512.7 Digital Transmission Error 542.8 How to Make the Best Choice Problem 562.9 The Viterbi Algorithm 592.10 All Eggs in One Basket 612.11 Summary 63Problems 633 Counting Methods and Applications 673.1 Basic Rules of Counting 673.2 Permutations and Combinations 723.2.1 Permutations without Replacement 733.2.2 Combinations without Replacement 733.2.3 Permutations with Replacement 743.2.4 Combinations with Replacement 743.3 Multinomial Counting 773.4 Special Arrangements and Selections 793.5 Applications 813.5.1 Game of Poker 813.5.2 Birthday Paradox 833.5.3 Quality Control 863.5.4 Best-of-Seven Championship Series 863.5.5 Lottery 893.6 Summary 90Problems 90Part II Random Variables 954 One Random Variable: Fundamentals 974.1 Types of Random Variables 974.2 The Cumulative Distribution Function 994.3 The Probability Mass Function 1024.4 The Probability Density Function 1044.5 Expected Values 1074.5.1 Mean of a Random Variable 1074.5.2 Variance of a Random Variable 1104.5.3 Moments of a Random Variable 1134.5.4 Mode and Median of a Random Variable 1144.6 Conditional Distributions 1164.7 Functions of a Random Variable 1204.7.1 pdf of a Function of a Continuous Random Variable 1214.7.2 pmf of a Function of a Discrete Random Variable 1234.7.3 Computer Generation of Random Variables 1244.8 Transform Methods 1254.8.1 Moment Generating Function of a Random Variable 1254.8.2 Characteristic Function of a Random Variable 1264.9 Upper Bounds on Probability 1274.9.1 Markov Bound 1274.9.2 Chebyshev Bound 1284.9.3 Chernoff Bound 1284.10 Summary 131Problems 1315 Special Probability Distributions and Applications 1375.1 Special Discrete Random Variables 1375.1.1 The Bernoulli Distribution 1375.1.2 The Binomial Distribution 1385.1.3 The Geometric Distribution 1405.1.4 The Pascal Distribution 1425.1.5 The Hypergeometric Distribution 1435.1.6 The Poisson Distribution 1445.1.7 The Discrete Uniform Distribution 1465.1.8 The Zipf (Zeta) Distribution 1475.2 Special Continuous Random Variables 1485.2.1 The Continuous Uniform Distribution 1485.2.2 The Exponential Distribution 1495.2.3 The Gamma Distribution 1515.2.4 The Erlang Distribution 1525.2.5 The Weibull Distribution 1525.2.6 The Beta Distribution 1535.2.7 The Laplace Distribution 1545.2.8 The Pareto Distribution 1555.3 Applications 1565.3.1 Digital Transmission: Regenerative Repeaters 1565.3.2 System Reliability: Failure Rate 1575.3.3 Queuing Theory: Servicing Customers 1585.3.4 Random Access: Slotted ALOHA 1595.3.5 Analog-to-Digital Conversion: Quantization 1605.4 Summary 161Problems 1626 Multiple Random Variables 1656.1 Pairs of Random Variables 1656.2 The Joint Cumulative Distribution Function of Two Random Variables 1676.2.1 Marginal Cumulative Distribution Function 1696.3 The Joint Probability Mass Function of Two Random Variables 1706.3.1 Marginal Probability Mass Function 1706.4 The Joint Probability Density Function of Two Random Variables 1716.4.1 Marginal Probability Density Function 1726.5 Expected Values of Functions of Two Random Variables 1736.5.1 Joint Moments 1746.6 Independence of Two Random Variables 1756.7 Correlation between Two Random Variables 1786.8 Conditional Distributions 1856.8.1 Conditional Expectations 1866.9 Distributions of Functions of Two Random Variables 1886.9.1 Joint Distribution of Two Functions of Two Random Variables 1916.10 Random Vectors 1926.11 Summary 197Problems 1987 The Gaussian Distribution 2017.1 The Gaussian Random Variable 2017.2 The Standard Gaussian Distribution 2047.3 Bivariate Gaussian Random Variables 2107.3.1 Linear Transformations of Bivariate Gaussian Random Variables 2137.4 Jointly Gaussian Random Vectors 2157.5 Sums of Random Variables 2177.5.1 Mean and Variance of Sum of Random Variables 2177.5.2 Mean and Variance of Sum of Independent, Identically Distributed Random Variables 2187.5.3 Distribution of Sum of Independent Random Variables 2187.5.4 Sum of a Random Number of Independent, Identically Distributed Random Variables 2197.6 The Sample Mean 2207.6.1 Laws of Large Numbers 2227.7 Approximating Distributions with the Gaussian Distribution 2237.7.1 Relation between the Gaussian and Binomial Distributions 2237.7.2 Relation between the Gaussian and Poisson Distributions 2257.7.3 The Central Limit Theorem 2267.8 Probability Distributions Related to the Gaussian Distribution 2307.8.1 The Rayleigh Distribution 2307.8.2 The Ricean Distribution 2317.8.3 The Log-Normal Distribution 2317.8.4 The Chi-Square Distribution 2327.8.5 The Maxwell–Boltzmann Distribution 2327.8.6 The Student’s t-Distribution 2337.8.7 The F Distribution 2347.8.8 The Cauchy Distribution 2347.9 Summary 234Problems 235Part III Statistics 2398 Descriptive Statistics 2418.1 Overview of Statistics 2418.2 Data Displays 2448.3 Measures of Location 2498.4 Measures of Dispersion 2508.5 Measures of Shape 2558.6 Summary 257Problems 2579 Estimation 2599.1 Parameter Estimation 2599.2 Properties of Point Estimators 2609.3 Maximum Likelihood Estimators 2659.4 Bayesian Estimators 2709.5 Confidence Intervals 2729.6 Estimation of a Random Variable 2749.7 Maximum a Posteriori Probability Estimation 2759.8 Minimum Mean Square Error Estimation 2779.9 Linear Minimum Mean Square Error Estimation 2799.10 Linear MMSE Estimation Using a Vector of Observations 2829.11 Summary 285Problems 28510 Hypothesis Testing 28710.1 Significance Testing 28710.2 Hypothesis Testing for Mean 29110.2.1 p-Value 29410.3 Decision Tests 30010.4 Bayesian Test 30310.4.1 Minimum Cost Test 30410.4.2 Maximum a Posteriori Probability (MAP) Test 30510.4.3 Maximum-Likelihood (ML) Test 30510.4.4 Minimax Test 30710.5 Neyman–Pearson Test 30710.6 Summary 309Problems 309Part IV Random Processes 31111 Introduction to Random Processes 31311.1 Classification of Random Processes 31311.1.1 State Space 31411.1.2 Index (Time) Parameter 31411.2 Characterization of Random Processes 31811.2.1 Joint Distributions of Time Samples 31811.2.2 Independent Identically Distributed Random Process 31911.2.3 Multiple Random Processes 32011.2.4 Independent Random Processes 32011.3 Moments of Random Processes 32011.3.1 Mean and Variance Functions of a Random Process 32111.3.2 Autocorrelation and Autocovariance Functions of a Random Process 32111.3.3 Cross-correlation and Cross-covariance Functions 32411.4 Stationary Random Processes 32611.4.1 Strict-Sense Stationary Processes 32611.4.2 Wide-Sense Stationary Processes 32711.4.3 Jointly Wide-Sense Stationary Processes 32911.4.4 Cyclostationary Processes 33111.4.5 Independent and Stationary Increments 33111.5 Ergodic Random Processes 33311.5.1 Strict-Sense Ergodic Processes 33311.5.2 Wide-Sense Ergodic Processes 33311.6 Gaussian Processes 33611.7 Poisson Processes 33911.8 Summary 341Problems 34112 Analysis and Processing of Random Processes 34512.1 Stochastic Continuity, Differentiation, and Integration 34512.1.1 Mean-Square Continuity 34512.1.2 Mean-Square Derivatives 34612.1.3 Mean-Square Integrals 34712.2 Power Spectral Density 34712.3 Noise 35312.3.1 White Noise 35312.4 Sampling of Random Signals 35512.5 Optimum Linear Systems 35712.5.1 Systems Maximizing Signal-to-Noise Ratio 35712.5.2 Systems Minimizing Mean-Square Error 35912.6 Summary 362Problems 362Bibliography 365Books 365Internet Websites 368Answers 369Index 387