Smart Grid using Big Data Analytics

Robert Caiming Qiu, Professor, Dept. of  ECE, Tennessee Technological University, Cookeville, TN, USA. Professor Qiu was Founder-CEO and President of Wiscom Technologies, Inc., manufacturing and marketing WCDMA chipsets. Wiscom was acquired by Intel in 2003. Prior to Wiscom, he worked for GTE Labs, Inc. (now Verizon), Waltham, MA, and Bell Labs, Lucent, Whippany, NJ. He holds 5 U.S. patents (another two pending) in WCDMA. Professor Qiu has contributed to 3GPP and IEEE standards bodies, and delivered invited seminars to institutions including Princeton University and the U.S. Army Research Lab. Dr. Qiu was made an IEEE Fellow in 2014.Dr. Paul Antonik, Chief Scientist, Information Directorate, Air Force Research Laboratory, Rome, N.Y., USA. Dr. Antonik serves as the directorate's principal scientific and technical adviser and primary authority for the technical content of the science and technology portfolio, providing principal technical oversight of a broad spectrum of information technologies.

• Preface xvAcknowledgments xixSome Notation xxi1 Introduction 11.1 Big Data: Basic Concepts 11.2 Data Mining with Big Data 91.3 A Mathematical Introduction to Big Data 131.4 A Mathematical Theory of Big Data 281.5 Smart Grid 341.6 Big Data and Smart Grid 361.7 Reading Guide 37Bibliographical Remarks 39Part I Fundamentals of Big Data 412 The Mathematical Foundations of Big Data Systems 432.1 Big Data Analytics 442.2 Big Data: Sense, Collect, Store, and Analyze 452.3 Intelligent Algorithms 482.4 Signal Processing for Smart Grid 482.5 Monitoring and Optimization for Power Grids 482.6 Distributed Sensing and Measurement for Power Grids 492.7 Real-time Analysis of Streaming Data 502.8 Salient Features of Big Data 512.9 Big Data for Quantum Systems 542.10 Big Data for Financial Systems 552.11 Big Data for Atmospheric Systems 732.12 Big Data for Sensing Networks 742.13 Big Data forWireless Networks 752.14 Big Data for Transportation 78Bibliographical Remarks 783 Large Random Matrices: An Introduction 793.1 Modeling of Large Dimensional Data as Random Matrices 793.2 A Brief of Random MatrixTheory 813.3 Change Point of Views: From Vectors to Measures 853.4 The Stieltjes Transform of Measures 863.5 A Fundamental Result: The Marchenko–Pastur Equation 883.6 Linear Eigenvalue Statistics and Limit Laws 893.7 Central LimitTheorem for Linear Eigenvalue Statistics 993.8 Central LimitTheorem for Random Matrix S−1T 1013.9 Independence for Random Matrices 1033.10 Matrix-Valued Gaussian Distribution 1103.11 Matrix-ValuedWishart Distribution 1123.12 Moment Method 1123.13 Stieltjes Transform Method 1133.14 Concentration of the Spectral Measure for Large Random Matrices 1143.15 Future Directions 117Bibliographical Remarks 1174 Linear Spectral Statistics of the Sample Covariance Matrix 1214.1 Linear Spectral Statistics 1214.2 Generalized Marchenko–Pastur Distributions 1224.3 Estimation of Spectral Density Functions 1274.4 Limiting Spectral Distribution of Time Series 146Bibliographical Remarks 1545 Large Hermitian Random Matrices and Free Random Variables 1555.1 Large Economic/Financial Systems 1565.2 Matrix-Valued Probability 1575.3 Wishart-Levy Free Stable Random Matrices 1665.4 Basic Concepts for Free Random Variables 1685.5 The Analytical Spectrum of theWishart–Levy Random Matrix 1725.6 Basic Properties of the Stieltjes Transform 1765.7 Basic Theorems for the Stieltjes Transform 1795.8 Free Probability for Hermitian Random Matrices 1855.9 Random Vandermonde Matrix 1965.10 Non-Asymptotic Analysis of State Estimation 200Bibliographical Remarks 2016 Large Non-Hermitian Random Matrices and Quatartenionic Free Probability Theory 2036.1 Quatartenionic Free ProbabilityTheory 2046.2 R-diagonalMatrices 2096.3 The Sum of Non-Hermitian Random Matrices 2166.4 The Product of Non-Hermitian Random Matrices 2206.5 Singular Value Equivalent Models 2266.6 The Power of the Non-Hermitian Random Matrix 2346.7 Power Series of Large Non-Hermitian Random Matrices 2396.8 Products of Random Ginibre Matrices 2466.9 Products of Rectangular Gaussian Random Matrices 2496.10 Product of ComplexWishart Matrices 2526.11 Spectral Relations between Products and Powers 2546.12 Products of Finite-Size I.I.D. Gaussian Random Matrices 2586.13 Lyapunov Exponents for Products of Complex Gaussian Random Matrices 2606.14 Euclidean Random Matrices 2646.15 Random Matrices with Independent Entries and the Circular Law 2736.16 The Circular Law and Outliers 2756.17 Random SVD, Single Ring Law, and Outliers 2856.18 The Elliptic Law and Outliers 295Bibliographical Remarks 3057 The Mathematical Foundations of Data Collection 3077.1 Architectures and Applications for Big Data 3077.2 Covariance Matrix Estimation 3087.3 Spectral Estimators for Large Random Matrices 3127.4 Asymptotic Framework for Matrix Reconstruction 3197.5 Optimum Shrinkage 3297.6 A Shrinkage Approach to Large-Scale Covariance Matrix Estimation 3317.7 Eigenvectors of Large Sample Covariance Matrix Ensembles 3387.8 A General Class of Random Matrices 351Bibliographical Remarks 3598 Matrix Hypothesis Testing using Large RandomMatrices 3618.1 Motivating Examples 3628.2 Hypothesis Test of Two Alternative Random Matrices 3638.3 Eigenvalue Bounds for Expectation and Variance 3648.4 Concentration of Empirical Distribution Functions 3698.5 Random Quadratic Forms 3818.6 Log-Determinant of Random Matrices 3828.7 General MANOVA Matrices 3838.8 Finite Rank Perturbations of Large Random Matrices 3868.9 Hypothesis Tests for High-Dimensional Datasets 3918.9.1 Motivation for Likelihood Ratio Test (LRT) and Covariance Matrix Tests 3928.10 Roy’s Largest Root Test 4288.11 Optimal Tests of Hypotheses for Large Random Matrices 4318.12 Matrix Elliptically Contoured Distributions 4448.13 Hypothesis Testing for Matrix Elliptically Contoured Distributions 446Bibliographical Remarks 452Part II Smart Grid 4559 Applications and Requirements of Smart Grid 4579.1 History 4579.2 Concepts and Vision 4589.3 Today’s Electric Grid 4599.4 Future Smart Electrical Energy System 46410 Technical Challenges for Smart Grid 471Bibliographical Remarks 48311 Big Data for Smart Grid 48511.1 Power in Numbers: Big Data and Grid Infrastructure 48511.2 Energy’s Internet:The Convergence of Big Data and the Cloud 48611.3 Edge Analytics: Consumers, Electric Vehicles, and Distributed Generation 48611.4 CrosscuttingThemes: Big Data 48611.5 Cloud Computing for Smart Grid 48811.6 Data Storage, Data Access and Data Analysis 48811.7 The State-of-the-Art Processing Techniques of Big Data 48811.8 Big Data Meets the Smart Electrical Grid 48811.9 4Vs of Big Data: Volume, Variety, Value and Velocity 48911.10 Cloud Computing for Big Data 49011.11 Big Data for Smart Grid 49011.12 Information Platforms for Smart Grid 491Bibliographical Remarks 49112 Grid Monitoring and State Estimation 49312.1 Phase Measurement Unit 49312.2 Optimal PMU Placement 49512.3 State Estimation 49512.4 Basics of State Estimation 49512.5 Evolution of State Estimation 49612.6 Static State Estimation 49712.7 Forecasting-Aided State Estimation 50012.8 Phasor Measurement Units 50112.9 Distributed System State Estimation 50212.10 Event-Triggered Approaches to State Estimation 50212.11 Bad Data Detection 50212.12 Improved Bad Data Detection 50412.13 Cyber-Attacks 50412.14 Line Outage Detection 504Bibliographical Remarks 50413 False Data Injection Attacks against State Estimation 50513.1 State Estimation 50513.2 False Data Injection Attacks 50713.3 MMSE State Estimation and Generalized Likelihood Ratio Test 50813.4 Sparse Recovery from Nonlinear Measurements 51213.5 Real-Time Intrusion Detection 515Bibliographical Remarks 51514 Demand Response 51714.1 Why Engage Demand? 51714.2 Optimal Real-time Pricing Algorithms 52014.3 Transportation Electrification and Vehicle-to-Grid Applications 52214.4 Grid Storage 522Bibliographical Remarks 523Part III Communications and Sensing 52515 Big Data for Communications 52715.1 5G and Big Data 52715.2 5GWireless Communication Networks 52715.3 Massive Multiple Input, Multiple Output 52815.4 Free Probability for the Capacity of the Massive MIMO Channel 53715.5 Spectral Sensing for Cognitive Radio 539Bibliographical Remarks 53916 Big Data for Sensing 54116.1 Distributed Detection and Estimation 54116.2 Euclidean Random Matrix 54716.3 Decentralized Computing 548Appendix A: Some Basic Results on Free Probability 551Appendix B: Matrix-Valued Random Variables 557References 567Index 601