Matrix Algebra Useful for Statistics

The late Shayle R. Searle, PhD, was professor emeritus of biometry at Cornell University. He was the author of Linear Models for Unbalanced Data and Linear Models and co-author of Generalized, Linear, and Mixed Models, Second Edition, Matrix Algebra for Applied Economics, and Variance Components, all published by Wiley. Dr. Searle received the Alexander von Humboldt Senior Scientist Award, and he was an honorary fellow of the Royal Society of New Zealand. André I. Khuri, PhD, is Professor Emeritus of Statistics at the University of Florida. He is the author of Advanced Calculus with Applications in Statistics, Second Edition and co-author of Statistical Tests for Mixed Linear Models, all published by Wiley. Dr. Khuri is a member of numerous academic associations, among them the American Statistical Association and the Institute of Mathematical Statistics.

• Preface xviiPreface to the First Edition xixIntroduction xxiAbout the Companion Website xxxiPart I Definitions, Basic Concepts, and Matrix Operations 11 Vector Spaces, Subspaces, and Linear Transformations 31.1 Vector Spaces 31.1.1 Euclidean Space 31.2 Base of a Vector Space 51.3 Linear Transformations 71.3.1 The Range and Null Spaces of a Linear Transformation 8Reference 9Exercises 92 Matrix Notation and Terminology 112.1 Plotting of a Matrix 142.2 Vectors and Scalars 162.3 General Notation 16Exercises 173 Determinants 213.1 Expansion by Minors 213.1.1 First- and Second-Order Determinants 223.1.2 Third-Order Determinants 233.1.3 n-Order Determinants 243.2 Formal Definition 253.3 Basic Properties 273.3.1 Determinant of a Transpose 273.3.2 Two Rows the Same 283.3.3 Cofactors 283.3.4 Adding Multiples of a Row (Column) to a Row (Column) 303.3.5 Products 303.4 Elementary Row Operations 343.4.1 Factorization 353.4.2 A Row (Column) of Zeros 363.4.3 Interchanging Rows (Columns) 363.4.4 Adding a Row to a Multiple of a Row 363.5 Examples 373.6 Diagonal Expansion 393.7 The Laplace Expansion 423.8 Sums and Differences of Determinants 443.9 A Graphical Representation of a 3 × 3 Determinant 45References 46Exercises 474 Matrix Operations 514.1 The Transpose of a Matrix 514.1.1 A Reflexive Operation 524.1.2 Vectors 524.2 Partitioned Matrices 524.2.1 Example 524.2.2 General Specification 544.2.3 Transposing a Partitioned Matrix 554.2.4 Partitioning Into Vectors 554.3 The Trace of a Matrix 554.4 Addition 564.5 Scalar Multiplication 584.6 Equality and the Null Matrix 584.7 Multiplication 594.7.1 The Inner Product of Two Vectors 594.7.2 A Matrix–Vector Product 604.7.3 A Product of Two Matrices 624.7.4 Existence of Matrix Products 654.7.5 Products With Vectors 654.7.6 Products With Scalars 684.7.7 Products With Null Matrices 684.7.8 Products With Diagonal Matrices 684.7.9 Identity Matrices 694.7.10 The Transpose of a Product 694.7.11 The Trace of a Product 704.7.12 Powers of a Matrix 714.7.13 Partitioned Matrices 724.7.14 Hadamard Products 744.8 The Laws of Algebra 744.8.1 Associative Laws 744.8.2 The Distributive Law 754.8.3 Commutative Laws 754.9 Contrasts With Scalar Algebra 764.10 Direct Sum of Matrices 774.11 Direct Product of Matrices 784.12 The Inverse of a Matrix 804.13 Rank of a Matrix—Some Preliminary Results 824.14 The Number of LIN Rows and Columns in a Matrix 844.15 Determination of the Rank of a Matrix 854.16 Rank and Inverse Matrices 874.17 Permutation Matrices 874.18 Full-Rank Factorization 894.18.1 Basic Development 894.18.2 The General Case 914.18.3 Matrices of Full Row (Column) Rank 91References 92Exercises 925 Special Matrices 975.1 Symmetric Matrices 975.1.1 Products of Symmetric Matrices 975.1.2 Properties of AA′ and A′A 985.1.3 Products of Vectors 995.1.4 Sums of Outer Products 1005.1.5 Elementary Vectors 1015.1.6 Skew-Symmetric Matrices 1015.2 Matrices Having All Elements Equal 1025.3 Idempotent Matrices 1045.4 Orthogonal Matrices 1065.4.1 Special Cases 1075.5 Parameterization of Orthogonal Matrices 1095.6 Quadratic Forms 1105.7 Positive Definite Matrices 113References 114Exercises 1146 Eigenvalues and Eigenvectors 1196.1 Derivation of Eigenvalues 1196.1.1 Plotting Eigenvalues 1216.2 Elementary Properties of Eigenvalues 1226.2.1 Eigenvalues of Powers of a Matrix 1226.2.2 Eigenvalues of a Scalar-by-Matrix Product 1236.2.3 Eigenvalues of Polynomials 1236.2.4 The Sum and Product of Eigenvalues 1246.3 Calculating Eigenvectors 1256.3.1 Simple Roots 1256.3.2 Multiple Roots 1266.4 The Similar Canonical Form 1286.4.1 Derivation 1286.4.2 Uses 1306.5 Symmetric Matrices 1316.5.1 Eigenvalues All Real 1326.5.2 Symmetric Matrices Are Diagonable 1326.5.3 Eigenvectors Are Orthogonal 1326.5.4 Rank Equals Number of Nonzero Eigenvalues for a Symmetric Matrix 1356.6 Eigenvalues of Orthogonal and Idempotent Matrices 1356.6.1 Eigenvalues of Symmetric Positive Definite and Positive Semidefinite Matrices 1366.7 Eigenvalues of Direct Products and Direct Sums of Matrices 1386.8 Nonzero Eigenvalues of AB and BA 140References 141Exercises 1417 Diagonalization of Matrices 1457.1 Proving the Diagonability Theorem 1457.1.1 The Number of Nonzero Eigenvalues Never Exceeds Rank 1457.1.2 A Lower Bound on r (A − λkI) 1467.1.3 Proof of the Diagonability Theorem 1477.1.4 All Symmetric Matrices Are Diagonable 1477.2 Other Results for Symmetric Matrices 1487.2.1 Non-Negative Definite (n.n.d.) 1487.2.2 Simultaneous Diagonalization of Two Symmetric Matrices 1497.3 The Cayley–Hamilton Theorem 1527.4 The Singular-Value Decomposition 153References 157Exercises 1578 Generalized Inverses 1598.1 The Moore–Penrose Inverse 1598.2 Generalized Inverses 1608.2.1 Derivation Using the Singular-Value Decomposition 1618.2.2 Derivation Based on Knowing the Rank 1628.3 Other Names and Symbols 1648.4 Symmetric Matrices 1658.4.1 A General Algorithm 1668.4.2 The Matrix X′X 166References 167Exercises 1679 Matrix Calculus 1719.1 Matrix Functions 1719.1.1 Function of Matrices 1719.1.2 Matrices of Functions 1749.2 Iterative Solution of Nonlinear Equations 1749.3 Vectors of Differential Operators 1759.3.1 Scalars 1759.3.2 Vectors 1769.3.3 Quadratic Forms 1779.4 Vec and Vech Operators 1799.4.1 Definitions 1799.4.2 Properties of Vec 1809.4.3 Vec-Permutation Matrices 1809.4.4 Relationships Between Vec and Vech 1819.5 Other Calculus Results 1819.5.1 Differentiating Inverses 1819.5.2 Differentiating Traces 1829.5.3 Derivative of a Matrix with Respect to Another Matrix 1829.5.4 Differentiating Determinants 1839.5.5 Jacobians 1859.5.6 Aitken’s Integral 1879.5.7 Hessians 1889.6 Matrices with Elements That Are Complex Numbers 1889.7 Matrix Inequalities 189References 193Exercises 194Part II Applications of Matrices in Statistics 19910 Multivariate Distributions and Quadratic Forms 20110.1 Variance-Covariance Matrices 20210.2 Correlation Matrices 20310.3 Matrices of Sums of Squares and Cross-Products 20410.3.1 Data Matrices 20410.3.2 Uncorrected Sums of Squares and Products 20410.3.3 Means, and the Centering Matrix 20510.3.4 Corrected Sums of Squares and Products 20510.4 The Multivariate Normal Distribution 20710.5 Quadratic Forms and χ2-Distributions 20810.5.1 Distribution of Quadratic Forms 20910.5.2 Independence of Quadratic Forms 21010.5.3 Independence and Chi-Squaredness of Several Quadratic Forms 21110.5.4 The Moment and Cumulant Generating Functions for a Quadratic Form 21110.6 Computing the Cumulative Distribution Function of a Quadratic Form 21310.6.1 Ratios of Quadratic Forms 214References 215Exercises 21511 Matrix Algebra of Full-Rank Linear Models 21911.1 Estimation of β by the Method of Least Squares 22011.1.1 Estimating the Mean Response and the Prediction Equation 22311.1.2 Partitioning of Total Variation Corrected for the Mean 22511.2 Statistical Properties of the Least-Squares Estimator 22611.2.1 Unbiasedness and Variances 22611.2.2 Estimating the Error Variance 22711.3 Multiple Correlation Coefficient 22911.4 Statistical Properties under the Normality Assumption 23111.5 Analysis of Variance 23311.6 The Gauss–Markov Theorem 23411.6.1 Generalized Least-Squares Estimation 23711.7 Testing Linear Hypotheses 23711.7.1 The Use of the Likelihood Ratio Principle in Hypothesis Testing 23911.7.2 Confidence Regions and Confidence Intervals 24111.8 Fitting Subsets of the x-Variables 24611.9 The Use of the R(.|.) Notation in Hypothesis Testing 247References 249Exercises 24912 Less-Than-Full-Rank Linear Models 25312.1 General Description 25312.2 The Normal Equations 25612.2.1 A General Form 25612.2.2 Many Solutions 25712.3 Solving the Normal Equations 25712.3.1 Generalized Inverses of X′X 25812.3.2 Solutions 25812.4 Expected Values and Variances 25912.5 Predicted y-Values 26012.6 Estimating the Error Variance 26112.6.1 Error Sum of Squares 26112.6.2 Expected Value 26212.6.3 Estimation 26212.7 Partitioning the Total Sum of Squares 26212.8 Analysis of Variance 26312.9 The R(⋅|⋅) Notation  26512.10 Estimable Linear Functions 26612.10.1 Properties of Estimable Functions 26712.10.2 Testable Hypotheses 26812.10.3 Development of a Test Statistic for H0 26912.11 Confidence Intervals 27212.12 Some Particular Models 27212.12.1 The One-Way Classification 27212.12.2 Two-Way Classification, No Interactions, Balanced Data 27312.12.3 Two-Way Classification, No Interactions, Unbalanced Data 27612.13 The R(⋅|⋅) Notation (Continued)  27712.14 Reparameterization to a Full-Rank Model 281References 282Exercises 28213 Analysis of Balanced Linear Models Using Direct Products of Matrices 28713.1 General Notation for Balanced Linear Models 28913.2 Properties Associated with Balanced Linear Models 29313.3 Analysis of Balanced Linear Models 29813.3.1 Distributional Properties of Sums of Squares 29813.3.2 Estimates of Estimable Linear Functions of the Fixed Effects 301References 307Exercises 30814 Multiresponse Models 31314.1 Multiresponse Estimation of Parameters 31414.2 Linear Multiresponse Models 31614.3 Lack of Fit of a Linear Multiresponse Model 31814.3.1 The Multivariate Lack of Fit Test 318References 323Exercises 324Part III Matrix Computations and Related Software 32715 SAS/IML 32915.1 Getting Started 32915.2 Defining a Matrix 32915.3 Creating a Matrix 33015.4 Matrix Operations 33115.5 Explanations of SAS Statements Used Earlier in the Text 354References 357Exercises 35816 Use of MATLAB in Matrix Computations 36316.1 Arithmetic Operators 36316.2 Mathematical Functions 36416.3 Construction of Matrices 36516.3.1 Submatrices 36516.4 Two- and Three-Dimensional Plots 37116.4.1 Three-Dimensional Plots 374References 378Exercises 37917 Use of R in Matrix Computations 38317.1 Two- and Three-Dimensional Plots 39617.1.1 Two-Dimensional Plots 39717.1.2 Three-Dimensional Plots 404References 408Exercises 408Appendix 413Index 475

"Matrix Algebra Useful for Statistics, Second Edition is an ideal textbook for advanced undergraduate and first-year graduate level courses in statistics and other related disciplines. The book is also appropriate as a reference for independent readers who use statistics and wish to improve their knowledge of matrix algebra." Mathematical Reviews, Sept 2017