Theory of Ridge Regression Estimation with Applications

A. K. Md. EHSANES SALEH, PhD, is a Professor Emeritus and Distinguished Research Professor in the school of Mathematics and Statistics, Carleton University, Ottawa, Canada. MOHAMMAD ARASHI, PhD, is an Associate Professor at Shahrood University of Technology, Iran and Extraordinary Professor and C2 rated researcher at University of Pretoria, Pretoria, South Africa. B. M. GOLAM KIBRIA, PhD, is a Professor in the Department of Mathematics and Statistics at Florida International University, Miami, FL.

• List of Figures xviiList of Tables xxiPreface xxviiAbbreviations and Acronyms xxxiList of Symbols xxxiii1 Introduction to Ridge Regression 11.1 Introduction 11.1.1 Multicollinearity Problem 31.2 Ridge Regression Estimator: Ridge Notion 51.3 LSE vs. RRE 61.4 Estimation of Ridge Parameter 71.5 Preliminary Test and Stein-Type Ridge Estimators 81.6 High-Dimensional Setting 91.7 Notes and References 111.8 Organization of the Book 122 Location and Simple Linear Models 152.1 Introduction 152.2 Location Model 162.2.1 Location Model: Estimation 162.2.2 Shrinkage Estimation of Location 172.2.3 Ridge Regression–Type Estimation of Location Parameter 182.2.4 LASSO for Location Parameter 182.2.5 Bias and MSE Expression for the LASSO of Location Parameter 192.2.6 Preliminary Test Estimator, Bias, and MSE 232.2.7 Stein-Type Estimation of Location Parameter 242.2.8 Comparison of LSE, PTE, Ridge, SE, and LASSO 242.3 Simple Linear Model 262.3.1 Estimation of the Intercept and Slope Parameters 262.3.2 Test for Slope Parameter 272.3.3 PTE of the Intercept and Slope Parameters 272.3.4 Comparison of Bias and MSE Functions 292.3.5 Alternative PTE 312.3.6 Optimum Level of Significance of Preliminary Test 332.3.7 Ridge-Type Estimation of Intercept and Slope 342.3.7.1 Bias and MSE Expressions 352.3.8 LASSO Estimation of Intercept and Slope 362.4 Summary and Concluding Remarks 393 ANOVA Model 433.1 Introduction 433.2 Model, Estimation, and Tests 443.2.1 Estimation of Treatment Effects 453.2.2 Test of Significance 453.2.3 Penalty Estimators 463.2.4 Preliminary Test and Stein-Type Estimators 473.3 Bias and Weighted L2 Risks of Estimators 483.3.1 Hard Threshold Estimator (Subset Selection Rule) 483.3.2 LASSO Estimator 493.3.3 Ridge Regression Estimator 513.4 Comparison of Estimators 523.4.1 Comparison of LSE with RLSE 523.4.2 Comparison of LSE with PTE 523.4.3 Comparison of LSE with SE and PRSE 533.4.4 Comparison of LSE and RLSE with RRE 543.4.5 Comparison of RRE with PTE, SE, and PRSE 563.4.5.1 Comparison Between 𝜽n^RR (kopt) and 𝜽n^PT (𝛼) 563.4.5.2 Comparison Between 𝜽n^RR (kopt) and 𝜽n^ s 563.4.5.3 Comparison of 𝜽n^RR (kopt) with 𝜽n^S+ 573.4.6 Comparison of LASSO with LSE and RLSE 583.4.7 Comparison of LASSO with PTE, SE, and PRSE 593.4.8 Comparison of LASSO with RRE 603.5 Application 603.6 Efficiency in Terms of Unweighted L2 Risk 633.7 Summary and Concluding Remarks 723A. Appendix 744 Seemingly Unrelated Simple Linear Models 794.1 Model, Estimation, and Test of Hypothesis 794.1.1 LSE of 𝜃 and 𝛽 804.1.2 Penalty Estimation of 𝛽 and 𝜃 804.1.3 PTE and Stein-Type Estimators of 𝛽 and 𝜃 814.2 Bias and MSE Expressions of the Estimators 824.3 Comparison of Estimators 864.3.1 Comparison of LSE with RLSE 864.3.2 Comparison of LSE with PTE 864.3.3 Comparison of LSE with SE and PRSE 874.3.4 Comparison of LSE and RLSE with RRE 874.3.5 Comparison of RRE with PTE, SE, and PRSE 894.3.5.1 Comparison Between 𝜽n^RR (kopt) and 𝜽n^PT 894.3.5.2 Comparison Between 𝜽n^RR (kopt) and 𝜽n^S 894.3.5.3 Comparison of 𝜽n^RR (kopt) with 𝜽n^S+ 904.3.6 Comparison of LASSO with RRE 904.3.7 Comparison of LASSO with LSE and RLSE 924.3.8 Comparison of LASSO with PTE, SE, and PRSE 924.4 Efficiency in Terms of Unweighted L2 Risk 934.4.1 Efficiency for 𝜷 944.4.2 Efficiency for 𝜽 954.5 Summary and Concluding Remarks 965 Multiple Linear Regression Models 1095.1 Introduction 1095.2 Linear Model and the Estimators 1105.2.1 Penalty Estimators 1115.2.2 Shrinkage Estimators 1135.3 Bias and Weighted L2 Risks of Estimators 1145.3.1 Hard Threshold Estimator 1145.3.2 Modified LASSO 1165.3.3 Multivariate Normal Decision Theory and Oracles for Diagonal Linear Projection 1175.3.4 Ridge Regression Estimator 1195.3.5 Shrinkage Estimators 1195.4 Comparison of Estimators 1205.4.1 Comparison of LSE with RLSE 1205.4.2 Comparison of LSE with PTE 1215.4.3 Comparison of LSE with SE and PRSE 1215.4.4 Comparison of LSE and RLSE with RRE 1225.4.5 Comparison of RRE with PTE, SE, and PRSE 1235.4.5.1 Comparison Between 𝜽n^RR (kopt) and 𝜽n^PT(𝛼) 1235.4.5.2 Comparison Between 𝜽n^RR (kopt) and 𝜽n^S 1245.4.5.3 Comparison of 𝜽n^RR (kopt) with 𝜽n^S+ 1245.4.6 Comparison of MLASSO with LSE and RLSE 1255.4.7 Comparison of MLASSO with PTE, SE, and PRSE 1265.4.8 Comparison of MLASSO with RRE 1275.5 Efficiency in Terms of Unweighted L2 Risk 1275.6 Summary and Concluding Remarks 1296 Ridge Regression in Theory and Applications 1436.1 Multiple Linear Model Specification 1436.1.1 Estimation of Regression Parameters 1436.1.2 Test of Hypothesis for the Coefficients Vector 1456.2 Ridge Regression Estimators (RREs) 1466.3 Bias, MSE, and L2 Risk of Ridge Regression Estimator 1476.4 Determination of the Tuning Parameters 1516.5 Ridge Trace 1516.6 Degrees of Freedom of RRE 1546.7 Generalized Ridge Regression Estimators 1556.8 LASSO and Adaptive Ridge Regression Estimators 1566.9 Optimization Algorithm 1586.9.1 Prostate Cancer Data 1606.10 Estimation of Regression Parameters for Low-Dimensional Models 1616.10.1 BLUE and Ridge Regression Estimators 1616.10.2 Bias and L2-risk Expressions of Estimators 1626.10.3 Comparison of the Estimators 1656.10.4 Asymptotic Results of RRE 1666.11 Summary and Concluding Remarks 1687 Partially Linear Regression Models 1717.1 Introduction 1717.2 Partial Linear Model and Estimation 1727.3 Ridge Estimators of Regression Parameter 1747.4 Biases and L2 Risks of Shrinkage Estimators 1777.5 Numerical Analysis 1787.5.1 Example: Housing Prices Data 1827.6 High-Dimensional PLM 1887.6.1 Example: Riboflavin Data 1927.7 Summary and Concluding Remarks 1938 Logistic Regression Model 1978.1 Introduction 1978.1.1 Penalty Estimators 1998.1.2 Shrinkage Estimators 2008.1.3 Results on MLASSO 2018.1.4 Results on PTE and Stein-Type Estimators 2028.1.5 Results on Penalty Estimators 2048.2 Asymptotic Distributional L2 Risk Efficiency Expressions of the Estimators 2048.2.1 MLASSO vs. MLE 2058.2.2 MLASSO vs. RMLE 2068.2.3 Comparison of MLASSO vs. PTE 2068.2.4 PT and MLE 2078.2.5 Comparison of MLASSO vs. SE 2088.2.6 Comparison of MLASSO vs. PRSE 2088.2.7 RRE vs. MLE 2098.2.7.1 RRE vs. RMLE 2098.2.8 Comparison of RRE vs. PTE 2118.2.9 Comparison of RRE vs. SE 2118.2.10 Comparison of RRE vs. PRSE 2128.2.11 PTE vs. SE and PRSE 2128.2.12 Numerical Comparison Among the Estimators 2138.3 Summary and Concluding Remarks 2139 Regression Models with Autoregressive Errors 2219.1 Introduction 2219.1.1 Penalty Estimators 2239.1.2 Shrinkage Estimators 2249.1.2.1 Preliminary Test Estimator 2249.1.2.2 Stein-Type and Positive-Rule Stein-Type Estimators 2259.1.3 Results on Penalty Estimators 2259.1.4 Results on PTE and Stein-Type Estimators 2269.1.5 Results on Penalty Estimators 2299.2 Asymptotic Distributional L2-risk Efficiency Comparison 2309.2.1 Comparison of GLSE with RGLSE 2309.2.2 Comparison of GLSE with PTE 2319.2.3 Comparison of LSE with SE and PRSE 2319.2.4 Comparison of LSE and RLSE with RRE 2329.2.5 Comparison of RRE with PTE, SE and PRSE 2339.2.5.1 Comparison Between 𝜷n^GRR(kopt) and 𝜷n^G(PT)2339.2.5.2 Comparison Between 𝜷n^GRR(kopt) and 𝜷n^G(S) 2349.2.5.3 Comparison of 𝜷n^GRR(kopt) with 𝜷n^G(S+) 2349.2.6 Comparison of MLASSO with GLSE and RGLSE 2359.2.7 Comparison of MLASSO with PTE, SE, and PRSE 2369.2.8 Comparison of MLASSO with RRE 2369.3 Example: Sea Level Rise at KeyWest, Florida 2379.3.1 Estimation of the Model Parameters 2379.3.1.1 Testing for Multicollinearity 2379.3.1.2 Testing for Autoregressive Process 2389.3.1.3 Estimation of Ridge Parameter k 2399.3.2 Relative Efficiency 2409.3.2.1 Relative Efficiency (REff) 2409.3.2.2 Effect of Autocorrelation Coefficient 𝜙 2439.4 Summary and Concluding Remarks 24510 Rank-Based Shrinkage Estimation 25110.1 Introduction 25110.2 LinearModel and Rank Estimation 25210.2.1 Penalty R-Estimators 25610.2.2 PTREs and Stein-type R-Estimators 25810.3 Asymptotic Distributional Bias and L2 Risk of the R-Estimators 25910.3.1 HardThreshold Estimators (Subset Selection) 25910.3.2 Rank-based LASSO 26010.3.3 Multivariate Normal DecisionTheory and Oracles for Diagonal Linear Projection 26110.4 Comparison of Estimators 26210.4.1 Comparison of RE with Restricted RE 26210.4.2 Comparison of RE with PTRE 26310.4.3 Comparison of RE with SRE and PRSRE 26310.4.4 Comparison of RE and Restricted RE with RRRE 26510.4.5 Comparison of RRRE with PTRE, SRE, and PRSRE 26610.4.6 Comparison of RLASSO with RE and Restricted RE 26710.4.7 Comparison of RLASSO with PTRE, SRE, and PRSRE 26710.4.8 Comparison of Modified RLASSO with RRRE 26810.5 Summary and Concluding Remarks 26811 High-Dimensional Ridge Regression 28511.1 High-Dimensional RRE 28611.2 High-Dimensional Stein-Type RRE 28811.2.1 Numerical Results 29111.2.1.1 Example: Riboflavin Data 29111.2.1.2 Monte Carlo Simulation 29111.3 Post Selection Shrinkage 29311.3.1 Notation and Assumptions 29611.3.2 Estimation Strategy 29711.3.3 Asymptotic Distributional L2-Risks 29911.4 Summary and Concluding Remarks 30012 Applications: Neural Networks and Big Data 30312.1 Introduction 30412.2 A Simple Two-Layer Neural Network 30712.2.1 Logistic Regression Revisited 30712.2.2 Logistic Regression Loss Function with Penalty 31012.2.3 Two-Layer Logistic Regression 31112.3 Deep Neural Networks 31312.4 Application: Image Recognition 31512.4.1 Background 31512.4.2 Binary Classification 31612.4.3 Image Preparation 31812.4.4 Experimental Results 32012.5 Summary and Concluding Remarks 323References 325Index 333