Rank-Based Methods for Shrinkage and Selection

A. K. Md. Ehsanes Saleh, PhD, is a Professor Emeritus and Distinguished Professor in the School of Mathematics and Statistics, Carleton University, Ottawa, Canada. He is Fellow of IMS, ASA and Honorary member of SSC, Canada.Mohammad Arashi, PhD, is an Associate Professor at Ferdowsi University of Mashhad in Iran and Extraordinary Professor and C2 rated researcher at University of Pretoria, Pretoria, South Africa. He is an elected member of ISI.Resve A. Saleh, M.Sc, PhD (Berkeley), is a Professor Emeritus in the Department of ECE at the University of British Columbia, Vancouver, Canada, and formerly with University of Illinois and Stanford University. He is the author of 4 books and Fellow of the IEEE.Mina Norouzirad, PhD, is a post-doctoral researcher at the Center for Mathematics and Applications (CMA) of Nova University of Lisbon, Portugal.

• 1 Introduction to Rank-based Regression 11.1 Introduction 11.2 Robustness of the Median 11.2.1 Mean vs. Median 11.2.2 Breakdown Point 41.2.3 Order and Rank Statistics 51.3 Simple Linear Regression 61.3.1 Least Squares Estimator (LSE) 61.3.2 Theil’s Estimator 71.3.3 Belgium Telephone Data Set 71.3.4 Estimation and Standard Error Comparison 91.4 Outliers and their Detection 111.4.1 Outlier Detection 121.5 Motivation for Rank-based Methods 131.5.1 Effect of a Single Outlier 131.5.2 Using Rank for the Location Model 161.5.3 Using Rank for the Slope 191.6 The Rank Dispersion Function 201.6.1 Ranking and Scoring Details 231.6.2 Detailed Procedure for R-estimation 251.7 Shrinkage Estimation and Subset Selection 301.7.1 Multiple Linear Regression using Rank 301.7.2 Penalty Functions 321.7.3 Shrinkage Estimation 341.7.4 Subset Selection 361.7.5 Blended Approaches 391.8 Summary 391.9 Problems 412 Characteristics of Rank-based Penalty Estimators 472.1 Introduction 472.2 Motivation for Penalty Estimators 472.3 Multivariate Linear Regression 492.3.1 Multivariate Least Squares Estimation 492.3.2 Multivariate R-estimation 512.3.3 Multicollinearity 512.4 Ridge Regression 532.4.1 Ridge Applied to Least Squares Estimation 532.4.2 Ridge Applied to Rank Estimation 552.5 Example: Swiss Fertility Data Set 562.5.1 Estimation and Standard Errors 592.5.2 Parameter Variance using Bootstrap 602.5.3 Reducing Variance using Ridge 612.5.4 Ridge Traces 622.6 Selection of Ridge Parameter 𝜆2 652.6.1 Quadratic Risk 652.6.2 K-fold Cross-validation Scheme 682.7 LASSO and aLASSO 712.7.1 Subset Selection 712.7.2 Least Squares with LASSO 712.7.3 The Adaptive LASSO and its Geometric Interpretation 732.7.4 R-estimation with LASSO and aLASSO 772.7.5 Oracle Properties 782.8 Elastic Net (Enet) 822.8.1 Naive Enet 822.8.2 Standard Enet 832.8.3 Enet in Machine Learning 842.9 Example: Diabetes Data Set 852.9.1 Model Building with R-aEnet 852.9.2 MSE vs. MAE 882.9.3 Model Building with LS-Enet 912.10 Summary 942.11 Problems 953 Location and Simple Linear Models 1013.1 Introduction 1013.2 Location Estimators and Testing 1043.2.1 Unrestricted R-estimator of 𝜃 1043.2.2 Restricted R-estimator of 𝜃 1073.3 Shrinkage R-estimators of Location 1083.3.1 Overview of Shrinkage R-estimators of 𝜃 1083.3.2 Derivation of the Ridge-type R-estimator 1133.3.3 Derivation of the LASSO-type R-estimator 1143.3.4 General Shrinkage R-estimators of 𝜃 1143.4 Ridge-type R-estimator of 𝜃 1173.5 Preliminary Test R-estimator of 𝜃 1183.5.1 Optimum Level of Significance of PTRE 1213.6 Saleh-type R-estimators 1223.6.1 Hard-Threshold R-estimator of 𝜃 1223.6.2 Saleh-type R-estimator of 𝜃 1233.6.3 Positive-rule Saleh-type (LASSO-type) R-estimator of 𝜃 1253.6.4 Elastic Net-type R-estimator of 𝜃 1273.7 Comparative Study of the R-estimators of Location 1293.8 Simple Linear Model 1323.8.1 Restricted R-estimator of Slope 1343.8.2 Shrinkage R-estimator of Slope 1353.8.3 Ridge-type R-estimation of Slope 1353.8.4 Hard-Threshold R-estimator of Slope 1363.8.5 Saleh-type R-estimator of Slope 1373.8.6 Positive-rule Saleh-type (LASSO-type) R-estimator of Slope 1383.8.7 The Adaptive LASSO (aLASSO-type) R-estimator 1383.8.8 nEnet-type R-estimator of Slope 1393.8.9 Comparative Study of R-estimators of Slope 1403.9 Summary 1413.10 Problems 1424 Analysis of Variance (ANOVA) 1494.1 Introduction 1494.2 Model, Estimation and Tests 1494.3 Overview of Multiple Location Models 1504.3.1 Example: Corn Fertilizers 1514.3.2 One-way ANOVA 1514.3.3 Effect of Variance on Shrinkage Estimators 1534.3.4 Shrinkage Estimators for Multiple Location 1564.4 Unrestricted R-estimator 1584.5 Test of Significance 1614.6 Restricted R-estimator 1624.7 Shrinkage Estimators 1634.7.1 Preliminary Test R-estimator 1634.7.2 The Stein–Saleh-type R-estimator 1644.7.3 The Positive-rule Stein–Saleh-type R-estimator 1654.7.4 The Ridge-type R-estimator 1674.8 Subset Selection Penalty R-estimators 1694.8.1 Preliminary Test Subset Selector R-estimator 1694.8.2 Saleh-type R-estimator 1704.8.3 Positive-rule Saleh Subset Selector (PRSS) 1714.8.4 The Adaptive LASSO (aLASSO) 1734.8.5 Elastic-net-type R-estimator 1774.9 Comparison of the R-estimators 1784.9.1 Comparison of URE and RRE 1794.9.2 Comparison of URE and Stein–Saleh-type R-estimators 1794.9.3 Comparison of URE and Ridge-type R-estimators 1794.9.4 Comparison of URE and PTSSRE 1804.9.5 Comparison of LASSO-type and Ridge-type R-estimators 1804.9.6 Comparison of URE, RRE and LASSO 1814.9.7 Comparison of LASSO with PTRE 1814.9.8 Comparison of LASSO with SSRE 1824.9.9 Comparison of LASSO with PRSSRE 1824.9.10 Comparison of nEnetRE with URE 1834.9.11 Comparison of nEnetRE with RRE 1834.9.12 Comparison of nEnetRE with HTRE 1834.9.13 Comparison of nEnetRE with SSRE 1844.9.14 Comparison of Ridge-type vs. nEnetRE 1844.10 Summary 1854.11 Problems 1855 Seemingly Unrelated Simple Linear Models 1915.1 Introduction 1915.1.1 Problem Formulation 1935.2 Signed and Signed Rank Estimators of Parameters 1945.2.1 General Shrinkage R-estimator of 𝛽 1985.2.2 Ridge-type R-estimator of 𝛽 1995.2.3 Preliminary Test R-estimator of 𝛽 2015.3 Stein–Saleh-type R-estimator of 𝛽 2025.3.1 Positive-rule Stein–Saleh R-estimators of 𝛽 2025.4 Saleh-type R-estimator of 𝛽 2035.4.1 LASSO-type R-estimator of the 𝛽 2055.5 Elastic-net-type R-estimators 206  5.6 R-estimator of Intercept When Slope Has Sparse Subset 2075.6.1 General Shrinkage R-estimator of Intercept 2075.6.2 Ridge-type R-estimator of 𝜃 2095.6.3 Preliminary Test R-estimators of 𝜃 2095.7 Stein–Saleh-type R-estimator of 𝜃 2105.7.1 Positive-rule Stein–Saleh-type R-estimator of 𝜃 2115.7.2 LASSO-type R-estimator of 𝜃 2135.8 Summary 2135.8.1 Problems 2146 Multiple Linear Regression Models 2156.1 Introduction 2156.2 Multiple Linear Model and R-estimation 2156.3 Model Sparsity and Detection 2186.4 General Shrinkage R-estimator of 𝛽 2216.4.1 Preliminary Test R-estimator 2226.4.2 Stein–Saleh-type R-estimator 2246.4.3 Positive-rule Stein–Saleh-type R-estimator 2256.5 Subset Selectors 2266.5.1 Preliminary Test Subset Selector R-estimator 2266.5.2 Stein–Saleh-type R-estimator 2286.5.3 Positive-rule Stein–Saleh-type R-estimator (LASSO-type) 2296.5.4 Ridge-type Subset Selector 2316.5.5 Elastic Net-type R-estimator 2316.6 Adaptive LASSO 2326.6.1 Introduction 2326.6.2 Asymptotics for LASSO-type R-estimator 2336.6.3 Oracle Property of aLASSO 2356.7 Summary 2386.8 Problems 2397 Partially Linear Multiple Regression Model 2417.1 Introduction 2417.2 Rank Estimation in the PLM 2427.2.1 Penalty R-estimators 2467.2.2 Preliminary Test and Stein–Saleh-type R-estimator 2487.3 ADB and ADL2-risk 2497.4 ADL2-risk Comparisons 2537.5 Summary: L2-risk Efficiencies 2607.6 Problems 2628 Liu Regression Models 2638.1 Introduction 2638.2 Linear Unified (Liu) Estimator 2638.2.1 Liu-type R-estimator 2668.3 Shrinkage Liu-type R-estimators 2688.4 Asymptotic Distributional Risk 2698.5 Asymptotic Distributional Risk Comparisons 2718.5.1 Comparison of SSLRE and PTLRE 2728.5.2 Comparison of PRSLRE and PTLRE 2748.5.3 Comparison of PRLRE and SSLRE 2768.5.4 Comparison of Liu-Type Rank Estimators With Counterparts 2778.6 Estimation of d 2798.7 Diabetes Data Analysis 2808.7.1 Penalty Estimators 2818.7.2 Performance Analysis 2848.8 Summary 2888.9 Problems 2889 Autoregressive Models 2919.1 Introduction 2919.2 R-estimation of 𝜌 for the AR(𝑝)-Model 2929.3 LASSO, Ridge, Preliminary Test and Stein–Saleh-type R-estimators 2949.4 Asymptotic Distributional L2-risk 2969.5 Asymptotic Distributional L2-risk Analysis 2999.5.1 Comparison of Unrestricted vs. Restricted R-estimators 3009.5.2 Comparison of Unrestricted vs. Preliminary Test R-estimator 3009.5.3 Comparison of Unrestricted vs. Stein–Saleh-type R-estimators 3009.5.4 Comparison of the Preliminary Test vs. Stein–Saleh-type R-estimators 3029.6 Summary 3039.7 Problems 30410 High-Dimensional Models 30710.1 Introduction 30710.2 Identifiability of 𝛽∗ and Projection 30910.3 Parsimonious Model Selection 30910.4 Some Notation and Separation 31110.4.1 Special Matrices 31110.4.2 Steps Towards Estimators 31210.4.3 Post-selection Ridge Estimation of 𝛽∗ 𝒮1 and 𝜷∗ 𝒮2 31210.4.4 Post-selection Ridge R-estimators for 𝛽∗ 𝒮1 and 𝜷∗ 𝒮2 31310.5 Post-selection Shrinkage R-estimators 31510.6 Asymptotic Properties of the Ridge R-estimators 31610.7 Asymptotic Distributional L2-Risk Properties 32110.8 Asymptotic Distributional Risk Efficiency 32410.9 Summary 32610.10 Problems 32711 Rank-based Logistic Regression 32911.1 Introduction 32911.2 Data Science and Machine Learning 32911.2.1 What is Robust Data Science? 32911.2.2 What is Robust Machine Learning? 33211.3 Logistic Regression 33311.3.1 Log-likelihood Setup 33411.3.2 Motivation for Rank-based Logistic Methods 33811.3.3 Nonlinear Dispersion Function 34111.4 Application to Machine Learning 34211.4.1 Example: Motor Trend Cars 34411.5 Penalized Logistic Regression 34711.5.1 Log-likelihood Expressions 34711.5.2 Rank-based Expressions 34811.5.3 Support Vector Machines 34911.5.4 Example: Circular Data 35311.6 Example: Titanic Data Set 35911.6.1 Exploratory Data Analysis 35911.6.2 RLR vs. LLR vs. SVM 36511.6.3 Shrinkage and Selection 36711.7 Summary 37011.8 Problems 37112 Rank-based Neural Networks 37712.1 Introduction 37712.2 Set-up for Neural Networks 37912.3 Implementing Neural Networks 38112.3.1 Basic Computational Unit 38212.3.2 Activation Functions 38212.3.3 Four-layer Neural Network 38412.4 Gradient Descent with Momentum 38612.4.1 Gradient Descent 38612.4.2 Momentum 38812.5 Back Propagation Example 38912.5.1 Forward Propagation 39012.5.2 Back Propagation 39212.5.3 Dispersion Function Gradients 39412.5.4 RNN Algorithm 39512.6 Accuracy Metrics 39612.7 Example: Circular Data Set 40012.8 Image Recognition: Cats vs. Dogs 40512.8.1 Binary Image Classification 40612.8.2 Image Preparation 40612.8.3 Over-fitting and Under-fitting 40912.8.4 Comparison of LNN vs. RNN 41012.9 Image Recognition: MNIST Data Set 41412.10 Summary 42112.11 Problems 421Bibliography 433Author Index 443Subject Index445