Resve Saleh, (PhD, UC Berkeley) is a Professor Emeritus at the University of British Columbia. He worked for a decade as a professor at the University of Illinois and as a visiting professor at Stanford University. He was Founder and Chairman of Simplex Solutions, Inc., which went public in 2001. He is an IEEE Fellow and Fellow of the Canadian Academy of Engineering. Sohaib Majzoub, (PhD, University of British Columbia) is an Associate Professor at the University of Sharjah, UAE. He also taught at the American University in Dubai, UAE and at King Saud University, KSA, and a visiting professor at Delft Technical University in The Netherlands. He is a Senior Member of the IEEE. A. K. MD. Ehsanes Saleh, (PhD, University of Western Ontario) is a Professor Emeritus and Distinguished Professor in the School of Mathematics and Statistics, Carleton University, Ottawa, Canada. He also taught as Simon Fraser University, the University of Toronto, and Stanford University. He is a Fellow of IMS, ASA and an Honorary Member of SSC, Canada.

• Preface xvAbout the Companion Website xix1 Introduction 11.1 Defining Outliers 21.2 Overview of the Book 31.3 What Is Robust Machine Learning? 31.3.1 Machine Learning Basics 41.3.2 Effect of Outliers 61.3.3 What Is Robust Data Science? 71.3.4 Noise in Datasets 71.3.5 Training and Testing Flows 81.4 Robustness of the Median 91.4.1 Mean vs. Median 91.4.2 Effect on Standard Deviation 101.5 l 1 and l 2 Norms 111.6 Review of Gaussian Distribution 121.7 Unsupervised Learning Case Study 131.7.1 Clustering Example 141.7.2 Clustering Problem Specification 141.8 Creating Synthetic Data for Clustering 161.8.1 One-Dimensional Datasets 161.8.2 Multidimensional Datasets 171.9 Clustering Algorithms 191.9.1 k-Means Clustering 191.9.2 k-Medians Clustering 211.10 Importance of Robust Clustering 221.10.1 Clustering with No Outliers 221.10.2 Clustering with Outliers 231.10.3 Detection and Removal of Outliers 251.11 Summary 27Problems 28References 342 Robust Linear Regression 352.1 Introduction 352.2 Supervised Learning 352.3 Linear Regression 362.4 Importance of Residuals 382.4.1 Defining Errors and Residuals 382.4.2 Residuals in Loss Functions 392.4.3 Distribution of Residuals 402.5 Estimation Background 422.5.1 Linear Models 422.5.2 Desirable Properties of Estimators 432.5.3 Maximum-Likelihood Estimation 442.5.4 Gradient Descent 472.6 M-Estimation 492.7 Least Squares Estimation (LSE) 522.8 Least Absolute Deviation (LAD) 542.9 Comparison of LSE and LAD 552.9.1 Simple Linear Model 552.9.2 Location Problem 562.10 Huber’s Method 582.10.1 Huber Loss Function 582.10.2 Comparison with LSE and LAD 632.11 Summary 64Problems 64References 673 The Log-Cosh Loss Function 693.1 Introduction 693.2 An Intuitive View of Log-Cosh 693.3 Hyperbolic Functions 713.4 M-Estimation 713.4.1 Asymptotic Behavior 723.4.2 Linear Regression Using Log-Cosh 743.5 Deriving the Distribution for Log-Cosh 753.6 Standard Errors for Robust Estimators 793.6.1 Example: Swiss Fertility Dataset 813.6.2 Example: Boston Housing Dataset 823.7 Statistical Properties of Log-Cosh Loss 833.7.1 Maximum-Likelihood Estimation 833.8 A General Log-Cosh Loss Function 843.9 Summary 88Problems 88References 934 Outlier Detection, Metrics, and Standardization 954.1 Introduction 954.2 Effect of Outliers 954.3 Outlier Diagnosis 974.3.1 Boxplots 984.3.2 Histogram Plots 1004.3.3 Exploratory Data Analysis 1014.4 Outlier Detection 1024.4.1 3-Sigma Edit Rule 1024.4.2 4.5-MAD Edit Rule 1044.4.3 1.5-IQR Edit Rule 1054.5 Outlier Removal 1054.5.1 Trimming Methods 1054.5.2 Winsorization 1054.5.3 Anomaly Detection Method 1064.6 Regression-Based Outlier Detection 1074.6.1 LS vs. LC Residuals 1084.6.2 Comparison of Detection Methods 1094.6.3 Ordered Absolute Residuals (OARs) 1104.6.4 Quantile–Quantile Plot 1114.6.5 Quad-Plots for Outlier Diagnosis 1134.7 Regression-Based Outlier Removal 1144.7.1 Iterative Boxplot Method 1144.8 Regression Metrics with Outliers 1164.8.1 Mean Square Error (MSE) 1174.8.2 Median Absolute Error (MAE) 1184.8.3 MSE vs. MAE on Realistic Data 1194.8.4 Selecting Hyperparameters for Robust Regression 1204.9 Dataset Standardization 1214.9.1 Robust Standardization 1224.10 Summary 126Problems 126References 1315 Robustness of Penalty Estimators 1335.1 Introduction 1335.2 Penalty Functions 1335.2.1 Multicollinearity 1335.2.2 Penalized Loss Functions 1355.3 Ridge Penalty 1365.4 LASSO Penalty 1375.5 Effect of Penalty Functions 1385.6 Penalty Functions with Outliers 1395.7 Ridge Traces 1425.8 Elastic Net (Enet) Penalty 1435.9 Adaptive LASSO (aLASSO) Penalty 1455.10 Penalty Effects on Variance and Bias 1465.10.1 Effect on Variance 1465.10.2 Geometric Interpretation of Bias 1485.11 Variable Importance 1515.11.1 The t-Statistic 1515.11.2 LASSO and aLASSO Traces 1535.12 Summary 155Problems 156References 1596 Robust Regularized Models 1616.1 Introduction 1616.2 Overfitting and Underfitting 1616.3 The Bias–Variance Trade-Off 1626.4 Regularization with Ridge 1646.4.1 Selection of Hyperparameter λ 1656.4.2 Example: Diabetes Dataset 1676.5 Generalization using Robust Estimators 1696.5.1 Training and Test Sets 1696.5.2 k-Fold Cross-validation 1716.6 Robust Generalization and Regularization 1736.6.1 Regularization with LC-Ridge 1746.7 Model Complexity 1756.7.1 Variable Selection Using LS-LASSO 1766.7.2 Variable Ordering Using LC-aLASSO 1766.7.3 Building a Compact Model 1796.8 Summary 182Problems 182References 1867 Quantile Regression Using Log-Cosh 1877.1 Introduction 1877.2 Understanding Quantile Regression 1887.3 The Crossing Problem 1897.4 Standard Quantile Loss Function 1907.5 Smooth Regression Quantiles (SMRQ) 1927.6 Evaluation of Quantile Methods 1957.6.1 Qualitative Assessment 1967.6.2 Quantitative Assessment 1987.7 Selection of Robustness Coefficient 2007.8 Maximum-Likelihood Procedure for SMRQ 2027.9 Standard Error Computation 2047.10 Summary 206Problems 207References 2098 Robust Binary Classification 2118.1 Introduction 2118.2 Binary Classification Problem 2128.2.1 Why Linear Regression Fails 2128.2.2 Outliers in Binary Classification 2138.3 The Cross-Entropy (CE) Loss 2158.3.1 Deriving the Cross-Entropy Loss 2168.3.2 Understanding Logistic Regression 2188.3.3 Gradient Descent 2218.4 The Log-Cosh (LC) Loss Function 2218.4.1 General Formulation 2238.5 Algorithms for Logistic Regression 2248.6 Example: Motor Trend Cars 2268.7 Regularization of Logistic Regression 2278.7.1 Overfitting and Underfitting 2288.7.2 k-Fold Cross-Validation 2298.7.3 Penalty Functions 2298.7.4 Effect of Outliers 2308.8 Example: Circular Dataset 2318.9 Outlier Detection 2348.10 Robustness of Binary Classifiers 2358.10.1 Support Vector Classifier (SVC) 2358.10.2 Support Vector Machines (SVMs) 2388.10.3 k-Nearest Neighbors (k-NN) 2418.10.4 Decision Trees and Random Forest 2438.11 Summary 244Problems 244Reference 2499 Neural Networks Using Log-Cosh 2519.1 Introduction 2519.2 A Brief History of Neural Networks 2519.3 Defining Neural Networks 2529.3.1 Basic Computational Unit 2539.3.2 Four-Layer Neural Network 2549.3.3 Activation Functions 2559.4 Training of Neural Networks 2579.5 Forward and Backward Propagation 2589.5.1 Forward Propagation 2599.5.2 Backward Propagation 2609.5.3 Log-Cosh Gradients 2639.6 Cross-entropy and Log-Cosh Algorithms 2649.7 Example: Circular Dataset 2669.8 Classification Metrics and Outliers 2699.8.1 Precision, Recall, F 1 Score 2699.8.2 Receiver Operating Characteristics (ROCs) 2719.9 Summary 273Problems 273References 28010 Multi-class Classification and Adam Optimization 28110.1 Introduction 28110.2 Multi-class Classification 28110.2.1 Multi-class Loss Functions 28210.2.2 Softmax Activation Function 28410.3 Example: MNIST Dataset 28810.3.1 Neural Network Architecture 28910.3.2 Comparing Cross-Entropy with Log-Cosh Losses 28910.3.3 Outliers in MNIST 29110.4 Optimization of Neural Networks 29110.4.1 Momentum 29310.4.2 rmsprop Approach 29410.4.3 Optimizer Warm-Up Phase 29510.4.4 Adam Optimizer 29610.5 Summary 297Problems 297References 30211 Anomaly Detection and Evaluation Metrics 30311.1 Introduction 30311.2 Anomaly Detection Methods 30311.2.1 k-Nearest Neighbors 30411.2.2 Dbscan 30811.2.3 Isolation Forest 31111.3 Anomaly Detection Using MADmax 31611.3.1 Robust Standardization 31711.3.2 k-Medians Clustering 31711.3.3 Selecting MADmax 31911.3.4 k-Nearest Neighbors (k-NN) 31911.3.5 k-Nearest Medians (k-NM) 32011.4 Qualitative Evaluation Methods 32311.5 Quantitative Evaluation Methods 32611.6 Summary 330Problems 330Reference 33612 Case Studies in Data Science 33712.1 Introduction 33712.2 Example: Boston Housing Dataset 33712.2.1 Exploratory Data Analysis 33812.2.2 Neural Network Architecture 33912.2.3 Comparison of LSNN and LCNN 34212.2.4 Predicting Housing Prices 34412.2.5 RMSE vs. MAE 34412.2.6 Correlation Coefficients 34512.3 Example: Titanic Dataset 34612.3.1 Exploratory Data Analysis 34612.3.2 LCLR vs. CELR 35112.3.3 Outlier Detection and Removal 35312.3.4 Robustness Coefficient for Log-Cosh 35512.3.5 The Implications of Robustness 35612.3.6 Ridge and aLASSO 35712.4 Application to Explainable Artificial Intelligence (XAI) 35912.4.1 Case Study: Logistic Regression 36012.4.2 Case Study: Neural Networks 36512.5 Time Series Example: Climate Change 36612.5.1 Autoregressive Model 36712.5.2 Forecasting Using AR(p) 36912.5.3 Stationary Time Series 37112.5.4 Moving Average 37412.5.5 Finding Outliers in Time Series 37512.6 Summary and Conclusions 376Problems 376References 382Index 383