Machine Learning for Business Analytics

Galit Shmueli, PhD, is Distinguished Professor and Institute Director at National Tsing Hua University’s Institute of Service Science. She has designed and instructed business analytics courses since 2004 at University of Maryland, Statistics.com, The Indian School of Business, and National Tsing Hua University, Taiwan. Peter C. Bruce, is Founder of the Institute for Statistics Education at Statistics.com, and Chief Learning Officer at Elder Research, Inc. Peter Gedeck, PhD, is Senior Data Scientist at Collaborative Drug Discovery and teaches at statistics.com and the UVA School of Data Science. His specialty is the development of machine learning algorithms to predict biological and physicochemical properties of drug candidates. Inbal Yahav, PhD, is a Senior Lecturer in The Coller School of Management at Tel Aviv University, Israel. Her work focuses on the development and adaptation of statistical models for use by researchers in the field of information systems. Nitin R. Patel, PhD, is Co-founder and Lead Researcher at Cytel Inc. He was also a Co-founder of Tata Consultancy Services. A Fellow of the American Statistical Association, Dr. Patel has served as a Visiting Professor at the Massachusetts Institute of Technology and at Harvard University, USA.

• Foreword by Ravi Bapna xixForeword by Gareth James xxiPreface to the Second R Edition xxiiiAcknowledgments xxviPart I Preliminaries Chapter 1 Introduction 31.1 What Is Business Analytics? 31.2 What Is Machine Learning? 51.3 Machine Learning, AI, and Related Terms 51.4 Big Data 71.5 Data Science 81.6 Why Are There So Many Different Methods? 81.7 Terminology and Notation 91.8 Road Maps to This Book 11Order of Topics 13Chapter 2 Overview of the Machine Learning Process 172.1 Introduction 172.2 Core Ideas in Machine Learning 18Classification 18Prediction 18Association Rules and Recommendation Systems 18Predictive Analytics 19Data Reduction and Dimension Reduction 19Data Exploration and Visualization 19Supervised and Unsupervised Learning 202.3 The Steps in a Machine Learning Project 212.4 Preliminary Steps 23Organization of Data 23Predicting Home Values in the West Roxbury Neighborhood 23Loading and Looking at the Data in R 24Sampling from a Database 26Oversampling Rare Events in Classification Tasks 27Preprocessing and Cleaning the Data 282.5 Predictive Power and Overfitting 35Overfitting 36Creating and Using Data Partitions 382.6 Building a Predictive Model 41Modeling Process 412.7 Using R for Machine Learning on a Local Machine 462.8 Automating Machine Learning Solutions 47Predicting Power Generator Failure 48Uber’s Michelangelo 502.9 Ethical Practice in Machine Learning 52Machine Learning Software: The State of the Market (by Herb Edelstein) 53Problems 57Part II Data Exploration and Dimension ReductionChapter 3 Data Visualization 633.1 Uses of Data Visualization 63Base R or ggplot? 653.2 Data Examples 65Example 1: Boston Housing Data 65Example 2: Ridership on Amtrak Trains 673.3 Basic Charts: Bar Charts, Line Charts, and Scatter Plots 67Distribution Plots: Boxplots and Histograms 70Heatmaps: Visualizing Correlations and Missing Values 733.4 Multidimensional Visualization 75Adding Variables: Color, Size, Shape, Multiple Panels, and Animation 76Manipulations: Rescaling, Aggregation and Hierarchies, Zooming, Filtering 79Reference: Trend Lines and Labels 83Scaling Up to Large Datasets 85Multivariate Plot: Parallel Coordinates Plot 85Interactive Visualization 883.5 Specialized Visualizations 91Visualizing Networked Data 91Visualizing Hierarchical Data: Treemaps 93Visualizing Geographical Data: Map Charts 953.6 Major Visualizations and Operations, by Machine Learning Goal 97Prediction 97Classification 97Time Series Forecasting 97Unsupervised Learning 98Problems 99Chapter 4 Dimension Reduction 1014.1 Introduction 1014.2 Curse of Dimensionality 1024.3 Practical Considerations 102Example 1: House Prices in Boston 1034.4 Data Summaries 103Summary Statistics 104Aggregation and Pivot Tables 1044.5 Correlation Analysis 1074.6 Reducing the Number of Categories in Categorical Variables 1094.7 Converting a Categorical Variable to a Numerical Variable 1114.8 Principal Component Analysis 111Example 2: Breakfast Cereals 111Principal Components 116Normalizing the Data 117Using Principal Components for Classification and Prediction 1204.9 Dimension Reduction Using Regression Models 1214.10 Dimension Reduction Using Classification and Regression Trees 121Problems 123Part III Performance EvaluationChapter 5 Evaluating Predictive Performance 1295.1 Introduction 1305.2 Evaluating Predictive Performance 130Naive Benchmark: The Average 131Prediction Accuracy Measures 131Comparing Training and Holdout Performance 133Cumulative Gains and Lift Charts 1335.3 Judging Classifier Performance 136Benchmark: The Naive Rule 136Class Separation 136The Confusion (Classification) Matrix 137Using the Holdout Data 138Accuracy Measures 139Propensities and Threshold for Classification 139Performance in Case of Unequal Importance of Classes 143Asymmetric Misclassification Costs 146Generalization to More Than Two Classes 1495.4 Judging Ranking Performance 150Cumulative Gains and Lift Charts for Binary Data 150Decile-wise Lift Charts 153Beyond Two Classes 154Gains and Lift Charts Incorporating Costs and Benefits 154Cumulative Gains as a Function of Threshold 1555.5 Oversampling 156Creating an Over-sampled Training Set 158Evaluating Model Performance Using a Non-oversampled Holdout Set 159Evaluating Model Performance If Only Oversampled Holdout Set Exists 159Problems 162Part IV Prediction and Classification MethodsChapter 6 Multiple Linear Regression 1676.1 Introduction 1676.2 Explanatory vs. Predictive Modeling 1686.3 Estimating the Regression Equation and Prediction 170Example: Predicting the Price of Used Toyota Corolla Cars 171Cross-validation and caret 1756.4 Variable Selection in Linear Regression 176Reducing the Number of Predictors 176How to Reduce the Number of Predictors 178Regularization (Shrinkage Models) 183Problems 188Chapter 7 k-Nearest Neighbors (kNN) 1937.1 The k-NN Classifier (Categorical Outcome) 193Determining Neighbors 194Classification Rule 194Example: Riding Mowers 195Choosing k 196Weighted k-NN 199Setting the Cutoff Value 200k-NN with More Than Two Classes 201Converting Categorical Variables to Binary Dummies 2017.2 k-NN for a Numerical Outcome 2017.3 Advantages and Shortcomings of k-NN Algorithms 204Problems 205Chapter 8 The Naive Bayes Classifier 2078.1 Introduction 207Threshold Probability Method 208Conditional Probability 208Example 1: Predicting Fraudulent Financial Reporting 2088.2 Applying the Full (Exact) Bayesian Classifier 209Using the “Assign to the Most Probable Class” Method 210Using the Threshold Probability Method 210Practical Difficulty with the Complete (Exact) Bayes Procedure 2108.3 Solution: Naive Bayes 211The Naive Bayes Assumption of Conditional Independence 212Using the Threshold Probability Method 212Example 2: Predicting Fraudulent Financial Reports, Two Predictors 213Example 3: Predicting Delayed Flights 214Working with Continuous Predictors 2188.4 Advantages and Shortcomings of the Naive Bayes Classifier 220Problems 223Chapter 9 Classification and Regression Trees 2259.1 Introduction 226Tree Structure 227Decision Rules 227Classifying a New Record 2279.2 Classification Trees 228Recursive Partitioning 228Example 1: Riding Mowers 228Measures of Impurity 2319.3 Evaluating the Performance of a Classification Tree 235Example 2: Acceptance of Personal Loan 2369.4 Avoiding Overfitting 239Stopping Tree Growth 242Pruning the Tree 243Best-Pruned Tree 2459.5 Classification Rules from Trees 2479.6 Classification Trees for More Than Two Classes 2489.7 Regression Trees 249Prediction 250Measuring Impurity 250Evaluating Performance 2509.8 Advantages and Weaknesses of a Tree 2509.9 Improving Prediction: Random Forests and Boosted Trees 252Random Forests 252Boosted Trees 254Problems 257Chapter 10 Logistic Regression 26110.1 Introduction 26110.2 The Logistic Regression Model 26310.3 Example: Acceptance of Personal Loan 264Model with a Single Predictor 265Estimating the Logistic Model from Data: Computing Parameter Estimates 267Interpreting Results in Terms of Odds (for a Profiling Goal) 27010.4 Evaluating Classification Performance 27110.5 Variable Selection 27310.6 Logistic Regression for Multi-Class Classification 274Ordinal Classes 275Nominal Classes 27610.7 Example of Complete Analysis: Predicting Delayed Flights 277Data Preprocessing 282Model-Fitting and Estimation 282Model Interpretation 282Model Performance 284Variable Selection 285Problems 289Chapter 11 Neural Nets 29311.1 Introduction 29311.2 Concept and Structure of a Neural Network 29411.3 Fitting a Network to Data 295Example 1: Tiny Dataset 295Computing Output of Nodes 296Preprocessing the Data 299Training the Model 300Example 2: Classifying Accident Severity 304Avoiding Overfitting 305Using the Output for Prediction and Classification 30511.4 Required User Input 30711.5 Exploring the Relationship Between Predictors and Outcome 30811.6 Deep Learning 309Convolutional Neural Networks (CNNs) 310Local Feature Map 311A Hierarchy of Features 311The Learning Process 312Unsupervised Learning 312Example: Classification of Fashion Images 313Conclusion 32011.7 Advantages and Weaknesses of Neural Networks 320Problems 322Chapter 12 Discriminant Analysis 32512.1 Introduction 325Example 1: Riding Mowers 326Example 2: Personal Loan Acceptance 32712.2 Distance of a Record from a Class 32712.3 Fisher’s Linear Classification Functions 32912.4 Classification Performance of Discriminant Analysis 33312.5 Prior Probabilities 33412.6 Unequal Misclassification Costs 33412.7 Classifying More Than Two Classes 336Example 3: Medical Dispatch to Accident Scenes 33612.8 Advantages and Weaknesses 339Problems 341Chapter 13 Generating, Comparing, and Combining Multiple Models 34513.1 Ensembles 346Why Ensembles Can Improve Predictive Power 346Simple Averaging or Voting 348Bagging 349Boosting 349Bagging and Boosting in R 349Stacking 350Advantages and Weaknesses of Ensembles 35113.2 Automated Machine Learning (AutoML) 352AutoML: Explore and Clean Data 352AutoML: Determine Machine Learning Task 353AutoML: Choose Features and Machine Learning Methods 354AutoML: Evaluate Model Performance 354AutoML: Model Deployment 356Advantages and Weaknesses of Automated Machine Learning 35713.3 Explaining Model Predictions 35813.4 Summary 360Problems 362345Part V Intervention and User FeedbackChapter 14 Interventions: Experiments, Uplift Models, and Reinforcement Learning 36714.1 A/B Testing 368Example: Testing a New Feature in a Photo Sharing App 369The Statistical Test for Comparing Two Groups (T-Test) 370Multiple Treatment Groups: A/B/n Tests 372Multiple A/B Tests and the Danger of Multiple Testing 37214.2 Uplift (Persuasion) Modeling 373Gathering the Data 374A Simple Model 376Modeling Individual Uplift 376Computing Uplift with R 378Using the Results of an Uplift Model 37814.3 Reinforcement Learning 380Explore-Exploit: Multi-armed Bandits 380Example of Using a Contextual Multi-Arm Bandit for Movie Recommendations 382Markov Decision Process (MDP) 38314.4 Summary 388Problems 390Part VI Mining Relationships Among RecordsChapter 15 Association Rules and Collaborative Filtering 39315.1 Association Rules 394Discovering Association Rules in Transaction Databases 394Example 1: Synthetic Data on Purchases of Phone Faceplates 394Generating Candidate Rules 395The Apriori Algorithm 397Selecting Strong Rules 397Data Format 399The Process of Rule Selection 400Interpreting the Results 401Rules and Chance 403Example 2: Rules for Similar Book Purchases 40515.2 Collaborative Filtering 407Data Type and Format 407Example 3: Netflix Prize Contest 408User-Based Collaborative Filtering: “People Like You” 409Item-Based Collaborative Filtering 411Evaluating Performance 412Example 4: Predicting Movie Ratings with MovieLens Data 413Advantages and Weaknesses of Collaborative Filtering 416Collaborative Filtering vs. Association Rules 41715.3 Summary 419Problems 421Chapter 16 Cluster Analysis 42516.1 Introduction 426Example: Public Utilities 42716.2 Measuring Distance Between Two Records 429Euclidean Distance 429Normalizing Numerical Variables 430Other Distance Measures for Numerical Data 432Distance Measures for Categorical Data 433Distance Measures for Mixed Data 43416.3 Measuring Distance Between Two Clusters 434Minimum Distance 434Maximum Distance 435Average Distance 435Centroid Distance 43516.4 Hierarchical (Agglomerative) Clustering 437Single Linkage 437Complete Linkage 438Average Linkage 438Centroid Linkage 438Ward’s Method 438Dendrograms: Displaying Clustering Process and Results 439Validating Clusters 441Limitations of Hierarchical Clustering 44316.5 Non-Hierarchical Clustering: The k-Means Algorithm 444Choosing the Number of Clusters (k) 445Problems 450Part VII Forecasting Time SeriesChapter 17 Handling Time Series 45517.1 Introduction 45517.2 Descriptive vs. Predictive Modeling 45717.3 Popular Forecasting Methods in Business 457Problems 466Chapter 18 Regression-Based Forecasting 46918.1 A Model with Trend 469Linear Trend 469Exponential Trend 473Polynomial Trend 474Problems 489Chapter 19 Smoothing and Deep Learning Methods for Forecasting 49919.1 Smoothing Methods: Introduction 50019.2 Moving Average 500Centered Moving Average for Visualization 500Trailing Moving Average for Forecasting 501Choosing Window Width (w) 504Problems 516Part VIII Data AnalyticsChapter 20 Social Network Analytics 52720.1 Introduction 52720.2 Directed vs. Undirected Networks 52920.3 Visualizing and Analyzing Networks 530Plot Layout 530Edge List 533Adjacency Matrix 533Using Network Data in Classification and Prediction 534Problems 548Chapter 21 Text Mining 54921.1 Introduction 54921.2 The Tabular Representation of Text 55021.3 Bag-of-Words vs. Meaning Extraction at Document Level 551Problems 570Chapter 22 Responsible Data Science 57322.1 Introduction 57322.2 Unintentional Harm 57422.3 Legal Considerations 57622.4 Principles of Responsible Data Science 577Non-maleficence 578Fairness 578Transparency 579Accountability 580Data Privacy and Security 580Problems 599Part IX CasesChapter 23 Cases 60323.1 Charles Book Club 603The Book Industry 603Database Marketing at Charles 604Machine Learning Techniques 606Assignment 60823.2 German Credit 610Background 610Data 610Assignment 614Index 647