Introduction to Linear Regression Analysis

DOUGLAS C. MONTGOMERY, PHD, is Regents Professor of Industrial Engineering and Statistics at Arizona State University. Dr. Montgomery is the co-author of several Wiley books including Introduction to Linear Regression Analysis, 5th Edition.ELIZABETH A. PECK, PHD, is Logistics Modeling Specialist at the Coca-Cola Company in Atlanta, Georgia. G. GEOFFREY VINING, PHD, is Professor in the Department of Statistics at Virginia Polytechnic and State University. Dr. Peck is co-author of Introduction to Linear Regression Analysis, 5th Edition.

• Preface xiiiAbout the Companion Website xvi1. Introduction 11.1 Regression and Model Building 11.2 Data Collection 51.3 Uses of Regression 91.4 Role of the Computer 102. Simple Linear Regression 122.1 Simple Linear Regression Model 122.2 Least-Squares Estimation of the Parameters 132.2.1 Estimation of β0 and β1 132.2.2 Properties of the Least-Squares Estimators and the Fitted Regression Model 182.2.3 Estimation of σ2 202.2.4 Alternate Form of the Model 222.3 Hypothesis Testing on the Slope and Intercept 222.3.1 Use of t Tests 222.3.2 Testing Significance of Regression 242.3.3 Analysis of Variance 252.4 Interval Estimation in Simple Linear Regression 292.4.1 Confidence Intervals on β0, β1, and σ2 292.4.2 Interval Estimation of the Mean Response 302.5 Prediction of New Observations 332.6 Coefficient of Determination 352.7 A Service Industry Application of Regression 372.8 Does Pitching Win Baseball Games? 392.9 Using SAS® and R for Simple Linear Regression 412.10 Some Considerations in the Use of Regression 442.11 Regression Through the Origin 462.12 Estimation by Maximum Likelihood 522.13 Case Where the Regressor x Is Random 532.13.1 x and y Jointly Distributed 542.13.2 x and y Jointly Normally Distributed: Correlation Model 54Problems 593. Multiple Linear Regression 693.1 Multiple Regression Models 693.2 Estimation of the Model Parameters 723.2.1 Least-Squares Estimation of the Regression Coefficients 723.2.2 Geometrical Interpretation of Least Squares 793.2.3 Properties of the Least-Squares Estimators 813.2.4 Estimation of σ2 823.2.5 Inadequacy of Scatter Diagrams in Multiple Regression 843.2.6 Maximum-Likelihood Estimation 853.3 Hypothesis Testing in Multiple Linear Regression 863.3.1 Test for Significance of Regression 863.3.2 Tests on Individual Regression Coefficients and Subsets of Coefficients 903.3.3 Special Case of Orthogonal Columns in X 953.3.4 Testing the General Linear Hypothesis 973.4 Confidence Intervals in Multiple Regression 993.4.1 Confidence Intervals on the Regression Coefficients 1003.4.2 ci Estimation of the Mean Response 1013.4.3 Simultaneous Confidence Intervals on Regression Coefficients 1023.5 Prediction of New Observations 1063.6 A Multiple Regression Model for the Patient Satisfaction Data 1063.7 Does Pitching and Defense Win Baseball Games? 1083.8 Using SAS and R for Basic Multiple Linear Regression 1103.9 Hidden Extrapolation in Multiple Regression 1113.10 Standardized Regression Coefficients 1153.11 Multicollinearity 1213.12 Why Do Regression Coefficients Have the Wrong Sign? 123Problems 1254. Model Adequacy Checking 1344.1 Introduction 1344.2 Residual Analysis 1354.2.1 Definition of Residuals 1354.2.2 Methods for Scaling Residuals 1354.2.3 Residual Plots 1414.2.4 Partial Regression and Partial Residual Plots 1484.2.5 Using Minitab®, SAS, and R for Residual Analysis 1514.2.6 Other Residual Plotting and Analysis Methods 1544.3 PRESS Statistic 1564.4 Detection and Treatment of Outliers 1574.5 Lack of Fit of the Regression Model 1614.5.1 A Formal Test for Lack of Fit 1614.5.2 Estimation of Pure Error from Near Neighbors 165Problems 1705. Transformations and Weighting To Correct Model Inadequacies 1775.1 Introduction 1775.2 Variance-Stabilizing Transformations 1785.3 Transformations to Linearize the Model 1825.4 Analytical Methods for Selecting a Transformation 1885.4.1 Transformations on y: The Box–Cox Method 1885.4.2 Transformations on the Regressor Variables 1905.5 Generalized and Weighted Least Squares 1945.5.1 Generalized Least Squares 1945.5.2 Weighted Least Squares 1965.5.3 Some Practical Issues 1975.6 Regression Models with Random Effects 2005.6.1 Subsampling 2005.6.2 The General Situation for a Regression Model with a Single Random Effect 2045.6.3 The Importance of the Mixed Model in Regression 208Problems 2086. Diagnostics for Leverage and Influence 2176.1 Importance of Detecting Influential Observations 2176.2 Leverage 2186.3 Measures of Influence: Cook’s D 2216.4 Measures of Influence: DFFITS and DFBETAS 2236.5 A Measure of Model Performance 2256.6 Detecting Groups of Influential Observations 2266.7 Treatment of Influential Observations 226Problems 2277. Polynomial Regression Models 2307.1 Introduction 2307.2 Polynomial Models in One Variable 2307.2.1 Basic Principles 2307.2.2 Piecewise Polynomial Fitting (Splines) 2367.2.3 Polynomial and Trigonometric Terms 2427.3 Nonparametric Regression 2437.3.1 Kernel Regression 2447.3.2 Locally Weighted Regression (Loess) 2447.3.3 Final Cautions 2497.4 Polynomial Models in Two or More Variables 2497.5 Orthogonal Polynomials 255Problems 2618. Indicator Variables 2688.1 General Concept of Indicator Variables 2688.2 Comments on the Use of Indicator Variables 2818.2.1 Indicator Variables versus Regression on Allocated Codes 2818.2.2 Indicator Variables as a Substitute for a Quantitative Regressor 2828.3 Regression Approach to Analysis of Variance 283Problems 2889. Multicollinearity 2939.1 Introduction 2939.2 Sources of Multicollinearity 2949.3 Effects of Multicollinearity 2969.4 Multicollinearity Diagnostics 3009.4.1 Examination of the Correlation Matrix 3009.4.2 Variance Inflation Factors 3049.4.3 Eigensystem Analysis of XʹX 3059.4.4 Other Diagnostics 3109.4.5 SAS and R Code for Generating Multicollinearity Diagnostics 3119.5 Methods for Dealing with Multicollinearity 3119.5.1 Collecting Additional Data 3119.5.2 Model Respecification 3129.5.3 Ridge Regression 3129.5.4 Principal-Component Regression 3299.5.5 Comparison and Evaluation of Biased Estimators 3349.6 Using SAS to Perform Ridge and Principal-Component Regression 336Problems 33810. Variable Selection and Model Building 34210.1 Introduction 34210.1.1 Model-Building Problem 34210.1.2 Consequences of Model Misspecification 34410.1.3 Criteria for Evaluating Subset Regression Models 34710.2 Computational Techniques for Variable Selection 35310.2.1 All Possible Regressions 35310.2.2 Stepwise Regression Methods 35910.3 Strategy for Variable Selection and Model Building 36710.4 Case Study: Gorman and Toman Asphalt Data Using SAS 370Problems 38311. Validation of Regression Models 38811.1 Introduction 38811.2 Validation Techniques 38911.2.1 Analysis of Model Coefficients and Predicted Values 38911.2.2 Collecting Fresh Data—Confirmation Runs 39111.2.3 Data Splitting 39311.3 Data from Planned Experiments 401Problems 40212. Introduction to Nonlinear Regression 40512.1 Linear and Nonlinear Regression Models 40512.1.1 Linear Regression Models 40512.1.2 Nonlinear Regression Models 40612.2 Origins of Nonlinear Models 40712.3 Nonlinear Least Squares 41112.4 Transformation to a Linear Model 41312.5 Parameter Estimation in a Nonlinear System 41612.5.1 Linearization 41612.5.2 Other Parameter Estimation Methods 42312.5.3 Starting Values 42412.6 Statistical Inference in Nonlinear Regression 42512.7 Examples of Nonlinear Regression Models 42712.8 Using SAS and R 428Problems 43213. Generalized Linear Models 44013.1 Introduction 44013.2 Logistic Regression Models 44113.2.1 Models with a Binary Response Variable 44113.2.2 Estimating the Parameters in a Logistic Regression Model 44213.2.3 Interpretation of the Parameters in a Logistic Regression Model 44713.2.4 Statistical Inference on Model Parameters 44913.2.5 Diagnostic Checking in Logistic Regression 45913.2.6 Other Models for Binary Response Data 46113.2.7 More Than Two Categorical Outcomes 46113.3 Poisson Regression 46313.4 The Generalized Linear Model 46913.4.1 Link Functions and Linear Predictors 47013.4.2 Parameter Estimation and Inference in the GLM 47113.4.3 Prediction and Estimation with the GLM 47313.4.4 Residual Analysis in the GLM 47513.4.5 Using R to Perform GLM Analysis 47713.4.6 Overdispersion 480Problems 48114. Regression Analysis of Time Series Data 49514.1 Introduction to Regression Models for Time Series Data 49514.2 Detecting Autocorrelation: The Durbin–Watson Test 49614.3 Estimating the Parameters in Time Series Regression Models 501Problems 51715. Other Topics in the Use of Regression Analysis 52115.1 Robust Regression 52115.1.1 Need for Robust Regression 52115.1.2 M-Estimators 52415.1.3 Properties of Robust Estimators 53115.2 Effect of Measurement Errors in the Regressors 53215.2.1 Simple Linear Regression 53215.2.2 The Berkson Model 53415.3 Inverse Estimation—The Calibration Problem 53415.4 Bootstrapping in Regression 53815.4.1 Bootstrap Sampling in Regression 53915.4.2 Bootstrap Confidence Intervals 54015.5 Classification and Regression Trees (CART) 54515.6 Neural Networks 54715.7 Designed Experiments for Regression 549Problems 557Appendix A. Statistical Tables 561Appendix B. Data Sets for Exercises 573Appendix C. Supplemental Technical Material 602C.1 Background on Basic Test Statistics 602C.2 Background from the Theory of Linear Models 605C.3 Important Results on SS R and SS Res 609C.4 Gauss-Markov Theorem, Var(ε) = σ 2 I 615C.5 Computational Aspects of Multiple Regression 617C.6 Result on the Inverse of a Matrix 618C.7 Development of the PRESS Statistic 619C.8 Development of S(i) 2 621C.9 Outlier Test Based on R-Student 622C.10 Independence of Residuals and Fitted Values 624C.11 Gauss–Markov Theorem, Var(ε) = V 625C.12 Bias in MSRes When the Model Is Underspecified 627C.13 Computation of Influence Diagnostics 628C.14 Generalized Linear Models 629Appendix D. Introduction to SAS 641D.1 Basic Data Entry 642D.2 Creating Permanent SAS Data Sets 646D.3 Importing Data from an EXCEL File 647D.4 Output Command 648D.5 Log File 648D.6 Adding Variables to an Existing SAS Data Set 650Appendix E. Introduction to R to Perform Linear Regression Analysis 651E.1 Basic Background on R 651E.2 Basic Data Entry 652E.3 Brief Comments on Other Functionality in R 654E.4 R Commander 655References 656Index 670