Numerical Issues in Statistical Computing for the Social Scientist

MICAH ALTMAN is Associate Director of the Harvard-MIT Data Center in Cambridge, Massachusetts. JEFF GILL is Associate Professor of Political Science at the University of California, Davis.MICHAEL P. McDONALD is Assistant Professor of Government and Politics at George Mason University in Fairfax, Virginia.

• Preface xi1 Introduction: Consequences of Numerical Inaccuracy 11.1 Importance of Understanding Computational Statistics 11.2 Brief History: Duhem to the Twenty-First Century 31.3 Motivating Example: Rare Events Counts Models 61.4 Preview of Findings 102 Sources of Inaccuracy in Statistical Computation 122.1 Introduction 122.1.1 Revealing Example: Computing the Coefficient Standard Deviation 122.1.2 Some Preliminary Conclusions 132.2 Fundamental Theoretical Concepts 152.2.1 Accuracy and Precision 152.2.2 Problems, Algorithms, and Implementations 152.3 Accuracy and Correct Inference 182.3.1 Brief Digression: Why Statistical Inference Is Harder in Practice Than It Appears 202.4 Sources of Implementation Errors 212.4.1 Bugs, Errors, and Annoyances 222.4.2 Computer Arithmetic 232.5 Algorithmic Limitations 292.5.1 Randomized Algorithms 302.5.2 Approximation Algorithms for Statistical Functions 312.5.3 Heuristic Algorithms for Random Number Generation 322.5.4 Local Search Algorithms 392.6 Summary 413 Evaluating Statistical Software 443.1 Introduction 443.1.1 Strategies for Evaluating Accuracy 443.1.2 Conditioning 473.2 Benchmarks for Statistical Packages 483.2.1 NIST Statistical Reference Datasets 493.2.2 Benchmarking Nonlinear Problems with StRD 513.2.3 Analyzing StRD Test Results 533.2.4 Empirical Tests of Pseudo-Random Number Generation 543.2.5 Tests of Distribution Functions 583.2.6 Testing the Accuracy of Data Input and Output 603.3 General Features Supporting Accurate and Reproducible Results 633.4 Comparison of Some Popular Statistical Packages 643.5 Reproduction of Research 653.6 Choosing a Statistical Package 694 Robust Inference 714.1 Introduction 714.2 Some Clarification of Terminology 714.3 Sensitivity Tests 734.3.1 Sensitivity to Alternative Implementations and Algorithms 734.3.2 Perturbation Tests 754.3.3 Tests of Global Optimality 844.4 Obtaining More Accurate Results 914.4.1 High-Precision Mathematical Libraries 924.4.2 Increasing the Precision of Intermediate Calculations 934.4.3 Selecting Optimization Methods 954.5 Inference for Computationally Difficult Problems 1034.5.1 Obtaining Confidence Intervals with Ill-Behaved Functions 1044.5.2 Interpreting Results in the Presence of Multiple Modes 1064.5.3 Inference in the Presence of Instability 1145 Numerical Issues in Markov Chain Monte Carlo Estimation 1185.1 Introduction 1185.2 Background and History 1195.3 Essential Markov Chain Theory 1205.3.1 Measure and Probability Preliminaries 1205.3.2 Markov Chain Properties 1215.3.3 The Final Word (Sort of) 1255.4 Mechanics of Common MCMC Algorithms 1265.4.1 Metropolis–Hastings Algorithm 1265.4.2 Hit-and-Run Algorithm 1275.4.3 Gibbs Sampler 1285.5 Role of Random Number Generation 1295.5.1 Periodicity of Generators and MCMC Effects 1305.5.2 Periodicity and Convergence 1325.5.3 Example: The Slice Sampler 1355.5.4 Evaluating WinBUGS 1375.6 Absorbing State Problem 1395.7 Regular Monte Carlo Simulation 1405.8 So What Can Be Done? 1416 Numerical Issues Involved in Inverting Hessian Matrices 143Jeff Gill and Gary King6.1 Introduction 1436.2 Means versus Modes 1456.3 Developing a Solution Using Bayesian Simulation Tools 1476.4 What Is It That Bayesians Do? 1486.5 Problem in Detail: Noninvertible Hessians 1496.6 Generalized Inverse/Generalized Cholesky Solution 1516.7 Generalized Inverse 1516.7.1 Numerical Examples of the Generalized Inverse 1546.8 Generalized Cholesky Decomposition 1556.8.1 Standard Algorithm 1566.8.2 Gill–Murray Cholesky Factorization 1566.8.3 Schnabel–Eskow Cholesky Factorization 1586.8.4 Numerical Examples of the Generalized Cholesky Decomposition 1586.9 Importance Sampling and Sampling Importance Resampling 1606.9.1 Algorithm Details 1606.9.2 SIR Output 1626.9.3 Relevance to the Generalized Process 1636.10 Public Policy Analysis Example 1636.10.1 Texas 1646.10.2 Florida 1686.11 Alternative Methods 1716.11.1 Drawing from the Singular Normal 1716.11.2 Aliasing 1736.11.3 Ridge Regression 1736.11.4 Derivative Approach 1746.11.5 Bootstrapping 1746.11.6 Respecification (Redux) 1756.12 Concluding Remarks 1767 Numerical Behavior of King’s EI Method 1777.1 Introduction 1777.2 Ecological Inference Problem and Proposed Solutions 1797.3 Numeric Accuracy in Ecological Inference 1807.3.1 Case Study 1: Examples from King (1997) 1827.3.2 Nonlinear Optimization 1867.3.3 Pseudo-Random Number Generation 1877.3.4 Platform and Version Sensitivity 1887.4 Case Study 2: Burden and Kimball (1998) 1897.4.1 Data Perturbation 1917.4.2 Option Dependence 1947.4.3 Platform Dependence 1957.4.4 Discussion: Summarizing Uncertainty 1967.5 Conclusions 1978 Some Details of Nonlinear Estimation 199B. D. McCullough8.1 Introduction 1998.2 Overview of Algorithms 2008.3 Some Numerical Details 2048.4 What Can Go Wrong? 2068.5 Four Steps 2108.5.1 Step 1: Examine the Gradient 2118.5.2 Step 2: Inspect the Trace 2118.5.3 Step 3: Analyze the Hessian 2128.5.4 Step 4: Profile the Objective Function 2128.6 Wald versus Likelihood Inference 2158.7 Conclusions 2179 Spatial Regression Models 219James P. LeSage9.1 Introduction 2199.2 Sample Data Associated with Map Locations 2199.2.1 Spatial Dependence 2199.2.2 Specifying Dependence Using Weight Matrices 2209.2.3 Estimation Consequences of Spatial Dependence 2229.3 Maximum Likelihood Estimation of Spatial Models 2239.3.1 Sparse Matrix Algorithms 2249.3.2 Vectorization of the Optimization Problem 2259.3.3 Trade-offs between Speed and Numerical Accuracy 2269.3.4 Applied Illustrations 2289.4 Bayesian Spatial Regression Models 2299.4.1 Bayesian Heteroscedastic Spatial Models 2309.4.2 Estimation of Bayesian Spatial Models 2319.4.3 Conditional Distributions for the SAR Model 2329.4.4 MCMC Sampler 2349.4.5 Illustration of the Bayesian Model 2349.5 Conclusions 23610 Convergence Problems in Logistic Regression 238Paul Allison10.1 Introduction 23810.2 Overview of Logistic Maximum Likelihood Estimation 23810.3 What Can Go Wrong? 24010.4 Behavior of the Newton–Raphson Algorithm under Separation 24310.4.1 Specific Implementations 24410.4.2 Warning Messages 24410.4.3 False Convergence 24610.4.4 Reporting of Parameter Estimates and Standard Errors 24710.4.5 Likelihood Ratio Statistics 24710.5 Diagnosis of Separation Problems 24710.6 Solutions for Quasi-Complete Separation 24810.6.1 Deletion of Problem Variables 24810.6.2 Combining Categories 24810.6.3 Do Nothing and Report Likelihood Ratio Chi-Squares 24910.6.4 Exact Inference 24910.6.5 Bayesian Estimation 25010.6.6 Penalized Maximum Likelihood Estimation 25010.7 Solutions for Complete Separation 25110.8 Extensions 25211 Recommendations for Replication and Accurate Analysis 25311.1 General Recommendations for Replication 25311.1.1 Reproduction, Replication, and Verification 25411.1.2 Recreating Data 25511.1.3 Inputting Data 25611.1.4 Analyzing Data 25711.2 Recommendations for Producing Verifiable Results 25911.3 General Recommendations for Improving the Numeric Accuracy of Analysis 26011.4 Recommendations for Particular Statistical Models 26111.4.1 Nonlinear Least Squares and Maximum Likelihood 26111.4.2 Robust Hessian Inversion 26211.4.3 MCMC Estimation 26311.4.4 Logistic Regression 26511.4.5 Spatial Regression 26611.5 Where Do We Go from Here? 266Bibliography 267Author Index 303Subject Index 315

"Uniquely accessible and abounding in modern-day tools, tricks, and advice, the text successfully bridges the gap between the current level of social science methodology and the more sophisticated technical coverage." (Zentralblatt Math 1130, May 2008) "Clarity of presentations is excellent. Applied statisticians and computer scientists will like this book and find it very useful." (Journal of Statistical Computation and Simulation, November 2005)"[The authors] …have succeeded in providing...a good understanding of the potential pitfalls involved in the implementation of methodology computationally, and...good advice on dealing with the problems that can arise." (Statistics in Medical Research, June 2005)“This book provides the researcher with an overview of the issues involved in the implementation and computation of common statistical procedures….” (Statistical Methods in Medical Research, Vol. 14, 2005)"…this book is a good reference for social scientists that are involved in computational statistics." (Journal of Statistical Software, April 2005)"…timely and interesting, and on the whole provides a good balance of theory, application, and computation." (Technometrics, May 2005)"…an excellent text. It has the potential to be enormously influential across the social sciences…It should be required reading for everyone who performs statistical computing at the advanced level…" (Journal of the American Statistical Association, June 2005)“…a compact guide to the voluminous literature on optimisation, numerical analysis, and computational statistics. This is no small achievement.” (Statistical Software Newsletter in Computational Statistics and Data Analysis)"…a very important one for researchers, social scientists, and…graduate and post-graduate students in various disciplines..." (Computing Reviews.com, July 6, 2004)"This comprehensive research and guidebook by Altman, Gill, and McDonald offers to social scientists modern tools and tricks previously lacking in other works.” (Choice, June 2004, Vol. 41 No. 10)