Nonlinear Regression Modeling for Engineering Applications

R. Russell Rhinehart, Oklahoma State University, USA.Professor Rhinehart obtained his Ph.D. in Chemical Engineering in 1985 from North Carolina State University, USA. His research interests include process improvement (modeling, optimization, and control), and product improvement (modeling and design). In 2004 he was named as one of InTECHs 50 most influential industry innovators of the past 50 years, and was inducted into the Automation Hall of Fame for the Process Industries in 2005.  He has written extensively for numerous journals and refereed articles.

• Series Preface xiiiPreface xvAcknowledgments xxiiiNomenclature xxvSymbols xxxviiPart I INTRODUCTION1 Introductory Concepts 31.1 Illustrative Example – Traditional Linear Least-Squares Regression 31.2 How Models Are Used 71.3 Nonlinear Regression 71.4 Variable Types 81.5 Simulation 121.6 Issues 131.7 Takeaway 15Exercises 152 Model Types 162.1 Model Terminology 162.2 A Classification of Mathematical Model Types 172.3 Steady-State and Dynamic Models 212.3.1 Steady-State Models 222.3.2 Dynamic Models (Time-Dependent, Transient) 242.4 Pseudo-First Principles – Appropriated First Principles 262.5 Pseudo-First Principles – Pseudo-Components 282.6 Empirical Models with Theoretical Grounding 282.6.1 Empirical Steady State 282.6.2 Empirical Time-Dependent 302.7 Empirical Models with No Theoretical Grounding 312.8 Partitioned Models 312.9 Empirical or Phenomenological? 322.10 Ensemble Models 322.11 Simulators 332.12 Stochastic and Probabilistic Models 332.13 Linearity 342.14 Discrete or Continuous 362.15 Constraints 362.16 Model Design (Architecture, Functionality, Structure) 372.17 Takeaway 37Exercises 37Part II PREPARATION FOR UNDERLYING SKILLS3 Propagation of Uncertainty 433.1 Introduction 433.2 Sources of Error and Uncertainty 443.2.1 Estimation 453.2.2 Discrimination 453.2.3 Calibration Drift 453.2.4 Accuracy 453.2.5 Technique 463.2.6 Constants and Data 463.2.7 Noise 463.2.8 Model and Equations 463.2.9 Humans 473.3 Significant Digits 473.4 Rounding Off 483.5 Estimating Uncertainty on Values 493.5.1 Caution 503.6 Propagation of Uncertainty – Overview – Two Types, Two Ways Each 513.6.1 Maximum Uncertainty 513.6.2 Probable Uncertainty 563.6.3 Generality 583.7 Which to Report? Maximum or Probable Uncertainty 593.8 Bootstrapping 593.9 Bias and Precision 613.10 Takeaway 65Exercises 664 Essential Probability and Statistics 674.1 Variation and Its Role in Topics 674.2 Histogram and Its PDF and CDF Views 674.3 Constructing a Data-Based View of PDF and CDF 704.4 Parameters that Characterize the Distribution 714.5 Some Representative Distributions 724.5.1 Gaussian Distribution 724.5.2 Log-Normal Distribution 724.5.3 Logistic Distribution 744.5.4 Exponential Distribution 744.5.5 Binomial Distribution 754.6 Confidence Interval 764.7 Central Limit Theorem 774.8 Hypothesis and Testing 784.9 Type I and Type II Errors, Alpha and Beta 804.10 Essential Statistics for This Text 824.10.1 t-Test for Bias 834.10.2 Wilcoxon Signed Rank Test for Bias 834.10.3 r-lag-1 Autocorrelation Test 844.10.4 Runs Test 874.10.5 Test for Steady State in a Noisy Signal 874.10.6 Chi-Square Contingency Test 894.10.7 Kolmogorov–Smirnov Distribution Test 894.10.8 Test for Proportion 904.10.9 F-Test for Equal Variance 904.11 Takeaway 91Exercises 915 Simulation 935.1 Introduction 935.2 Three Sources of Deviation: Measurement, Inputs, Coefficients 935.3 Two Types of Perturbations: Noise (Independent) and Drifts (Persistence) 955.4 Two Types of Influence: Additive and Scaled with Level 985.5 Using the Inverse CDF to Generate n and u from UID(0, 1) 995.6 Takeaway 100Exercises 1006 Steady and Transient State Detection 1016.1 Introduction 1016.1.1 General Applications 1016.1.2 Concepts and Issues in Detecting Steady State 1046.1.3 Approaches and Issues to SSID and TSID 1046.2 Method 1066.2.1 Conceptual Model 1066.2.2 Equations 1076.2.3 Coefficient, Threshold, and Sample Frequency Values 1086.2.4 Noiseless Data 1116.3 Applications 1126.3.1 Applications of the R-Statistic Approach for Process Monitoring 1126.3.2 Applications of the R-Statistic Approach for Determining Regression Convergence 1126.4 Takeaway 114Exercises 114Part III REGRESSION, VALIDATION, DESIGN7 Regression Target – Objective Function 1197.1 Introduction 1197.2 Experimental and Measurement Uncertainty – Static and Continuous Valued 1197.3 Likelihood 1227.4 Maximum Likelihood 1247.5 Estimating σx and σy Values 1277.6 Vertical SSD – A Limiting Consideration of Variability Only in the Response Measurement 1277.7 r-Square as a Measure of Fit 1287.8 Normal, Total, or Perpendicular SSD 1307.9 Akaho’s Method 1327.10 Using a Model Inverse for Regression 1347.11 Choosing the Dependent Variable 1357.12 Model Prediction with Dynamic Models 1367.13 Model Prediction with Classification Models 1377.14 Model Prediction with Rank Models 1387.15 Probabilistic Models 1397.16 Stochastic Models 1397.17 Takeaway 139Exercises 1408 Constraints 1418.1 Introduction 1418.2 Constraint Types 1418.3 Expressing Hard Constraints in the Optimization Statement 1428.4 Expressing Soft Constraints in the Optimization Statement 1438.5 Equality Constraints 1478.6 Takeaway 148Exercises 1489 The Distortion of Linearizing Transforms 1499.1 Linearizing Coefficient Expression in Nonlinear Functions 1499.2 The Associated Distortion 1519.3 Sequential Coefficient Evaluation 1549.4 Takeaway 155Exercises 15510 Optimization Algorithms 15710.1 Introduction 15710.2 Optimization Concepts 15710.3 Gradient-Based Optimization 15910.3.1 Numerical Derivative Evaluation 15910.3.2 Steepest Descent – The Gradient 16110.3.3 Cauchy’s Method 16210.3.4 Incremental Steepest Descent (ISD) 16310.3.5 Newton–Raphson (NR) 16310.3.6 Levenberg–Marquardt (LM) 16510.3.7 Modified LM 16610.3.8 Generalized Reduced Gradient (GRG) 16710.3.9 Work Assessment 16710.3.10 Successive Quadratic (SQ) 16710.3.11 Perspective 16810.4 Direct Search Optimizers 16810.4.1 Cyclic Heuristic Direct Search 16910.4.2 Multiplayer Direct Search Algorithms 17010.4.3 Leapfrogging 17110.5 Takeaway 17311 Multiple Optima 17611.1 Introduction 17611.2 Quantifying the Probability of Finding the Global Best 17811.3 Approaches to Find the Global Optimum 17911.4 Best-of-N Rule for Regression Starts 18011.5 Interpreting the CDF 18211.6 Takeaway 18412 Regression Convergence Criteria 18512.1 Introduction 18512.2 Convergence versus Stopping 18512.3 Traditional Criteria for Claiming Convergence 18612.4 Combining DV Influence on OF 18812.5 Use Relative Impact as Convergence Criterion 18912.6 Steady-State Convergence Criterion 19012.7 Neural Network Validation 19712.8 Takeaway 198Exercises 19813 Model Design – Desired and Undesired Model Characteristics and Effects 19913.1 Introduction 19913.2 Redundant Coefficients 19913.3 Coefficient Correlation 20113.4 Asymptotic and Uncertainty Effects When Model is Inverted 20313.5 Irrelevant Coefficients 20513.6 Poles and Sign Flips w.r.t. the DV 20613.7 Too Many Adjustable Coefficients or Too Many Regressors 20613.8 Irrelevant Model Coefficients 21513.8.1 Standard Error of the Estimate 21613.8.2 Backward Elimination 21613.8.3 Logical Tests 21613.8.4 Propagation of Uncertainty 21613.8.5 Bootstrapping 21713.9 Scale-Up or Scale-Down Transition to New Phenomena 21713.10 Takeaway 218Exercises 21814 Data Pre- and Post-processing 22014.1 Introduction 22014.2 Pre-processing Techniques 22114.2.1 Steady- and Transient-State Selection 22114.2.2 Internal Consistency 22114.2.3 Truncation 22214.2.4 Averaging and Voting 22214.2.5 Data Reconciliation 22314.2.6 Real-Time Noise Filtering for Noise Reduction (MA, FoF, STF) 22414.2.7 Real-Time Noise filtering for Outlier Removal (Median Filter) 22714.2.8 Real-Time Noise Filtering, Statistical Process Control 22814.2.9 Imputation of Input Data 23014.3 Post-processing 23114.3.1 Outliers and Rejection Criterion 23114.3.2 Bimodal Residual Distributions 23314.3.3 Imputation of Response Data 23514.4 Takeaway 235Exercises 23515 Incremental Model Adjustment 23715.1 Introduction 23715.2 Choosing the Adjustable Coefficient in Phenomenological Models 23815.3 Simple Approach 23815.4 An Alternate Approach 24015.5 Other Approaches 24115.6 Takeaway 241Exercises 24116 Model and Experimental Validation 24216.1 Introduction 24216.1.1 Concepts 24216.1.2 Deterministic Models 24416.1.3 Stochastic Models 24616.1.4 Reality! 24916.2 Logic-Based Validation Criteria 25016.3 Data-Based Validation Criteria and Statistical Tests 25116.3.1 Continuous-Valued, Deterministic, Steady State, or End-of-Batch 25116.3.2 Continuous-Valued, Deterministic, Transient 26316.3.3 Class/Discrete/Rank-Valued, Deterministic, Batch, or Steady State 26416.3.4 Continuous-Valued, Stochastic, Batch, or Steady State 26516.3.5 Test for Normally Distributed Residuals 26616.3.6 Experimental Procedure Validation 26616.4 Model Discrimination 26716.4.1 Mechanistic Models 26716.4.2 Purely Empirical Models 26816.5 Procedure Summary 26816.6 Alternate Validation Approaches 26916.7 Takeaway 270Exercises 27017 Model Prediction Uncertainty 27217.1 Introduction 27217.2 Bootstrapping 27317.3 Takeaway 27618 Design of Experiments for Model Development and Validation 27718.1 Concept – Plan and Data 27718.2 Sufficiently Small Experimental Uncertainty – Methodology 27718.3 Screening Designs – A Good Plan for an Alternate Purpose 28118.4 Experimental Design – A Plan for Validation and Discrimination 28218.4.1 Continually Redesign 28218.4.2 Experimental Plan 28318.5 EHS&LP 28618.6 Visual Examples of Undesired Designs 28718.7 Example for an Experimental Plan 28918.8 Takeaway 291Exercises 29219 Utility versus Perfection 29319.1 Competing and Conflicting Measures of Excellence 29319.2 Attributes for Model Utility Evaluation 29419.3 Takeaway 295Exercises 29620 Troubleshooting 29720.1 Introduction 29720.2 Bimodal and Multimodal Residuals 29720.3 Trends in the Residuals 29820.4 Parameter Correlation 29820.5 Convergence Criterion – Too Tight, Too Loose 29920.6 Overfitting (Memorization) 30020.7 Solution Procedure Encounters Execution Errors 30020.8 Not a Sharp CDF (OF) 30020.9 Outliers 30120.10 Average Residual Not Zero 30220.11 Irrelevant Model Coefficients 30220.12 Data Work-Up after the Trials 30220.13 Too Many rs! 30320.14 Propagation of Uncertainty Does Not Match Residuals 30320.15 Multiple Optima 30420.16 Very Slow Progress 30420.17 All Residuals are Zero 30420.18 Takeaway 305Exercises 305Part IV CASE STUDIES AND DATA21 Case Studies 30921.1 Valve Characterization 30921.2 CO2 Orifice Calibration 31121.3 Enrollment Trend 31221.4 Algae Response to Sunlight Intensity 31421.5 Batch Reaction Kinetics 316Appendix A: VBA Primer: Brief on VBA Programming – Excel in Office 2013 319Appendix B: Leapfrogging Optimizer Code for Steady-State Models 328Appendix C: Bootstrapping with Static Model 341References and Further Reading 350Index 355