Surrogate Modeling and Optimization

Nam-Ho Kim is a Professor in the Department of Mechanical and Aerospace Engineering at the University of Florida. His research interests include design under uncertainty, prognostics and health management, verification validation and uncertainty quantification, and nonlinear structural mechanics. He has more than twenty years of experience teaching materials in these fields to graduate students.

• Preface xiiiAcknowledgment xviiAbout the Companion Website xixPart I Basics of Surrogate Modeling 11 Introduction to Surrogate Models 31.1 What Is Surrogate Modeling? 31.2 Surrogate Models 51.3 Design of Experiments: Sampling 71.4 Interpolation Versus Extrapolation 81.5 Flowchart of Surrogate Modeling 101.6 Overview of Surrogate Modeling 121.7 Smoothness and Loss Function 142 Polynomial Response Surfaces 172.1 Introduction 172.2 Curve Fitting 192.3 Linear Regression 232.3.1 Polynomial Response Surface 232.3.2 Polynomial Response Surface in Multiple Dimensions 282.3.3 Curse of Dimensionality 302.3.4 Assumptions in Linear Regression 322.4 Goodness of Fit 332.4.1 Estimation of Noise in Samples 342.4.2 Coefficient of Multiple Determination 382.4.3 Cross-validation 432.5 Confidence of Coefficients and Backward Elimination 482.6 Prediction Variance 512.6.1 Prediction Uncertainty 522.6.2 Sample Sensitivity 532.6.3 Prediction Variance with Variable Noise 552.7 Outliers 572.8 Statistical View of Linear Regression 59Exercise 653 Design of Experiments 713.1 Introduction 713.2 Design of Experiments in Box-like Domains 733.2.1 Scaling of Input Variables 733.2.2 Interpolation, Extrapolation, and Prediction Variance 753.2.3 Designs for Linear Polynomial Response Surfaces 793.2.4 Designs for Quadratic Polynomial Response Surfaces 803.3 Optimal Design of Experiments 903.3.1 D-Optimal Design 913.3.2 A-Optimal Design 953.3.3 G-Optimal Design 963.3.4 Minimum Bias Design 993.4 Space-Filling Design of Experiments 1043.4.1 Monte Carlo Simulation 1043.4.2 Latin Hypercube Sampling 1053.4.3 Orthogonal Arrays 1093.5 Review of Various Designs of Experiments 1113.5.1 Guideline for Selecting Designs of Experiments 1113.5.2 Good Practice for Design of Experiments 112Exercise 113Part II Design Optimization 1174 Optimization Definition and Formulation 1194.1 Introduction 1194.2 Design Optimization Definition 1204.2.1 Design Optimization Process 1204.2.2 Design Variables and Feasible Domain 1224.2.3 Graphical Optimization 1264.3 Optimization Problem Formulation 1284.3.1 Three-step Problem Definition 1284.3.2 Standard Form 1294.3.3 Normalization 1304.3.4 Convex Function and Convex Problem 1334.4 Optimality Criteria 1354.4.1 Global Versus Local Optimum 1354.4.2 Unconstrained Optimization 1364.4.3 Constrained Optimization 1414.4.4 Effect of Constraint Limit 1494.4.5 Sensitivity of Optimum Solution to Parameters 151Exercise 1535 Numerical Optimization Algorithms 1615.1 Introduction 1615.2 Overview of the Numerical Optimization Process 1625.3 Determination of Step Size 1645.3.1 Descent Direction 1645.3.2 Step-Size Termination Criterion 1655.3.3 Interval Reduction Method 1665.3.4 Quadratic Interpolation Method 1675.4 Unconstrained Optimization Algorithms 1685.4.1 Steepest Descent Method 1695.4.2 Conjugate Gradient Method 1715.4.3 Newton Method 1735.4.4 Quasi-Newton Method 1745.4.5 Rate of Convergence 1775.5 Constrained Optimization Using Unconstrained Algorithms 1785.5.1 Lagrange Multiplier Method 1795.5.2 Penalty Function Method 1805.6 Constrained Optimization Using Direct Methods 1825.6.1 Sequential Linear Programming (SLP) Method 1825.6.2 Quadratic Programming (QP) Subproblem 1835.6.3 Constrained Steepest Descent Method 1845.6.4 Feasible Direction Method 1855.6.5 Constrained Quasi-Newton Method 1865.7 Matlab Optimization Toolbox 1875.8 Practical Suggestions for Numerical Optimization 191Exercise 1946 Global Search Optimization Algorithms 1976.1 Introduction 1976.2 Nelder–Mead Sequential Simplex Algorithm 1986.3 DIRECT Method 2026.3.1 Lipschitzian Optimization 2026.3.2 DIRECT in 1D 2046.3.3 DIRECT Algorithm 2056.4 Genetic Algorithms 2086.4.1 Representation of Design 2096.4.2 Genetic Operators 2116.4.3 Procedure of Genetic Algorithms 2126.4.4 Genetic Algorithm in Matlab 2156.4.5 When to Use Genetic Algorithm? 2166.5 Particle Swarm Optimization 2176.6 Simulated Annealing Optimization 221Exercise 225Part III Advanced Topics in Surrogate Modeling 2277 Kriging Surrogate–Gaussian Process Model 2297.1 Introduction 2297.2 Kriging Philosophy 2307.2.1 Correlation Between Two Random Variables 2317.2.2 Kriging Surrogate Approximation 2337.2.3 Correlation Model 2357.3 Kriging Surrogate Model 2387.3.1 Global Function and Distribution of Errors 2387.3.2 Local Departure 2437.3.3 Hyperparameters and Likelihood Function 2487.4 Issues in Determining Hyperparameters 2527.4.1 Lower and Upper Bounds of Hyperparameter 2527.4.2 Finding Optimum Value of Hyperparameter 2537.4.3 Issues Related to the Number of Samples 2557.4.4 Computational Cost of Kriging Surrogate 2567.4.5 Hyperparameter and Extrapolation Accuracy 2617.4.6 Uncertainty in Kriging Predictions 2637.4.7 Effect of Global Function 2697.5 Numerical Implementation of Kriging Surrogate 2707.6 Kriging with Nuggets—Fitting with Noisy Data (Gaussian Process Regression) 2827.6.1 Kriging Surrogate with Correlated Noise 2847.6.2 Kriging Surrogate with Homogeneous Noise 285Exercise 2908 Neural Network Model 2958.1 Introduction 2958.2 Feedforward Neural Network Model 2978.2.1 Concept of Feedforward Neural Network 2978.2.2 Feedforward Mechanism 2998.2.3 Activation Functions 3028.2.4 Backpropagation Process 3068.3 Matlab Functions for Feedforward Neural Network 3108.4 Uncertainty Quantification in Neural Network Models 3168.4.1 Training Uncertainty 3178.4.2 Sampling Uncertainty 3218.4.3 Confidence Intervals and Prediction Intervals 3258.5 Issues in Feedforward Neural Network 3278.5.1 Adaptive Learning Rate 3278.5.2 Scaling Input Data 3288.5.3 Overfitting 3298.6 Neural Networks with Constraints 3408.6.1 Penalty Method for Constraints 3428.6.2 Regularization Using Soft-maximum 3438.6.3 Backpropagation of Penalty Constraints 3468.6.4 Updating Penalty Parameter 3478.6.5 Numerical Examples 349Exercise 3509 Multi-fidelity Surrogate Models 3539.1 Introduction 3539.2 Multi-fidelity Surrogate Models 3569.3 Regression-based Multi-fidelity Surrogate 3629.4 Kriging-based Multi-fidelity Surrogate 3699.4.1 Multi-fidelity Surrogate with Low-fidelity Function 3699.4.2 Multi-fidelity Surrogate with Low-fidelity Samples 3789.5 Sampling Strategy for Multi-fidelity Surrogate Modeling 3869.5.1 Selecting Locations for LF and HF Samples 3879.5.2 Allocating LF and HF Samples 3889.6 Challenges and Recommendations 3929.6.1 Deciding Whether to Use LF Samples 3929.6.2 Choosing Between Multiple LF Datasets 3939.6.3 Selecting ρ for Other Surrogates 3939.6.4 Recommendations on Using MF Surrogates 393Exercise 39410 Efficient Global Optimization 39710.1 Introduction 39710.2 Efficient Global Optimization 39910.2.1 Expected Improvement 40010.2.2 Probability of Improvement 40410.2.3 Adaptive Target for Probability of Improvement 41010.2.4 Expected Feasibility 41110.3 Efficient Global Optimization Using Polynomial Response Surface 41310.4 Efficient Global Optimization Using Kriging Surrogate 421Exercise 427References 429Index 435