Planning and Executing Credible Experiments

ROBERT J. MOFFAT, PHD, is a Professor Emeritus of Mechanical Engineering at Stanford University. He proved engines for General Motors and is the former President of Moffat Thermosciences, Inc. His main areas of research include convective heat transfer in engineering systems, experimental methods in heat transfer and fluid mechanics, and biomedical thermal issues. ROY W. HENK, PHD, designed aerospace engine components and has conducted experimental tests in industry, a government lab and internationally. He has been a Professor in the USA, South Korea, and most notably at Kyoto University in Japan. His research includes experiment design, energy conversion, turbomachinery, and thermal fluid physics.

• About the Authors xxiPreface xxiiiAcknowledgments xxviiAbout the Companion Website xxix1 Choosing Credibility 11.1 The Responsibility of an Experimentalist 21.2 Losses of Credibility 21.3 Recovering Credibility 31.4 Starting with a Sharp Axe 31.5 A Systems View of Experimental Work 41.6 In Defense of Being a Generalist 5Panel 1.1 The Bundt Cake Story 6References 6Homework 62 The Nature of Experimental Work 72.1 Tested Guide of Strategy and Tactics 72.2 What Can Be Measured and What Cannot? 82.2.1 Examples Not Measurable 82.2.2 Shapes 92.2.3 Measurable by the Human Sensory System 102.2.4 Identifying and Selecting Measurable Factors 112.2.5 Intrusive Measurements 112.3 Beware Measuring Without Understanding: Warnings from History 122.4 How Does Experimental Work Differ from Theory and Analysis? 132.4.1 Logical Mode 132.4.2 Persistence 132.4.3 Resolution 132.4.4 Dimensionality 152.4.5 Similarity and Dimensional Analysis 152.4.6 Listening to Our Theoretician Compatriots 16Panel 2.1 Positive Consequences of the Reproducibility Crisis 17Panel 2.2 Selected Invitations to Experimental Research, Insights from Theoreticians 18Panel 2.3 Prepublishing Your Experiment Plan 212.4.7 Surveys and Polls 222.5 Uncertainty 232.6 Uncertainty Analysis 23References 24Homework 253 An Overview of Experiment Planning 273.1 Steps in an Experimental Plan 273.2 Iteration and Refinement 283.3 Risk Assessment/Risk Abatement 283.4 Questions to Guide Planning of an Experiment 29Homework 304 Identifying the Motivating Question 314.1 The Prime Need 31Panel 4.1 There’s a Hole in My Bucket 324.2 An Anchor and a Sieve 334.3 Identifying the Motivating Question Clarifies Thinking 334.3.1 Getting Started 334.3.2 Probe and Focus 344.4 Three Levels of Questions 354.5 Strong Inference 364.6 Agree on the Form of an Acceptable Answer 364.7 Specify the Allowable Uncertainty 374.8 Final Closure 37Reference 38Homework 385 Choosing the Approach 395.1 Laying Groundwork 395.2 Experiment Classifications 405.2.1 Exploratory 405.2.2 Identifying the Important Variables 405.2.3 Demonstration of System Performance 415.2.4 Testing a Hypothesis 415.2.5 Developing Constants for Predetermined Models 415.2.6 Custody Transfer and System Performance Certification Tests 425.2.7 Quality-Assurance Tests 425.2.8 Summary 435.3 Real or Simplified Conditions? 435.4 Single-Sample or Multiple-Sample? 43Panel 5.1 A Brief Summary of “Dissertation upon Roast Pig” 44Panel 5.2 Consider a Spherical Cow 445.5 Statistical or Parametric Experiment Design? 455.6 Supportive or Refutative? 475.7 The Bottom Line 47References 48Homework 486 Mapping for Safety, Operation, and Results 516.1 Construct Multiple Maps to Illustrate and Guide Experiment Plan 516.2 Mapping Prior Work and Proposed Work 516.3 Mapping the Operable Domain of an Apparatus 536.4 Mapping in Operator’s Coordinates 576.5 Mapping the Response Surface 596.5.1 Options for Organizing a Table 596.5.2 Options for Presenting the Response on a Scatter-Plot-Type Graph 61Homework 647 Refreshing Statistics 657.1 Reviving Key Terms to Quantify Uncertainty 657.1.1 Population 657.1.2 Sample 667.1.3 Central Value 677.1.4 Mean, μ or Ȳ 677.1.5 Residual 677.1.6 Variance, σ2 or S2 687.1.7 Degrees of Freedom, Df 687.1.8 Standard Deviation, σY or SY 687.1.9 Uncertainty of the Mean, δμ 697.1.10 Chi‐Squared, χ2 697.1.11 p‐Value 707.1.12 Null Hypothesis 707.1.13 F‐value of Fisher Statistic 717.2 The Data Distribution Most Commonly Encountered The Normal Distribution for Samples of Infinite Size 717.3 Account for Small Samples: The t‐Distribution 727.4 Construct Simple Models by Computer to Explain the Data 737.4.1 Basic Statistical Analysis of Quantitative Data 737.4.2 Model Data Containing Categorical and Quantitative Factors 757.4.3 Display Data Fit to One Categorical Factor and One Quantitative Factor 767.4.4 Quantify How Each Factor Accounts for Variation in the Data 767.5 Gain Confidence and Skill at Statistical Modeling Via the R Language 777.5.1 Model and Plot Results of a Single Variable Using the Example Data diceshoe.csv 777.5.2 Evaluate Alternative Models of the Example Data hiloy.csv 787.5.2.1 Inspect the Data 787.5.3 Grand Mean 787.5.4 Model by Groups: Group‐Wise Mean 787.5.5 Model by a Quantitative Factor 787.5.6 Model by Multiple Quantitative Factors 787.5.7 Allow Factors to Interact (So Each Group Gets Its Own Slope) 797.5.8 Include Polynomial Factors (a Statistical Linear Model Can Be Curved) 807.6 Report Uncertainty 807.7 Decrease Uncertainty (Improve Credibility) by Isolating Distinct Groups 817.8 Original Data, Summary, and R 82References 83Homework 838 Exploring Statistical Design of Experiments 878.1 Always Seeking Wiser Strategies 878.2 Evolving from Novice Experiment Design 878.3 Two‐Level and Three‐Level Factorial Experiment Plans 888.4 A Three‐Level, Three‐Factor Design 898.5 The Plackett–Burman 12‐Run Screening Design 938.6 Details About Analysis of Statistically Designed Experiments 958.6.1 Model Main Factors to Original Raw Data 958.6.2 Model Main Factors to Original Data Around Center of Each Factor 968.6.3 Model Including All Interaction Terms 978.6.4 Model Including Only Dominant Interaction Terms 978.6.5 Model Including Dominant Interaction Term Plus Quadratic Term 988.6.6 Model All Factors of Example 2, Centering Each Quantitative Factor 998.6.7 Refine Model of Example 2 Including Only Dominant Terms 1008.7 Retrospect of Statistical Design Examples 1018.8 Philosophy of Statistical Design 1018.9 Statistical Design for Conditions That Challenge Factorial Designs 1028.10 A Highly Recommended Tool for Statistical Design of Experiments 1038.11 More Tools for Statistical Design of Experiments 1038.12 Conclusion 103Further Reading 104Homework 1049 Selecting the Data Points 1079.1 The Three Categories of Data 1079.1.1 The Output Data 1079.1.2 Peripheral Data 1089.1.3 Backup Data 1089.1.4 Other Data You May Wish to Acquire 1089.2 Populating the Operating Volume 1099.2.1 Locating the Data Points Within the Operating Volume 1099.2.2 Estimating the Topography of the Response Surface 1099.3 Example from Velocimetry 1099.3.1 Sharpen Our Approach 1109.3.2 Lessons Learned from Velocimetry Example 1119.4 Organize the Data 1129.4.1 Keep a Laboratory Notebook 1129.4.2 Plan for Data Security 1129.4.3 Decide Data Format 1129.4.4 Overview Data Guidelines 1129.4.5 Reasoning Through Data Guidelines 1139.5 Strategies to Select Next Data Points 1149.5.1 Overview of Option 1: Default Strategy with Intensive Experimenter Involvement 1159.5.1.1 Choosing the Data Trajectory 1159.5.1.2 The Default Strategy: Be Bold 1159.5.1.3 Anticipate, Check, Course Correct 1169.5.1.4 Other Aspects to Keep in Mind 1169.5.1.5 Endpoints 1179.5.2 Reintroducing Gosset 1189.5.3 Practice Gosset Examples (from Gosset User Manual) 1199.6 Demonstrate Gosset for Selecting Data 1209.6.1 Status Quo of Experiment Planning and Execution (Prior to Selecting More Samples) 1209.6.1.1 Specified Motivating Question 1209.6.1.2 Identified Pertinent Candidate Factors 1219.6.1.3 Selected Initial Sample Points Using Plackett–Burman 1219.6.1.4 Executed the First 12 Runs at the PB Sample Conditions 1229.6.1.5 Analyzed Results. Identified Dominant First-Order Factors. Estimated First-Order Uncertainties of Factors 1239.6.1.6 Generated Draft Predictive Equation 1249.6.2 Use Gosset to Select Additional Data Samples 1259.6.2.1 Example Gosset Session: User Input to Select Next Points 1259.6.2.2 Example Gosset Session: How We Chose User Input 1269.6.2.3 Example Gosset Session: User Input Along with Gosset Output 1289.6.2.4 Example Gosset Session: Convert the Gosset Design to Operator Values 1319.6.2.5 Results of Example Gosset Session: Operator Plots of Total Experiment Plan 1329.6.2.6 Execute Stage Two of the Experiment Plan: User Plus Gosset Sample Points 1329.7 Use Gosset to Analyze Results 1339.8 Other Options and Features of Gosset 1339.9 Using Gosset to Find Local Extrema in a Function of Several Variables 1349.10 Summary 137Further Reading 137Homework 13710 Analyzing Measurement Uncertainty 14310.1 Clarifying Uncertainty Analysis 14310.1.1 Distinguish Error and Uncertainty 14410.1.1.1 Single-Sample vs. Multiple-Sample 14510.1.2 Uncertainty as a Diagnostic Tool 14610.1.2.1 What Can Uncertainty Analysis Tell You? 14610.1.2.2 What is Uncertainty Analysis Good For? 14810.1.2.3 Uncertainty Analysis Can Redirect a Poorly Conceived Experiment 14810.1.2.4 Uncertainty Analysis Improves the Quality of Your Work 14810.1.2.5 Slow Sampling and “Randomness” 14910.1.2.6 Uncertainty Analysis Makes Results Believable 15010.1.3 Uncertainty Analysis Aids Management Decision-Making 15010.1.3.1 Management’s Task: Dealing with Warranty Issues 15010.1.4 The Design Group’s Task: Setting Tolerances on Performance Test Repeatability 15210.1.5 The Performance Test Group’s Task: Setting the Tolerances on Measurements 15210.2 Definitions 15310.2.1 True Value 15310.2.2 Corrected Value 15310.2.3 Data Reduction Program 15310.2.4 Accuracy 15310.2.5 Error 15410.2.6 XXXX Error 15410.2.7 Fixed Error 15410.2.8 Residual Fixed Error 15410.2.9 Random Error 15410.2.10 Variable (but Deterministic) Error 15510.2.11 Uncertainty 15510.2.12 Odds 15510.2.13 Absolute Uncertainty 15510.2.14 Relative Uncertainty 15510.3 The Sources and Types of Errors 15610.3.1 Types of Errors: Fixed, Random, and Variable 15610.3.2 Sources of Errors: The Measurement Chain 15610.3.2.1 The Undisturbed Value 15810.3.2.2 The Available Value 15810.3.2.3 The Achieved Value 15810.3.2.4 The Observed Value 15910.3.2.5 The Corrected Value 15910.3.3 Specifying the True Value 16010.3.3.1 If the Achieved Value is Taken as the True Value 16010.3.3.2 If the Available Value is Taken as the True Value 16310.3.3.3 If the Undisturbed Value is Taken as the True Value 16610.3.3.4 If the Mixed Mean Gas Temperature is Taken as the True Value 16710.3.4 The Role of the End User 16710.3.4.1 The End-Use Equations Implicitly Define the True Value 16710.3.5 Calibration 16810.4 The Basic Mathematics 17010.4.1 The Root-Sum-Squared (RSS) Combination 17010.4.2 The Fixed Error in a Measurement 17110.4.3 The Random Error in a Measurement 17210.4.4 The Uncertainty in a Measurement 17310.4.5 The Uncertainty in a Calculated Result 17410.4.5.1 The Relative Uncertainty in a Result 17610.5 Automating the Uncertainty Analysis 17810.5.1 The Mathematical Basis 17810.5.2 Example of Uncertainty Analysis by Spreadsheet 17910.6 Single-Sample Uncertainty Analysis 18110.6.1 Assembling the Necessary Inputs 18410.6.2 Calculating the Uncertainty in the Result 18510.6.3 The Three Levels of Uncertainty: Zeroth-, First-, and Nth-Order 18510.6.3.1 Zeroth-Order Replication 18610.6.3.2 First-Order Replication 18710.6.3.3 Nth-Order Replication 18810.6.4 Fractional-Order Replication for Special Cases 18810.6.5 Summary of Single-Sample Uncertainty Levels 18910.6.5.1 Zeroth-Order 18910.6.5.2 First-Order 19010.6.5.3 Nth-Order 190References 190Further Reading 191Homework 19111 Using Uncertainty Analysis in Planning and Execution 19711.1 Using Uncertainty Analysis in Planning 19711.1.1 The Physical Situation and Energy Analysis 19811.1.2 The Steady‐State Method 19911.1.3 The Transient Method 20011.1.4 Reflecting on Assumptions Made During DRE Derivations 20111.2 Perform Uncertainty Analysis on the DREs 20211.2.1 Uncertainty Analysis: General Form 20211.2.2 Uncertainty Analysis of the Steady‐State Method 20311.2.3 Uncertainty Analysis – Transient Method 20411.2.4 Compare the Results of Uncertainty Analysis of the Methods 20511.2.5 What Does the Calculated Uncertainty Interval Mean? 20611.2.6 Cross‐Checking the Experiment 20711.2.7 Conclusions 20711.3 Using Uncertainty Analysis in Selecting Instruments 20811.4 Using Uncertainty Analysis in Debugging an Experiment 20911.4.1 Handling Overall Scatter 20911.4.2 Sources of Scatter 21011.4.3 Advancing Toward Calibration 21111.4.4 Selecting Thresholds 21211.4.5 Iterating Analysis 21211.4.6 Rechecking Situational Uncertainty 21211.5 Reporting the Uncertainties in an Experiment 21311.5.1 Progress in Uncertainty Reporting 21411.6 Multiple‐Sample Uncertainty Analysis 21411.6.1 Revisiting Single‐Sample and Multiple‐Sample Uncertainty Analysis 21411.6.2 Examples of Multiple‐Sample Uncertainty Analysis 21511.6.3 Fixed Error and Random Error 21611.7 Coordinate with Uncertainty Analysis Standards 21611.7.1 Describing Fixed and Random Errors in a Measurement 21711.7.2 The Bias Limit 21711.7.2.1 Fossilization 21811.7.2.2 Bias Limits 21811.7.3 The Precision Index 21911.7.4 The Number of Degrees of Freedom 22011.8 Describing the Overall Uncertainty in a Single Measurement 22011.8.1 Adjusting for a Single Measurement 22011.8.2 Describing the Overall Uncertainty in a Result 22111.8.3 Adding the Overall Uncertainty to Predictive Models 22211.9 Additional Statistical Tools and Elements 22211.9.1 Pooled Variance 22211.9.1.1 Student’s t‐Distribution – Pooled Examples 22311.9.2 Estimating the Standard Deviation of a Population from the Standard Deviation of a Small Sample: The Chi‐Squared χ2 Distribution 223References 225Homework 22612 Debugging an Experiment, Shakedown, and Validation 23112.1 Introduction 23112.2 Classes of Error 23112.3 Using Time-Series Analysis in Debugging 23212.4 Examples 23212.4.1 Gas Temperature Measurement 23212.4.2 Calibration of a Strain Gauge 23312.4.3 Lessons Learned from Examples 23412.5 Process Unsteadiness 23412.6 The Effect of Time-Constant Mismatching 23512.7 Using Uncertainty Analysis in Debugging an Experiment 23612.7.1 Calibration and Repeatability 23612.7.2 Stability and Baselining 23812.8 Debugging the Experiment via the Data Interpretation Program 23912.8.1 Debug the Experiment via the DIP 23912.8.2 Debug the Interface of the DIP 23912.8.3 Debug Routines in the DIP 24012.9 Situational Uncertainty 24113 Trimming Uncertainty 24313.1 Focusing on the Goal 24313.2 A Motivating Question for Industrial Production 24313.2.1 Agreed Motivating Questions for Industrial Example 24413.2.2 Quick Answers to Motivating Questions 24413.2.3 Challenge: Precheck Analysis and Answers 24513.3 Plackett–Burman 12-Run Results and Motivating Question #3 24513.4 PB 12-Run Results and Motivating Question #1 24713.4.1 Building a Predictive Model Equation from R-Language Linear Model 24813.4.2 Parsing the Dual Predictive Model Equation 24913.4.3 Uncertainty of the Intercept in the Dual Predictive Model Equation 25013.4.4 Mapping an Answer to Motivating Question #1 25113.4.5 Tentative Answers to Motivating Question #1 25213.5 Uncertainty Analysis of Dual Predictive Model and Motivating Question #2 25213.5.1 Uncertainty of the Constant in the Dual Predictive Model Equation 25213.5.2 Uncertainty of Other Factors in the Dual Predictive Model Equation 25313.5.3 Include All Coefficient Uncertainties in the Dual Predictive Model Equation 25413.5.4 Overall Uncertainty from All Factors in the Predictive Model Equation 25413.5.5 Improved Tentative Answers to Motivating Questions, Including Uncertainties 25613.5.6 Search for Improved Predictive Models 25613.6 The PB 12-Run Results and Individual Machine Models 25613.6.1 Individual Machine Predictive Model Equations 25813.6.2 Uncertainty of the Intercept in the Individual Predictive Model Equations 25813.6.3 Uncertainty of the Constant in the Individual Predictive Model Equations 25913.6.4 Uncertainty of Other Factors in the Individual Predictive Model Equation 25913.6.4.1 Uncertainties of Machine 1 25913.6.4.2 Uncertainties of Machine 2 26013.6.4.3 Including Instrument and Measurement Uncertainties 26013.6.5 Include All Coefficient Uncertainties in the Individual Predictive Model Equations 26013.6.6 Overall Uncertainty from All Factors in the Individual Predictive Model Equations 26113.6.7 Quick Overview of Individual Machine Performance Over the Operating Map 26213.7 Final Answers to All Motivating Questions for the PB Example Experiment 26313.7.1 Answers to Motivating Question #1 26313.7.2 Answers to Motivating Question #2 26313.7.3 Answers to Motivating Question #3 (Expanded from Section 13.3) 26313.7.4 Answers to Motivating Question #4 26413.7.5 Other Recommendations (to Our Client) 26413.8 Conclusions 265Homework 26614 Documenting the Experiment: Report Writing 26914.1 The Logbook 26914.2 Report Writing 26914.2.1 Organization of the Reports 27014.2.2 Who Reads What? 27014.2.3 Picking a Viewpoint 27114.2.4 What Goes Where? 27114.2.4.1 What Goes in the Abstract? 27214.2.4.2 What Goes in the Foreword? 27214.2.4.3 What Goes in the Objective? 27314.2.4.4 What Goes in the Results and Conclusions? 27314.2.4.5 What Goes in the Discussion? 27414.2.4.6 References 27414.2.4.7 Figures 27514.2.4.8 Tables 27614.2.4.9 Appendices 27614.2.5 The Mechanics of Report Writing 27614.2.6 Clear Language Versus “JARGON” 277Panel 14.1 The Turbo-Encabulator 27814.2.7 “Gobbledygook”: Structural Jargon 279Panel 14.2 U.S. Code, Title 18, No. 793 27914.2.8 Quantitative Writing 28114.2.8.1 Substantive Versus Descriptive Writing 281Panel 14.3 The Descriptive Bank Statement 28114.2.8.2 Zero-Information Statements 28114.2.8.3 Change 28214.3 International Organization for Standardization, ISO 9000 and other Standards 28214.4 Never Forget. Always Remember 282Appendix A: Distributing Variation and Pooled Variance 283A.1 Inescapable Distributions 283A.1.1 The Normal Distribution for Samples of Infinite Size 283A.1.2 Adjust Normal Distributions with Few Data: The Student’s t-Distribution 283A.2 Other Common Distributions 286A.3 Pooled Variance (Advanced Topic) 286Appendix B: Illustrative Tables for Statistical Design 289B.1 Useful Tables for Statistical Design of Experiments 289B.1.1 Ready-made Ordering for Randomized Trials 289B.1.2 Exhausting Sets of Two-Level Factorial Designs (≤ Five Factors) 289B.2 The Plackett–Burman (PB) Screening Designs 289Appendix C: Hand Analysis of a Two-Level Factorial Design 293C.1 The General Two-Level Factorial Design 293C.2 Estimating the Significance of the Apparent Factor Effects 298C.3 Hand Analysis of a Plackett–Burman (PB) 12-Run Design 299C.4 Illustrative Practice Example for the PB 12-Run Pattern 302C.4.1 Assignment: Find Factor Effects and the Linear Coefficients Absent Noise 302C.4.2 Assignment: Find Factor Effects and the Linear Coefficients with Noise 303C.5 Answer Key: Compare Your Hand Calculations 303C.5.1 Expected Results Absent Noise (compare C.4.1) 303C.5.2 Expected Results with Random Gaussian Noise (cf. C.4.2) 304C.6 Equations for Hand Calculations 305Appendix D: Free Recommended Software 307D.1 Instructions to Obtain the R Language for Statistics 307D.2 Instructions to Obtain LibreOffice 308D.3 Instructions to Obtain Gosset 308D.4 Possible Use of RStudio 309Index 311