Uncertain Judgements

Professor Anthony O’Hagan is the Director of The Centre for Bayesian Statistics in Health Economics at the University of Sheffield. The Centre is a collaboration between the Department of Probability and Statistics and the School of Health and Related Research (ScHARR). The Department of Probability and Statistics is internationally respected for its research in Bayesian statistics, while ScHARR is one of the leading UK centres for economic evaluation. Prof O’Hagan is an internationally leading expert in Bayesian Statistics.Co-authors:Professor Paul Gathwaite – Open University, Prof of Statistics, Maths and ComputingDr Jeremy Oakley – Sheffield UniversityProfessor John Brazier – Director of Health Economics Group, University of SheffieldDr Tim Rakow – University of Essex, Psychology DepartmentDr Alireza Daneshkhah – University of Sheffield, Medical Statistics DepartmentDr Jim Chilcott - School of Health Research, University of Sheffield, Department of OR

• Preface xi1 Fundamentals of Probability and Judgement 11.1 Introduction 11.2 Probability and elicitation 11.2.1 Probability 11.2.2 Random variables and probability distributions 31.2.3 Summaries of distributions 51.2.4 Joint distributions 71.2.5 Bayes’ Theorem 81.2.6 Elicitation 91.3 Uncertainty and the interpretation of probability 101.3.1 Aleatory and epistemic uncertainty 101.3.2 Frequency and personal probabilities 111.3.3 An extended example 121.3.4 Implications for elicitation 141.4 Elicitation and the psychology of judgement 141.4.1 Judgement – absolute or relative? 151.4.2 Beyond perception 181.4.3 Implications for elicitation 201.5 Of what use are such judgements? 201.5.1 Normative theories of probability 211.5.2 Coherence 211.5.3 Do elicited probabilities have the desired interpretation? 221.6 Conclusions 241.6.1 Elicitation practice 241.6.2 Research questions 242 The Elicitation Context 252.1 How and who? 252.1.1 Choice of format 252.1.2 What is an expert? 262.2 The elicitation process 272.2.1 Roles within the elicitation process 282.2.2 A model for the elicitation process 282.3 Conventions in Chapters 3 to 9 312.4 Conclusions 312.4.1 Elicitation practice 312.4.2 Research question 313 The Psychology of Judgement Under Uncertainty 333.1 Introduction 333.1.1 Why psychology? 333.1.2 Chapter overview 343.2 Understanding the task and the expert 353.2.1 Cognitive capabilities: the proper view of human information processing? 353.2.2 Constructive processes: the proper view of the process? 363.3 Understanding research on human judgement 373.3.1 Experts versus the rest: the proper focus of research? 373.3.2 Early research on subjective probability: ‘conservatism’ in Bayesian probability revision 383.4 The heuristics and biases research programme 383.4.1 Availability 393.4.2 Representativeness 413.4.3 Do frequency representations remove the biases attributed to availability and representativeness? 463.4.4 Anchoring-and-adjusting 473.4.5 Support theory 493.4.6 The affect heuristic 513.4.7 Critique of the heuristics and biases approach 523.5 Experts and expertise 523.5.1 The heuristics and biases approach 533.5.2 The cognitive science approach 533.5.3 ‘The middle way’ 543.6 Three meta-theories of judgement 553.6.1 The cognitive continuum 563.6.2 The inside versus the outside view 563.6.3 The naive intuitive statistician metaphor 583.7 Conclusions 583.7.1 Elicitation practice 583.7.2 Research questions 594 The Elicitation of Probabilities 614.1 Introduction 614.2 The calibration of subjective probabilities 624.2.1 Research methods in calibration research 674.2.2 Calibration research: general findings 684.2.3 Calibration research in applied settings 724.2.4 A case study in probability judgement: calibration research in medicine 744.3 The calibration of subjective probabilities: theories and explanations 774.3.1 Explanations of probability judgement in calibration tasks 774.3.2 Theories of the calibration of subjective probabilities 794.4 Representations and methods 824.4.1 Different modes for representing uncertainty 834.4.2 Different formats for eliciting responses 874.4.3 Key lessons 894.5 Debiasing 894.5.1 General principles for debiasing judgement 904.5.2 Managing noise 914.5.3 Redressing insufficient regressiveness in prediction 924.5.4 A caveat concerning post hoc corrections 944.6 Conclusions 954.6.1 Elicitation practice 954.6.2 Research questions 955 Eliciting Distributions – General 975.1 From probabilities to distributions 975.1.1 From a few to infinity 985.1.2 Summaries 995.1.3 Fitting 1005.1.4 Overview 1005.2 Eliciting univariate distributions 1005.2.1 Summaries based on probabilities 1005.2.2 Proportions 1045.2.3 Other summaries 1055.3 Eliciting multivariate distributions 1075.3.1 Structuring 1075.3.2 Eliciting association 1085.3.3 Joint and conditional probabilities 1115.3.4 Regression 1125.3.5 Many variables 1135.4 Uncertainty and imprecision 1145.4.1 Quantifying elicitation error 1145.4.2 Sensitivity analysis 1155.4.3 Feedback and overfitting 1165.5 Conclusions 1185.5.1 Elicitation practice 1185.5.2 Research questions 1196 Eliciting and Fitting a Parametric Distribution 1216.1 Introduction 1216.2 Outline of this chapter 1226.3 Eliciting opinion about a proportion 1246.4 Eliciting opinion about a general scalar quantity 1326.5 Eliciting opinion about a set of proportions 1376.6 Eliciting opinion about the parameters of a multivariate normal distribution 1396.7 Eliciting opinion about the parameters of a linear regression model 1426.8 Eliciting opinion about the parameters of a generalised linear model 1456.9 Elicitation methods for other problems 1476.10 Deficiencies in existing research 1496.11 Conclusions 1506.11.1 Elicitation practice 1506.11.2 Research questions 1517 Eliciting Distributions – Uncertainty and Imprecision 1537.1 Introduction 1537.2 Imprecise probabilities 1537.3 Incomplete information 1567.4 Summary 1607.5 Conclusions 1607.5.1 Elicitation practice 1607.5.2 Research questions 1608 Evaluating Elicitation 1618.1 Introduction 1618.1.1 Good elicitation 1618.1.2 Inaccurate knowledge 1618.1.3 Automatic calibration 1628.1.4 Lessons of the psychological literature 1638.1.5 Outline of this chapter 1638.2 Scoring rules 1638.2.1 Scoring rules for discrete probability distributions 1658.2.2 Scoring rules for continuous probability distributions 1698.3 Coherence, feedback and overfitting 1718.3.1 Coherence and calibration 1718.3.2 Feedback and overfitting 1738.4 Conclusions 1768.4.1 Elicitation practice 1768.4.2 Research questions 1779 Multiple Experts 1799.1 Introduction 1799.2 Mathematical aggregation 1809.2.1 Bayesian methods 1809.2.2 Opinion pooling 1819.2.3 Cooke’s method 1849.2.4 Performance of mathematical aggregation 1859.3 Behavioural aggregation 1869.3.1 Group elicitation 1869.3.2 Other methods of behavioural aggregation 1889.3.3 Performance of behavioural methods 1909.4 Discussion 1909.5 Elicitation practice 1919.6 Research questions 19110 Published Examples of the Formal Elicitation of Expert Opinion 19310.1 Some applications 19310.2 An example of an elicitation interview – eliciting engine sales 19310.3 Medicine 19510.3.1 Diagnosis and treatment decisions 19510.3.2 Clinical trials 19910.3.3 Survival analysis 20110.3.4 Clinical psychology 20210.4 The nuclear industry 20410.5 Veterinary science 20610.6 Agriculture 20710.7 Meteorology 20810.8 Business studies, economics and finance 20910.9 Other professions 21210.10 Other examples of the elicitation of subjective probabilities 21311 Guidance on Best Practice 21712 Areas for Research 223Glossary 227Bibliography 267Author Index 307Index 313

“This book, written by a group of expert statisticians and psychologists, provides an introduction to the subject and a detailed overview of the existing literature. The book guides the reader through the design of an elicitation method and details examples from a cross section of literature in the statistics, psychology, engineering and health sciences disciplines.”  (Zentralblatt Math, 1 August 2013)"This is an interesting, well-written book that will be valuable to any decision maker who much rely on expert judgments, any statistician who uses Bayesian statistics, and any researcher who wishes to understand the field of elicitation." (Journal of the American Statistical Association, March 2009)"This book provides an excellent introduction and working reference to the subject of its title and should be an invaluable aid to producers and consumers of expert opinion." (Biometrics, September 2008)"I recommend 'Uncertain Judgements' as an excellent source for a wide variety of research." (Psychometrika, March 2008)“…will be of interest to those who are concerned with or interested primarily in the practicalities of modeling expert judgement and opinion.” (International Journal of Marketing, January 2007)