Forecast Verification

Editors Ian T. Jolliff e and David B. Stephenson, both at the University of Exeter, UK

• List of Contributors xiPreface xiiiPreface to the First Edition xv1 Introduction 1Ian T. Jolliffe and David B. Stephenson1.1 A brief history and current practice 11.1.1 History 11.1.2 Current practice 21.2 Reasons for forecast verification and its benefits 31.3 Types of forecast and verification data 41.4 Scores, skill and value 51.4.1 Skill scores 61.4.2 Artificial skill 61.4.3 Statistical significance 71.4.4 Value added 81.5 Data quality and other practical considerations 81.6 Summary 92 Basic concepts 11Jacqueline M. Potts2.1 Introduction 112.2 Types of predictand 112.3 Exploratory methods 122.4 Numerical descriptive measures 152.5 Probability, random variables and expectations 202.6 Joint, marginal and conditional distributions 202.7 Accuracy, association and skill 222.8 Properties of verification measures 222.9 Verification as a regression problem 232.10 The Murphy–Winkler framework 252.11 Dimensionality of the verification problem 283 Deterministic forecasts of binary events 31Robin J. Hogan and Ian B. Mason3.1 Introduction 313.2 Theoretical considerations 333.2.1 Some basic descriptive statistics 333.2.2 A general framework for verification: the distributions-oriented approach 343.2.3 Performance measures in terms of factorizations of the joint distribution 373.2.4 Diagrams for visualizing performance measures 383.2.5 Case study: verification of cloud-fraction forecasts 413.3 Signal detection theory and the ROC 423.3.1 The signal detection model 433.3.2 The relative operating characteristic (ROC) 443.4 Metaverification: criteria for assessing performance measures 453.4.1 Desirable properties 453.4.2 Other properties 493.5 Performance measures 503.5.1 Overview of performance measures 513.5.2 Sampling uncertainty and confidence intervals for performance measures 553.5.3 Optimal threshold probabilities 57Acknowledgements 594 Deterministic forecasts of multi-category events 61Robert E. Livezey4.1 Introduction 614.2 The contingency table: notation, definitions, and measures of accuracy 624.2.1 Notation and definitions 624.2.2 Measures of accuracy 644.3 Skill scores 644.3.1 Desirable attributes 654.3.2 Gandin and Murphy equitable scores 664.3.3 Gerrity equitable scores 694.3.4 LEPSCAT 714.3.5 SEEPS 724.3.6 Summary remarks on scores 734.4 Sampling variability of the contingency table and skill scores 735 Deterministic forecasts of continuous variables 77Michel Déqué5.1 Introduction 775.2 Forecast examples 775.3 First-order moments 795.3.1 Bias 795.3.2 Mean Absolute Error 805.3.3 Bias correction and artificial skill 815.3.4 Mean absolute error and skill 815.4 Second- and higher-order moments 825.4.1 Mean Squared Error 825.4.2 MSE skill score 825.4.3 MSE of scaled forecasts 835.4.4 Correlation 845.4.5 An example: testing the ‘limit of predictability’ 865.4.6 Rank correlations 875.4.7 Comparison of moments of the marginal distributions 885.4.8 Graphical summaries 905.5 Scores based on cumulative frequency 915.5.1 Linear Error in Probability Space (LEPS) 915.5.2 Quantile-quantile plots 925.5.3 Conditional quantile plots 925.6 Summary and concluding remarks 946 Forecasts of spatial fields 95Barbara G. Brown, Eric Gilleland and Elizabeth E. Ebert6.1 Introduction 956.2 Matching methods 966.3 Traditional verification methods 976.3.1 Standard continuous and categorical approaches 976.3.2 S1 and anomaly correlation 986.3.3 Distributional methods 996.4 Motivation for alternative approaches 1006.5 Neighbourhood methods 1036.5.1 Comparing neighbourhoods of forecasts and observations 1046.5.2 Comparing spatial forecasts with point observations 1046.6 Scale separation methods 1056.7 Feature-based methods 1086.7.1 Feature-matching techniques 1086.7.2 Structure-Amplitude-Location (SAL) technique 1106.8 Field deformation methods 1116.8.1 Location metrics 1116.8.2 Field deformation 1126.9 Comparison of approaches 1136.10 New approaches and applications: the future 1146.11 Summary 1167 Probability forecasts 119Jochen Broecker7.1 Introduction 1197.2 Probability theory 1207.2.1 Basic concepts from probability theory 1207.2.2 Probability forecasts, reliability and sufficiency 1217.3 Probabilistic scoring rules 1227.3.1 Definition and properties of scoring rules 1227.3.2 Commonly used scoring rules 1247.3.3 Decomposition of scoring rules 1257.4 The relative operating characteristic (ROC) 1267.5 Evaluation of probabilistic forecasting systems from data 1287.5.1 Three examples 1287.5.2 The empirical ROC 1307.5.3 The empirical score as a measure of performance 1307.5.4 Decomposition of the empirical score 1317.5.5 Binning forecasts and the leave-one-out error 1327.6 Testing reliability 1347.6.1 Reliability analysis for forecast A: the reliability diagram 1347.6.2 Reliability analysis for forecast B: the chi-squared test 1367.6.3 Reliability analysis for forecast C: the PIT 138Acknowledgements 1398 Ensemble forecasts 141Andreas P. Weigel8.1 Introduction 1418.2 Example data 1428.3 Ensembles interpreted as discrete samples 1438.3.1 Reliability of ensemble forecasts 1448.3.2 Multidimensional reliability 1528.3.3 Discrimination 1578.4 Ensembles interpreted as probabilistic forecasts 1598.4.1 Probabilistic interpretation of ensembles 1598.4.2 Probabilistic skill metrics applied to ensembles 1608.4.3 Effect of ensemble size on skill 1638.5 Summary 1669 Economic value and skill 167David S. Richardson9.1 Introduction 1679.2 The cost/loss ratio decision model 1689.2.1 Value of a deterministic binary forecast system 1699.2.2 Probability forecasts 1729.2.3 Comparison of deterministic and probabilistic binary forecasts 1749.3 The relationship between value and the ROC 1759.4 Overall value and the Brier Skill Score 1789.5 Skill, value and ensemble size 1809.6 Applications: value and forecast users 1829.7 Summary 18310 Deterministic forecasts of extreme events and warnings 185Christopher A.T. Ferro and David B. Stephenson10.1 Introduction 18510.2 Forecasts of extreme events 18610.2.1 Challenges 18610.2.2 Previous studies 18710.2.3 Verification measures for extreme events 18910.2.4 Modelling performance for extreme events 19110.2.5 Extreme events: summary 19410.3 Warnings 19510.3.1 Background 19510.3.2 Format of warnings and observations for verification 19610.3.3 Verification of warnings 19710.3.4 Warnings: summary 200Acknowledgements 20111 Seasonal and longer-range forecasts 203Simon J. Mason11.1 Introduction 20311.2 Forecast formats 20411.2.1 Deterministic and probabilistic formats 20411.2.2 Defining the predictand 20611.2.3 Inclusion of climatological forecasts 20611.3 Measuring attributes of forecast quality 20711.3.1 Skill 20711.3.2 Other attributes 21511.3.3 Statistical significance and uncertainty estimates 21611.4 Measuring the quality of individual forecasts 21711.5 Decadal and longer-range forecast verification 21811.6 Summary 22012 Epilogue: new directions in forecast verification 221Ian T. Jolliffe and David B. Stephenson12.1 Introduction 22112.2 Review of key concepts 22112.3 Forecast evaluation in other disciplines 22312.3.1 Statistics 22312.3.2 Finance and economics 22512.3.3 Medical and clinical studies 22612.4 Current research and future directions 228Acknowledgements 230Appendix: Verification Software 231Matthew PocernichA.1 What is good software? 231A.1.1 Correctness 232A.1.2 Documentation 232A.1.3 Open source/closed source/commercial 232A.1.4 Large user base 232A.2 Types of verification users 232A.2.1 Students 233A.2.2 Researchers 233A.2.3 Operational forecasters 233A.2.4 Institutional use 233A.3 Types of software and programming languages 233A.3.1 Spreadsheets 235A.3.2 Statistical programming languages 235A.4 Institutional supported software 238A.4.1 Model Evaluation Tool (MET) 238A.4.2 Ensemble Verification System (EVS) 239A.4.3 EUMETCAL Forecast Verification Training Module 239A.5 Displays of verification information 239A.5.1 National Weather Service Performance Management 240A.5.2 Forecast Evaluation Tool 240Glossary 241References 251Index 267