Cochrane Handbook for Systematic Reviews of Diagnostic Test Accuracy

Jonathan J. Deeks is Professor at the Institute of Applied Health Research at the University of Birmingham in the United Kingdom. He is a member of Cochrane’s Diagnostic Test Accuracy Editorial team. Patrick M. Bossuyt is Professor at the Amsterdam UMC, University of Amsterdam, Department of Epidemiology and Data Science in the Netherlands. Mariska M. Leeflang is Associate Professor at the Amsterdam UMC, University of Amsterdam, Department of Epidemiology and Data Science in the Netherlands. She is a Convenor of Cochrane’s Screening and Diagnostic Tests Methods Group. Yemisi Takwoingi is Professor at the Institute of Applied Health Research at the University of Birmingham, UK and a Convenor of Cochrane’s Screening and Diagnostic Tests Methods Group.

• Contributors xvPreface xixPart One About Cochrane Reviews of diagnostic test accuracy 11 Planning a Cochrane Review of diagnostic test accuracy 31.1 Introduction 41.2 Why do a systematic review of test accuracy? 41.3 Undertaking a Cochrane Review of diagnostic test accuracy 51.3.1 The role of the Diagnostic Test Accuracy Editorial Team 51.3.2 Expectations for the conduct and reporting of Cochrane Reviews of diagnostic test accuracy 51.3.3 Data management and quality assurance 61.3.4 Keeping the Review up to date 61.4 Proposing a new Cochrane Review of diagnostic test accuracy 61.5 Cochrane Protocols 71.6 The author team 111.6.1 The importance of the team 111.6.2 Criteria for authorship 121.6.3 Incorporating relevant perspectives and stakeholder involvement 121.7 Resources and support 131.7.1 Identifying resources and support 131.7.2 Funding and conflicts of interest 141.7.3 Training 141.7.4 Software 151.8 Chapter information 151.9 References 16Part Two Introducing test accuracy 192 Evaluating medical tests 212.1 Introduction 212.2 Types of medical tests 222.3 Test accuracy 232.4 How do diagnostic tests affect patient outcomes? 242.4.1 Direct test effects 252.4.2 Altering clinical decisions and actions 252.4.3 Changes to time frames and populations 252.4.4 Influencing patient and clinician perceptions 262.5 Evaluations of test accuracy during test development 262.5.1 Evaluations of accuracy during biomarker discovery 262.5.2 Early evaluations of test accuracy 272.5.3 Clinical evaluations of test accuracy 282.6 Other purposes of medical testing 282.6.1 Predisposition 292.6.2 Risk stratification 292.6.3 Screening 292.6.4 Staging 292.6.5 Prognosis 302.6.6 Treatment selection 302.6.7 Treatment efficacy 312.6.8 Therapeutic monitoring 312.6.9 Surveillance for progression or recurrence 312.7 Chapter information 322.8 References 323 Understanding the design of test accuracy studies 353.1 Introduction 353.2 The basic design for a test accuracy study 363.3 Multiple groups of participants 393.4 Multiple reference standards 423.5 More on reference standards 443.5.1 Delayed verification 443.5.2 Composite reference standard 443.5.3 Panel- based reference 443.5.4 Latent class analysis 453.5.5 Gold standard 453.5.6 Clinical reference standard 453.6 Comparative test accuracy studies 453.6.1 Paired comparative accuracy study 463.6.2 Randomized comparative accuracy study 463.6.3 Non- randomized comparative accuracy study 473.7 Additional aspects of study designs 473.7.1 Prospective versus retrospective 483.7.2 Pragmatic versus explanatory 483.8 Concluding remarks 493.9 Chapter information 493.10 References 504 Understanding test accuracy measures 534.1 Introduction 534.2 Types of test data 544.3 Inconclusive index test results 554.4 Target condition 564.5 Analysis of a primary test accuracy study 564.5.1 Sensitivity and specificity 574.5.2 Predictive values 584.5.3 Proportion with the target condition 584.5.4 Pre- test and post- test probabilities 594.5.5 Interpretation of sensitivity, specificity and predictive values 594.5.6 Confidence intervals 604.5.7 Other test accuracy measures 614.6 Positivity thresholds 644.7 Receiver operating characteristic curves 664.8 Analysis of a comparative accuracy study 684.9 Chapter information 714.10 References 72Part Three Methods and presentation of systematic reviews of test accuracy 735 Defining the review question 755.1 Introduction 755.2 Aims of systematic reviews of test accuracy 765.2.1 Investigations of heterogeneity 775.3 Identifying the clinical problem 775.3.1 Role of a new test 775.3.2 Defining the clinical pathway 805.3.3 Unclear and multiple clinical pathways 835.4 Defining the review question 845.4.1 Population 845.4.2 Index test(s) 855.4.3 Target condition 855.4.4 The review question: PIT 865.4.5 From review question to objectives 865.4.6 Broad versus narrow questions 875.5 Defining eligibility criteria 885.5.1 Types of studies 885.5.2 Participants 895.5.3 Index test(s) 905.5.4 Target condition 915.5.5 Reference standard 925.6 Chapter information 935.7 References 936 Searching for and selecting studies 976.1 Introduction 986.2 Searching for studies 986.2.1 Working in partnership 1006.2.2 Advice for review teams that do not include an information specialist 1016.3 Sources to search 1016.3.1 Bibliographic databases 1016.3.1.1 MEDLINE, PubMed and Embase 1026.3.1.2 National and regional databases 1036.3.1.3 Subject- specific databases 1036.3.1.4 Dissertations and theses databases 1046.3.2 Additional sources to search 1046.3.2.1 Related reviews, guidelines and reference lists as sources of studies 1056.3.2.2 Handsearching 1056.3.2.3 Forward citation searching and co- citation searching 1056.3.2.4 Web searching 1066.3.2.5 Grey literature databases 1076.3.2.6 Trial registries 1076.3.2.7 Contacting colleagues, study authors and manufacturers 1086.4 Designing search strategies 1086.4.1 Structuring the search strategy 1096.4.2 Controlled vocabulary and text words 1106.4.3 Text word or keyword searching 1126.4.4 Search filters 1136.4.5 Language, date and type of document restrictions 1136.4.6 Identifying fraudulent studies, other retracted publications, errata and comments 1146.4.7 Minimizing the risk of bias through search methods 1146.5 Documenting and reporting the search process 1156.5.1 Documenting the search process 1166.5.2 Reporting the search process 1166.5.2.1 Reporting the search process in the protocol 1166.5.2.2 Reporting the search process in the review 1176.6 Selecting relevant studies 1196.6.1 Examine full- text reports for compliance of studies with eligibility criteria 1206.7 Future developments in literature searching and selection 1216.8 Chapter information 1216.9 References 1227 Collecting data 1317.1 Introduction 1327.2 Sources of data 1327.2.1 Studies (not reports) as the unit of interest 1337.2.2 Correspondence with investigators 1347.3 What data to collect 1357.3.1 What are data? 1357.3.2 Study methods (participant recruitment and sampling) 1377.3.3 Participant characteristics and setting 1387.3.4 Index test(s) 1397.3.5 Target condition and reference standard 1407.3.6 Flow and timing 1407.3.7 Extracting study results and converting to the desired format 1417.3.7.1 Obtaining 2×2 data from accuracy measures 1417.3.7.2 Using global measures 1447.3.7.3 Challenges defining reference standard positive and negative: strategies when there are more than two categories 1457.3.7.4 Challenges defining index test positive and negative: inconclusive results 1457.3.7.5 Challenges defining index test positive and negative: test failures 1477.3.7.6 Challenges defining index test positive and negative: dealing with multiple thresholds and extracting data from ROC curves or other graphics 1477.3.7.7 Extracting data from figures with software 1487.3.7.8 Corrections for missing data: adjusting for partial verification bias 1487.3.7.9 Multiple index tests from the same study 1487.3.7.10 Subgroups of patients 1507.3.7.11 Individual patient data 1507.3.7.12 Extracting covariates 1517.3.8 Other information to collect 1517.4 Data collection tools 1527.4.1 Rationale for data collection forms 1527.4.2 Considerations in selecting data collection tools 1527.4.3 Design of a data collection form 1547.5 Extracting data from reports 1577.5.1 Introduction 1577.5.2 Who should extract data? 1577.5.3 Training data extractors 1587.5.4 Extracting data from multiple reports of the same study 1587.5.5 Reliability and reaching consensus 1597.5.6 Suspicions of scientific misconduct 1597.5.7 Key points in planning and reporting data extraction 1607.6 Managing and sharing data and tools 1607.7 Chapter information 1637.8 References 1648 Assessing risk of bias and applicability 1698.1 Introduction 1708.2 Understanding bias and applicability 1718.2.1 Bias and imprecision 1718.2.2 Bias versus applicability 1718.2.3 Biases in test accuracy studies: empirical evidence 1728.3 Quadas- 2 1738.3.1 Background 1738.3.2 Risk- of- bias assessment 1738.3.3 Applicability assessment 1748.3.4 Using and tailoring QUADAS- 2 1748.3.5 Flow diagram 1748.3.6 Performing the QUADAS- 2 assessment 1758.4 Domain 1: Participant selection 1768.4.1 Participant selection: risk- of- bias signalling questions (QUADAS- 2) 1768.4.2 Participant selection: additional signalling questions for comparative accuracy studies (QUADAS- C) 1788.4.3 Participant selection: concerns regarding applicability 1818.5 Domain 2: Index test 1828.5.1 Index test: risk- of- bias signalling questions (QUADAS- 2) 1828.5.2 Index test: additional signalling questions for comparative accuracy studies (QUADAS- C) 1838.5.3 Index test: concerns regarding applicability 1868.6 Domain 3: Reference standard 1878.6.1 Reference standard: risk- of- bias signalling questions (QUADAS- 2) 1878.6.2 Reference standard: additional signalling questions for comparative accuracy studies (QUADAS- C) 1888.6.3 Reference standard: concerns regarding applicability 1898.7 Domain 4: Flow and timing 1918.7.1 Flow and timing: risk- of- bias signalling questions (QUADAS- 2) 1918.7.2 Flow and timing: additional signalling questions for comparative accuracy studies (QUADAS- C) 1938.8 Presentation of risk- of- bias and applicability assessments 1968.9 Narrative summary of risk- of- bias and applicability assessments 1978.10 Chapter information 1978.11 References 1989 Understanding meta- analysis 2039.1 Introduction 2039.1.1 Aims of meta- analysis for systematic reviews of test accuracy 2049.1.2 When not to use a meta- analysis in a review 2049.1.3 How does meta- analysis of diagnostic test accuracy differ from metaanalysis of interventions? 2059.1.4 Questions that can be addressed in test accuracy analyses 2069.1.4.1 What is the accuracy of a test? 2069.1.4.2 How does the accuracy vary with clinical and methodological characteristics? 2069.1.4.3 How does the accuracy of two or more tests compare? 2069.1.5 Planning the analysis 2079.2 Graphical and tabular presentation 2089.2.1 Coupled forest plots 2089.2.2 Summary ROC plots 2089.2.3 Linked SROC plots 2109.2.3.1 Example 1: Anti- CCP for the diagnosis of rheumatoid arthritis – descriptive plots 2109.2.4 Tables of results 2119.3 Meta- analytical summaries 2119.3.1 Should I estimate an SROC curve or a summary point? 2129.3.2 Heterogeneity 2149.4 Fitting hierarchical models 2159.4.1 Bivariate model 2169.4.2 Example 1 continued: anti- CCP for the diagnosis of rheumatoid arthritis 2179.4.3 The Rutter and Gatsonis HSROC model 2199.4.4 Example 2: Rheumatoid factor as a marker for rheumatoid arthritis 2209.4.5 Data reported at multiple thresholds per study 2219.4.6 Investigating heterogeneity 2229.4.6.1 Criteria for model selection 2239.4.6.2 Heterogeneity and regression analysis using the bivariate model 2239.4.6.3 Example 1 continued: Investigation of heterogeneity in diagnostic performance of anti- CCP 2249.4.6.4 Heterogeneity and regression analysis using the Rutter and Gatsonis HSROC model 2279.4.6.5 Example 2 continued: Investigating heterogeneity in diagnostic accuracy of rheumatoid factor (RF) 2289.4.7 Comparing index tests 2309.4.7.1 Test comparisons based on all available studies 2309.4.7.2 Test comparisons using the bivariate model 2319.4.7.3 Example 3: CT versus MRI for the diagnosis of coronary artery disease 2329.4.7.4 Test comparisons using the Rutter and Gatsonis HSROC model 2349.4.7.5 Test comparison based on studies that directly compare tests 2359.4.7.6 Example 3 continued: CT versus MRI for the diagnosis of coronary artery disease 2369.4.8 Approaches to analysis with small numbers of studies 2389.4.9 Sensitivity analysis 2399.5 Special topics 2419.5.1 Imperfect reference standard 2419.5.2 Investigating and handling verification bias 2419.5.3 Investigating and handling publication bias 2429.5.4 Developments in meta- analysis for systematic reviews of test accuracy 2439.6 Chapter information 2439.7 References 24410 Undertaking meta- analysis 24910.1 Introduction 24910.2 Estimation of a summary point 25110.2.1 Fitting the bivariate model using SAS 25110.2.2 Fitting the bivariate model using Stata 25310.2.3 Fitting the bivariate model using R 25610.2.4 Bayesian estimation of the bivariate model 26110.2.4.1 Specification of the bivariate model in rjags 26110.2.4.2 Monitoring convergence 26310.2.4.3 Summary statistics 26410.2.4.4 Generating an SROC plot 26510.2.4.5 Sensitivity analyses 26610.3 Estimation of a summary curve 26610.3.1 Fitting the HSROC model using SAS 26810.3.2 Bayesian estimation of the HSROC model 26810.3.2.1 Specification of the HSROC model in rjags 26810.3.2.2 Monitoring convergence 27010.3.2.3 Summary statistics and SROC plot 27110.3.2.4 Sensitivity analyses 27210.4 Comparison of summary points 27210.4.1 Fitting the bivariate model in SAS to compare summary points 27410.4.2 Fitting the bivariate model in Stata to compare summary points 28010.4.3 Fitting the bivariate model in R to compare summary points 28410.4.4 Bayesian inference for comparing summary points 28710.4.4.1 Summary statistics 28910.5 Comparison of summary curves 29110.5.1 Fitting the HSROC model in SAS to compare summary curves 29210.5.2 Bayesian estimation of the HSROC model for comparing summary curves 29410.5.2.1 Monitoring convergence 29510.5.2.2 Summary statistics 29510.6 Meta- analysis of sparse data and a typical data sets 29610.6.1 Facilitating convergence 29710.6.2 Simplifying hierarchical models 30110.7 Meta- analysis with multiple thresholds per study 30510.7.1 Meta- analysis of multiple thresholds with R 30610.7.2 Meta- analysis of multiple thresholds with rjags 31110.8 Meta- analysis with imperfect reference standard: latent class meta- analysis 31610.8.1 Specification of the latent class bivariate meta- analysis model in rjags 31610.8.2 Monitoring convergence 31710.8.3 Summary statistics and summary ROC plot 31710.8.4 Sensitivity analyses 32010.9 Concluding remarks 32110.10 Chapter information 32110.11 References 32211 Presenting findings 32711.1 Introduction 32711.2 Results of the search 32811.3 Description of included studies 32811.4 Methodological quality of included studies 32911.5 Individual and summary estimates of test accuracy 32911.5.1 Presenting results from included studies 33011.5.2 Presenting summary estimates of sensitivity and specificity 33011.5.3 Presenting SROC curves 33011.5.4 Describing uncertainty in summary statistics 33211.5.5 Describing heterogeneity in summary statistics 33311.6 Comparisons of test accuracy 33311.6.1 Comparing tests using summary points 33311.6.2 Comparing tests using SROC curves 33411.6.3 Interpretation of confidence intervals for differences in test accuracy 33611.7 Investigations of sources of heterogeneity 33611.8 Re- expressing summary estimates numerically 34011.8.1 Frequencies 34011.8.2 Predictive values 34111.8.3 Likelihood ratios 34411.9 Presenting findings when meta- analysis cannot be performed 34411.10 Chapter information 34611.11 References 34712 Drawing conclusions 34912.1 Introduction 34912.2 ‘Summary of findings’ tables 35012.3 Assessing the strength of the evidence 35212.3.1 Key issues to consider when assessing the strength of the evidence 35212.3.1.1 How valid are the summary estimates? 35912.3.1.2 How applicable are the summary estimates? 35912.3.1.3 How heterogeneous are the individual study estimates? 35912.3.1.4 How precise are the summary estimates? 36012.3.1.5 How complete is the body of evidence? 36112.3.1.6 Were index test comparisons made between or within primary studies? 36212.4 GRADE approach for assessing the certainty of evidence 36212.4.1 GRADE domains for assessing certainty of evidence for test accuracy 36312.4.1.1 Risk of bias 36312.4.1.2 Indirectness (applicability) 36312.4.1.3 Inconsistency (heterogeneity) 36412.4.1.4 Imprecision 36512.4.1.5 Publication bias 36512.5 Summary of main results in the Discussion section 36512.6 Strengths and weaknesses of the review 36612.6.1 Strengths and weaknesses of included studies 36612.6.2 Strengths and weaknesses of the review 36712.6.2.1 Strengths and weaknesses due to the search and selection process 36712.6.2.2 Strengths and weaknesses due to methodological quality assessment and data extraction 36712.6.2.3 Weaknesses due to the review analyses 36812.6.2.4 Direct and indirect comparisons 36812.6.3 Comparisons with previous research 36912.7 Applicability of findings to the review question 36912.8 Drawing conclusions 36912.8.1 Implications for practice 37012.8.2 Implications for research 37312.9 Chapter information 37412.10 References 37413 Writing a plain language summary 37713.1 Introduction 37713.2 Audience and writing style 37813.3 Contents and structure of a plain language summary 37913.3.1 Title 38013.3.2 Key messages 38013.3.3 ‘Why is improving [ ] diagnosis important?’ 38113.3.4 ‘What is the [ ] test?’ 38213.3.5 What did we want to find out? 38213.3.6 What did we do? 38313.3.7 What did we find? 38313.3.7.1 Describing the included studies 38313.3.7.2 Presenting information on test accuracy 38413.3.7.3 Presenting single estimates of accuracy 38513.3.7.4 Presenting multiple estimates of accuracy: two index tests 38613.3.7.5 Presenting multiple estimates of accuracy: more than two index tests 38713.3.7.6 When presenting a numerical summary of test accuracy is not appropriate 38713.3.7.7 Graphical illustration of test accuracy results 38813.3.8 What are the limitations of the evidence? 39113.3.9 How up to date is this evidence? 39213.4 Chapter information 39213.5 References 39313.6 Appendix: Additional example plain language summary 394Index 399