Evaluation Design for Complex Global Initiatives

Every year, public and private funders spend many billions of dollars on large-scale, complex, multi-national health initiatives. The only way to know whether these initiatives are achieving their objectives is through evaluations that examine the links between program activities and desired outcomes. Investments in such evaluations, which, like the initiatives being evaluated, are carried out in some of the world's most challenging settings, are a relatively new phenomenon. In the last five years, evaluations have been conducted to determine the effects of some of the world's largest and most complex multi-national health initiatives. Evaluation Design for Complex Global Initiatives is the summary of a workshop convened by the Institute of Medicine in January 2014 to explore these recent evaluation experiences and to consider the lessons learned from how these evaluations were designed, carried out, and used. The workshop brought together more than 100 evaluators, researchers in the field of evaluation science, staff involved in implementing large-scale health programs, local stakeholders in the countries where the initiatives are carried out, policy makers involved in the initiatives, representatives of donor organizations, and others to derive lessons learned from past large-scale evaluations and to discuss how to apply these lessons to future evaluations. This report discusses transferable insights gained across the spectrum of choosing the evaluator, framing the evaluation, designing the evaluation, gathering and analyzing data, synthesizing findings and recommendations, and communicating key messages. The report also explores the relative benefits and limitations of different quantitative and qualitative approaches within the mixed methods designs used for these complex and costly evaluations. Table of Contents Front Matter 1 Introduction and Overview of the Workshop 2 Overview Framework for Complex Evaluations 3 Framing the Evaluation 4 Developing the Evaluation Design and Selecting Methods 5 Mapping Data Sources and Gathering and Assessing Data 6 Applying Qualitative Methods to Evaluation on a Large Scale 7 Applying Quantitative Methods to Evaluation on a Large Scale 8 Analysis Through Triangulation and Synthesis to Interpret Data in a Mixed Methods Evaluation 9 Evolving Methods in Evaluation Science 10 Lessons from Large-Scale Program Evaluation on a Not-Quite-as-Large Scale 11 Using Evaluation Findings and Communicating Key Messages 12 Envisioning a Future for Evaluation References Appendix A: Statement of Task Appendix B: Workshop Agenda Appendix C: Participant Biographies Appendix D: Evaluation Information Summary for Core Example Initiatives Appendix E: Evaluation Design Resources Highlighted at the Workshop