Evaluation Essentials

Beth Osborne DaPonte, Ph.D. has taught program evaluation at the graduate level at Yale University’s School of Management, Carnegie Mellon University, and the University of Pittsburgh. Additionally, Beth is Principal and Owner of Social Science Consultants, LLC, a firm that provides consulting and advisory services to the non-profit/NGO, religious, governmental, and international sectors. Clients have included the United Nations (secretariat and UN-System funds, programs, and agencies); foundations (e.g., Forbes Funds, Heinz Endowments, Hillman Family Foundations); educational institutions (e.g., University of Pittsburgh); and US-based nonprofits (e.g., Greater Pittsburgh Community Food Bank, Allies for Children).

• Preface ixAcknowledgments xOne: Introduction 1Learning Objectives 1Introduction 2Size and Importance of Nonprofit Sector 3The Evaluation Framework 5Structure of Book 9Summary 9Key Terms 10Discussion Questions 10Two: Describing the Initiative 11Learning Objectives 11Introduction 12Reasons to Rigorously Describe the Program 13The Logistics of Describing the Program 14Common Mistakes Made Describing Initiative and Programs 16The Program Is Alive and So Is Its Description 18Theory of Change 19The Program Logic Model 25Analyzing the Program Logic Model 34Challenges of Multisite Programs 35Program Implementation Model 36Examples of Program Descriptions 37Summary 67Key Terms 68Discussion Questions 68Three: Laying the Evaluation Ground Work 69Learning Objectives 69Evaluation Approaches 70Framing Evaluation Questions 72Insincere Reasons for Evaluation 75High Stakes Evaluation 75Who Will do the Evaluation? 76External Evaluators 77Internal Evaluators 79Independence of the Evaluator 79External Versus Internal Evaluators 80Confidentiality and Ownership of Evaluations 81Endorsement and Action on Recommendations 82Evaluation Policy 83The Evaluation Report 84Summary 86Key Terms 86Discussion Questions 86Four: Causation 87Learning Objectives 87Introduction 88Necessary and Sufficient 88Setting Cutoff Points and Causal Interpretation 91Intervening Variables 95Types of Causal Effects 97Lagged Effects 98Permanency of Effects 98Functional Form of Impact 99Spectacular Causes and Effects 100Summary 101Key Terms 102Discussion Questions 102Five: the Prisms of Validity 103Learning Objectives 103Introduction 104Statistical Conclusion Validity 105Types of Erroneous Conclusions: Type 1 and Type 2 Errors 106Threats to Statistical Conclusion Validity 107Internal Validity 111Threats to Internal Validity 111Summary 118Construct Validity 119Threats to Construct Validity 120Summary 122External Validity 122Considerations for Determining the External Validity of Studies 122Summary 123Conclusions 124Key Terms 124Discussion Questions 124Six: Attributing Outcomes to the Program: Quasi-Experimental Design 126Learning Objectives 126Introduction 127Quasi-Experimental Notation 127Frequently used Designs that do not Show Causation 128One-Group Posttest-Only 128Posttest-Only with Nonequivalent Groups 129Participants’ Pretest–Posttest 130Designs that Generally Permit Causal Inferences 131Untreated Control Group Design with Pretest and Posttest 131Delayed Treatment Control Group 137Different Samples from the Same Populations Design 140Nonequivalent Observations Drawn from One Group 141Equivalent Groups Using Switched Measures 142Cohort Designs 144Time Series Designs 146Summary 148Key Terms 149Discussion Questions 149Seven: Collecting Data 150Learning Objectives 150Introduction 151Informal and Loosely Structured Interviews 152Focus Groups 154Surveys 160Survey Instrument Design 161Target Populations and Sampling 169Informal Sampling Approaches 170Formal Sampling Approaches 171Best Practices in Administering Surveys to a Purposive Sample 172Follow-up for Nonresponse 173Secondary Data Sources 173Summary 174Key Terms 174Discussion Questions 175Eight: Conclusions 176Learning Objectives 176Using Evaluation Tools to Write Grant Proposals 177Hiring Evaluation Consultants 177Writing Recommendations 180The Tone of Evaluation Reports 180Finalization Process of Evaluation Reports 181Conclusions 182Discussion Questions 182Glossary 183Index 188