ABCs of Evaluation

John Boulmetis, PhD, is a Professor Emeritus at the University of Rhode Island School of Education. Dr. Boulmetis received his master’s degree in education from the University of Rhode Island and his doctoral degree in adult and vocational education from The Ohio State University. Since his retirement, he has consulted as an independent third-party evaluator on federal grants and contracts. Phyllis Dutwin, MA, was president of Dutwin Associates, a training and development firm, and former vice president of Reading and Educational Services of New Jersey. She received her master’s degree in adult education, training, and development from the University of Rhode Island.

• List of Figures ixList of Tables and Exhibits xiPreface xiiiThe Authors xxiAbout the Companion Website xxiii1 What Is Evaluation? 1Common Denominators 3Two Definitions of Evaluation 4Evaluating Efficiency, Effectiveness, and Impact 5Evaluating Alternatives 8Identifying Areas to Improve 9Other Levels 11Formal Reasons to Evaluate 12Practical Application 15Formative and Summative Evaluations 16Putting It All Together 16Key Words and Concepts 22Questions and Exercises 22Further Reading 282 Why Evaluate? 29Grants and Grantsmanship 31Benefits 33Limitations 36Considerations 37Putting It All Together 40Key Words and Concepts 41Questions and Exercise 41Further Reading 463 Decision- Making: Whom to Involve, How, and Why? 47Program and Project Cycles 49Evaluation and Monitoring 52Program Planning Cycle and Evaluation 59Putting It All Together 63Key Words and Concepts 65Questions and Exercises 65Further Reading 734 Starting Point: The Evaluator’s Program Description 75What an Evaluator’s Program Description Can Do 77Developing an Evaluator’s Program Description 79Putting It All Together 83Key Words and Concepts 85Questions and Exercises 85Further Reading 975 Choosing an Evaluation Model 99More Than One Way to Evaluate 100Four Evaluation Models 103Other Models 110Choosing the Right Model 112Evaluation Design Format 113Other Considerations 115Putting It All Together 116Key Words and Concepts 117Questions and Exercise 117Further Reading 1196 Data Sources 121Starting Point 122Qualitative and Quantitative Data Sources 126Four Levels of Data 127Nature of Data Collection 129Existing Data 130Newly Collected Data 132Putting It All Together 143Key Words and Concepts 143Questions and Exercise 143Further Reading 1467 Data Analysis 147Data Analysis 148Data 150Terms 151Levels of Data 157Putting It All Together 163Key Words and Concepts 164Questions and Exercise 164Further Reading 166Software for Personal Computers 1678 Is It Evaluation or Is It Research? 169Determining Evaluation or Research from Intent 170Determining Evaluation or Research from Design 172Qualitative and Quantitative Characteristics 174Sampling 175Sampling and Surveys 180Putting It All Together 181Key Words and Concepts 183Questions and Exercise 183Further Reading 1849 Writing the Evaluation Report 185Communicating Your Findings 187Evaluation Reports 188The Report Outline 189Putting It All Together 201Key Words and Concepts 202Questions and Exercise 202Further Reading 20310 Evaluation as a Business 205Is There a Market for Program Evaluation? 206How Do You Find the Business? 213Proactive Resources 215Reactive Resources 217Emergency Resources 218How Do You Negotiate a Contract? 219Keeping the Business Alive 223Putting It All Together 225Key Words and Concepts 226Further Reading 227A: More on Data Analysis 229Analyses for Each Level 229Significance Testing 233Statistical and Practical Significance 237B: Sample Evaluation Report 239C: Sample Evaluation Contract 271Evaluation of the XYZ Training Project 271D: Sample Evaluator Job Description 277Evaluator— MA, MEd, Education Technology and Instructional Design 277Disclaimer 280References 281Index 285

Select Guide Rating