Performance Evaluation

Ingrid J. Guerra-López, PhD, is an associate professor at Wayne State University, director of the Institute for Learning and Performance Improvement, associate research professor at the Sonora Institute of Technology in Mexico, and principal of Intelligence Gathering Systems.

• Acknowledgments xiPreface xiiiThe Author xvPart One: Introduction To EvaluationOne: Foundations of Evaluation 3A Brief Overview of Evaluation History 4Evaluation: Purpose and Definition 5Performance Improvement: A Conceptual Framework 8Making Evaluation Happen: Ensuring Stakeholders’ Buy-In 9The Evaluator: A Job or a Role? 10The Relationship to Other Investigative Processes 11When Does Evaluation Occur? 15General Evaluation Orientations 18Challenges That Evaluators Face 20Ensuring Commitment 23Benefits of Evaluation 24Basic Definitions 25Two: Principles of Performance-based Evaluation 27Principle 1: Evaluation Is Based on Asking the Right Questions 28Principle 2: Evaluation of Process Is a Function of Obtained Results 32Principle 3: Goals and Objectives of Organizations Should Be Based on Valid Needs 33Principle 4: Derive Valid Needs Using a Top-Down Approach 34Principle 5: Every Organization Should Aim for the Best That Society Can Attain 34Principle 6: The Set of Evaluation Questions Drives the Evaluation Study 35Part Two: Models of EvaluationThree: Overview of Existing Evaluation Models 39Overview of Classic Evaluation Models 40Selected Evaluation Models 42Selecting a Model 43Conceptualizing a Useful Evaluation That Fits the Situation 44Four: KIRKPATRICK’S Four LEVELS Of Evaluation 47Kirkpatrick’s Levels 49Comments on the Model 54Strengths and Limitations 55Application Example: Wagner (1995) 56Five: Phillips’s Return-on-investment Methodology 61Phillips’s ROI Process 63Comments on the Model 67Strengths and Limitations 70Application Example: Blake (1999) 70Six: Brinkerhoff’s Success Case Method 75The SCM Process 77Strengths and Weaknesses 78Application Example: Brinkerhoff (2005) 79Seven: the Impact Evaluation Process 81The Elements of the Process 83Comments on the Model 96Strengths and Limitations 97Application Example 97Eight: the Cipp Model 107Stufflebeam’s Four Types of Evaluation 108Articulating Core Values of Programs and Solutions 111Methods Used in CIPP Evaluations 112Strengths and Limitations 113Application Example: Filella-Guiu and Blanch-Pana (2002) 113Nine: Evaluating Evaluations 117Evaluation Standards 119The American Evaluation Association Principles for Evaluators 120Application Example: Lynch et al. (2003) 122Part Three: Tools and Techniques of EvaluationTen: Data 133Characteristics of Data 135Scales of Measurement 137Defining Required Data from Performance Objectives 139Deriving Measurable Indicators 141Finding Data Sources 152Follow-Up Questions and Data 155Eleven: Data Collection 159Observation Methodology and the Purpose of Measurement 160Designing the Experiment 186Problems with Classic Experimental Studies in Applied Settings 188Time-Series Studies 188Simulations and Games 189Document-Centered Methods 191Conclusion 192Twelve: Analysis of Evaluation Data: Tools and Techniques 195Analysis of Models and Patterns 196Analysis Using Structured Discussion 197Methods of Quantitative Analysis 199Statistics 200Graphical Representations of Data 210Measures of Relationship 212Inferential Statistics: Parametric and Nonparametric 214Interpretation 217Thirteen: Communicating The Findings 221Recommendations 222Considerations for Implementing Recommendations 225Developing the Report 226The Evaluator’s Role After the Report 235Part Four: Continual ImprovementFourteen: Common Errors in Evaluation 239Errors of System Mapping 240Errors of Logic 242Errors of Procedure 244Conclusion 246Fifteen: Continual Improvement 249What Is Continual Improvement? 250Monitoring Performance 250Adjusting Performance 253The Role of Leadership 254Sixteen: Contracting for Evaluation Services 257The Contract 258Contracting Controls 260Ethics and Professionalism 262Sample Statement of Work 262Seventeen: Intelligence Gathering For Decision Making 271Performance Measurement Systems 273Issues in Performance Measurement Systems 275Conclusion 277Eighteen: the Future of Evaluation in Performance Improvement 279Evaluation and Measurement in Performance Improvement Today 281What Does the Future Hold? 282Conclusion 283References and Related Readings 285Index 295