Implementation of Large-Scale Education Assessments

Edited byPetra Lietz and John C. Cresswell, Australian Council for Educational Research (ACER), Australia Keith F. Rust, Westat and the University of Maryland at College Park, USA Raymond J. Adams, Australian Council for Educational Research (ACER), Australia

• Notes on Contributors xvForeword xviiAcknowledgements xxAbbreviations xxi1 Implementation of Large]Scale Education Assessments 1Petra Lietz, John C. Cresswell, Keith F. Rust and Raymond J. Adams1.1 Introduction 11.2 International, Regional and National Assessment Programmes in Education 31.3 Purposes of LSAs in Education 41.4 Key Areas for the Implementation of LSAs in Education 101.5 Summary and Outlook 16Appendix 1.A 18References 222 Test Design and Objectives 26Dara Ramalingam2.1 Introduction 262.2 PISA 272.3 TIMSS 342.4 PIRLS and Pre]PIRLS 412.5 ASER 452.6 SACMEQ 522.7 Conclusion 56References 583 Test Development 63Juliette Mendelovits3.1 Introduction 633.2 Developing an Assessment Framework: A Collaborative and Iterative Process 653.3 Generating and Collecting Test Material 683.4 Refinement of Test Material 723.5 Beyond Professional Test Development: External Qualitative Review of Test Material 813.6 Introducing Innovation 863.7 Conclusion 90References 904 Design, Development and Implementation of Contextual Questionnaires in Large]Scale Assessments 92Petra Lietz4.1 Introduction 924.2 The Role of Questionnaires in LSAs 934.3 Steps in Questionnaire Design and Implementation 954.4 Questions and Response Options in LSAs 1154.5 Alternative Item Formats 1194.6 Computer]Based/Online Questionnaire Instruments 1284.7 Conclusion and Future Perspectives 131Acknowledgements 132References 1325 Sample Design, Weighting, and Calculation of Sampling Variance 137Keith F. Rust, Sheila Krawchuk and Christian Monseur5.1 Introduction 1375.2 Target Population 1385.3 Sample Design 1445.4 Weighting 1485.5 Sampling Adjudication Standards 1535.6 Estimation of Sampling Variance 156References 1666 Translation and Cultural Appropriateness of Survey Material in Large]Scale Assessments 168Steve Dept, Andrea Ferrari and Béatrice Halleux6.1 Introduction 1686.2 Overview of Translation/Adaptation and Verification Approaches Used in Current Multilingual Comparative Surveys 1696.3 Step]by]Step Breakdown of a Sophisticated Localisation Design 1746.4 Measuring the Benefits of a Good Localisation Design 1846.5 Checklist of Requirements for a Robust Localisation Design 190References 1917 Quality Assurance 193John C. Cresswell7.1 Introduction 1937.2 The Development and Agreement of Standardised Implementation Procedures 1947.3 The Production of Manuals which Reflect Agreed Procedures 1967.4 The Recruitment and Training of Personnel in Administration and Organisation: Especially the Test Administrator and the School Coordinator 1977.5 The Quality Monitoring Processes: Recruiting and Training Quality Monitors to Visit National Centres and Schools 1987.6 Other Quality Monitoring Procedures 2017.7 Conclusion 204Reference 2048 Processing Responses to Open]Ended Survey Questions 205Ross Turner8.1 Introduction 2058.2 The Fundamental Objective 2078.3 Contextual Factors: Survey Respondents and Items 2078.4 Administration of the Coding Process 2148.5 Quality Assurance and Control: Ensuring Consistent and Reliable Coding 2218.6 Conclusion 229References 2299 Computer]Based Delivery of Cognitive Assessment and Questionnaires 231Maurice Walker9.1 Introduction 2319.2 Why Implement Computer]Based Assessments? 2329.3 Implementation of International Comparative Computer]Based Assessments 2389.4 Assessment Architecture 2449.5 Item Design Issues 2479.6 State]of]the]Art and Emerging Technologies 2509.7 Summary and Conclusion 250References 25110 Data Management Procedures 253Falk Brese and Mark Cockle10.1 Introduction 25310.2 Historical Review: From Data Entry and Data Cleaning to Integration into the Entire Study Process 25410.3 The Life Cycle of a LSA Study 25510.4 Standards for Data Management 25610.5 The Data Management Process 25810.6 Outlook 272References 27411 Test Implementation in the Field: The Case of PASEC 276Oswald Koussihouèdé, Antoine Marivin and Vanessa Sy11.1 Introduction 27611.2 Test Implementation 27811.3 Data Entry 29411.4 Data Cleaning 29511.5 Data Analysis 29511.6 Governance and Financial Management of the Assessments 295Acknowledgments 296References 29712 Test Implementation in the Field: The Experience of Chile in International Large]Scale Assessments 298Ema Lagos Campos12.1 Introduction 29812.2 International Studies in Chile 302Annex A 321References 32113 Why Large]Scale Assessments Use Scaling and Item Response Theory 323Alla Berezner and Raymond J. Adams13.1 Introduction 32313.2 Item Response Theory 32513.3 Test Development and Construct Validation 32913.4 Rotated Test Booklets 34513.5 Comparability of Scales Across Settings and Over Time 34713.6 Construction of Performance Indicators 34913.7 Conclusion 354References 35414 Describing Learning Growth 357Ross Turner and Raymond J. Adams14.1 Background 35714.2 Terminology: The Elements of a Learning Metric 35814.3 Example of a Learning Metric 36014.4 Issues for Consideration 36014.5 PISA Described Proficiency Scales 36514.6 Defining and Interpreting Proficiency Levels 37414.7 Use of Learning Metrics 379Acknowledgement 380References 38115 Scaling of Questionnaire Data in International Large]Scale Assessments 384Wolfram Schulz15.1 Introduction 38415.2 Methodologies for Construct Validation and Scaling 38615.3 Classical Item Analysis 38715.4 Exploratory Factor Analysis 38815.5 Confirmatory Factor Analysis 38915.6 IRT Scaling 39215.7 Described IRT Questionnaire Scales 39615.8 Deriving Composite Measures of Socio]economic Status 39915.9 Conclusion and Future Perspectives 404References 40516 Database Production for Large]Scale Educational Assessments 411Eveline Gebhardt and Alla Berezner16.1 Introduction 41116.2 Data Collection 41216.3 Cleaning, Recoding and Scaling 41616.4 Database Construction 41816.5 Assistance 421References 42317 Dissemination and Reporting 424John C. Cresswell17.1 Introduction 42417.2 Frameworks 42517.3 Sample Items 42617.4 Questionnaires 42717.5 Video 42717.6 Regional and International Reports 42817.7 National Reports 42817.8 Thematic Reports 42917.9 Summary Reports 42917.10 Analytical Services and Support 43017.11 Policy Papers 43017.12 Web]Based Interactive Display 43117.13 Capacity]Building Workshops 43217.14 Manuals 43217.15 Technical Reports 43217.16 Conclusion 433References 433Index 436