Understanding Measurement: Reliability

Patrick Meyer, Assistant Professor, Curry School of Education, University of Virginia.

• Chapter I: Introduction The Importance of Test Score Reliability Organization of the Book General Concepts The Measurement Procedure Sampling in Measurement Sampling the Measurement Procedure. Sampling Examinees. Replicating the Measurement Procedure and Sampling Examinees Classical Test Theory Replicating the Measurement Procedure Additional Restrictions that Define Classical Test Theory Sources of Variance The Reliability Coefficient Estimating Unobserved Quantities Classification decisions Replicating the Measurement Procedure Strong True Score Theory Distributional Assumptions Domain Scores. Conditional Distribution of Observed Scores. Observed Scores. Estimating Domain Scores. Reliability and Conditional Standard Error of Measurement Generalizability Theory Generalizability Study Variance Components. Decision Study Replicating the Measurement Procedure. Sources of Variance. Types of Decisions. Reliability Coefficients. Estimating Unobserved Quantities. Chapter Summary Chapter 2: Data Collection Designs Description of Real-World Measurement Procedure Data Collections Designs in Classical Test Theory Whole Test Replications Test-Retest Reliability Coefficient Alternate Forms Reliability Coefficient Test-Retest with Alternate Forms Two Part-Test Replications Split-halves Other Two-Part Divisions. Multiple Part-test Divisions. Data Collection Designs for Classification Decisions Data Collection Designs in Generalizability Theory Common Data Collection Designs Random Designs Mixed Designs Classical Test Theory Designs Restated as Data Collection Designs Data Collection Designs for the ELA Benchmark Assessment Chapter Summary Chapter 3: Assumptions Dimensionality Violation of Unidimensionality Error Score Correlations Violation of Uncorrelated Errors The Nature of Measurement Procedure Replications Classical Test Theory Congeneric Measures. Classical Congeneric Measures. Essentially Tau-equivalent Measures. Tau-equivalent Measures. Parallel Measures. Violation of Essential tau-equivalence Classification Decisions and Generalizability Theory Randomly Parallel Measures Distributional Assumptions Violation of Randomly Parallel Measures Chapter Summary Chapter 4: Methods Reliability Coefficients for Relative Decisions Whole Test Replications Two Part-test Replications Multiple Part-test Replications Other Methods for Estimating Reliability Estimating Reliability for Absolute Decisions Two Whole-test Replications Squared Error Loss Coefficients Threshold Loss Indices One Whole-test Replication Squared Error Loss Coefficients Threshold Loss Indices Other Methods Chapter Summary Chapter 5: Results Documentation of Score Reliability Characterize the Examinee Population Describe the Measurement Procedure and All Major Sources of Error Present Evidence in Support of Assumptions Provide Estimates of Reliability and the Standard Error of Measurement Benchmark ELA Assessment Sample Results Section Palmetto Achievement Challenge Test of Mathematics Sample Results Section The Responsive Classroom Efficacy Study and MSCAN Sample Results Section Chapter Summary Chapter 6: Discussion and Recommended Readings ELA Benchmark Assessment Some Interesting Points about the Analysis Palmetto Achievement Challenge Test of Mathematics The Responsive Classroom Efficacy Study and MSCAN Recommended Readings References