Structural Equation Modeling of Multiple Rater Data

Michael Eid, PhD, is Professor of Methods and Evaluation at the Free University of Berlin in Germany. His research focuses on measurement theory, in particular on the development of psychometric models for longitudinal and multimethod research. Since the early 2000s, he has been contributing to the development of structural equation models for analyzing multirater data for different types of raters and research designs. His more applied research contributions are in the areas of subjective well-being, mood regulation, and health psychology.Christian Geiser, PhD, is a former professor of quantitative psychology. He currently works as an instructor and statistical consultant with QuantFish. His areas of expertise are in structural equation modeling, measurement, longitudinal data analysis, latent class modeling, and multitrait–multimethod analysis.Tobias Koch, PhD, is Professor of Psychological Methods at the Friedrich-Schiller-Universität Jena in Germany. His research focuses on measurement theory and psychometrics, structural equation modeling, longitudinal data analysis, multilevel analysis, Bayesian analysis, and multitrait–multimethod analysis.

• 1. Introduction: The Importance of Multiple Rater Data1.1 Advantages and Limitations of Self-Reports1.2 Advantages and Limitations of Other Reports1.21 Models of accuracy of other ratings1.22 Sources of accuracy of other ratings1.3 Usefulness of Multiple Rater Studies1.31 Analyzing the validity of ratings1.32 Improving the validity of inferences by multiple ratings1.4 The Role of Measurement Models1.5 Summary1.6 Suggested Further Readings2. Basic Methodological Concepts2.1 Design Issues2.1.1 Interchangeable and structurally different raters2.1.2 Measurement designs2.2 Confirmatory Factor Analysis of Multiple Rater Data2.3 Stochastic Measurement Theory: Basic Ideas2.3.1 Sampling process for structurally different raters2.3.2 Sampling process for interchangeable raters2.3.3 Differences between structurally different and interchangeable raters2.4 Overview of the Present Book2.5 Summary2.6 Suggested Further Readings3. Basic Models for Structurally Different Raters3.1 Basic Decomposition of Observed Variables3.2 Basic Model with Correlated First-Order Factors3.2.1 Application of the MTMR model with correlated first-order factors: Loneliness and flourishing3.3 Model with Indicator-Specific Factors3.3.1 Application of the MTMR model with correlated first-order factors and indicator-specific factors: Loneliness and flourishing3.3.2 Recommendations: Model Selection3.4 Basic Model with Measurement Invariance Across Raters3.4.1 Statistical tests for testing measurement invariance3.4.2 Partial measurement invariance and measurement invariance in models with indicator-specific factors3.4.3 How important is measurement invariance in multirater studies?3.5 Application of the Models to Other Measurement Designs3.6 Chapter Summary3.7 Suggested Further Readings4. Models with Method Factors for Structurally Different Raters4.1 Basic CTC(M-1) Model4.1.1 Choice of a reference rater group4.1.2 Application of the CTC(M-1) model: Loneliness and flourishing4.1.3 Comparing the CTC(M-1) model with the model with correlated first-order factors4.2 Restricted CTC(M-1) Model as Reformulation of the Model with Correlated First-Order Factors4.3 Restricted CTC(M-1) Model with Measurement Invariance Across Raters4.4 CTC(M-1) Models with Indicator-Specific Effects4.4.1 CTC(M-1) models with indicator-specific factors4.4.2 CTC(M-1) model with indicator-specific traits4.4.3. Choosing a CTC(M-1) model with indicator-specific effects4.5 Alternative Models4.5.1 Latent difference model4.5.2 Latent means model4.5.3 Reference rating as outcome model4.6 Models with Covariates4.7 Application of the Models to Other Measurement Designs4.7.1 Analyzing round-robin data for structurally different raters4.8 Chapter Summary4.9 Suggested Further Readings5. Single-Level CFA Models for Interchangeable Raters5.1 The Interchangeable-Saturated (I-SAT) Model5.2 Basic Decomposition5.3 Basic Models with Correlated First-Order Factors5.3.1 Adjustment of fit statistics with interchangeable models5.3.2 Application of the basic model with correlated first-order factors with and without indicator-specific factors: Loneliness and flourishing5.4 Models with Method Factors5.4.1 Basic correlated traits-interchangeable methods (CTIM) model5.4.2 Application of the basic CTIM model: Loneliness and flourishing5.4.3 Restricted CTIM model5.4.4 Application of the restricted CTIM model: Loneliness and flourishing5.4.5 CTIM models with indicator-specific effects5.5 Chapter Summary5.6 Suggested Further Readings6. Multilevel CFA Models for Interchangeable Raters6.1 Multilevel Confirmatory Factor Analysis6.1.1 Wide- versus long-format data6.2 Basic Models with Correlated First-Order Factors and CTIM Models6.2.1 Invariance constraints, variance decomposition, and coefficients6.2.2 Application of CTIM models: Loneliness and flourishing6.3 Practical Issues6.4 Models for Round-Robin Designs with Interchangeable Raters6.4.1 Analyzing round-robin data with interchangeable raters6.4.2 Application to flourishing and loneliness data6.5 Chapter Summary6.6 Suggested Further Readings7. Models for a Combination of Structurally Different and Interchangeable Raters7.1 Basic Multilevel Models with Correlated First-Order Factors for a Traditional Combination of Different Types of Raters7.2 Multilevel CTC(M-1) Models for a Combination of Structurally Different and Interchangeable Rate7.2.1 Application of the multilevel CTC(M-1) model: Loneliness and flourishing7.2.2 The multilevel CTC(M-1) model as an extension of the trait-reputation-identity model to multiple traits7.3 Extensions of the models to other measurement designs7.3.1 Multilevel CTC(M-1) model for multiple sets of interchangeable raters7.3.2 Multilevel CTC(M-1) model for fully nested structurally different raters7.3.3 Application of CFA-MTMR models for a special combination of rater groups7.4 Chapter Summary7.5 Suggested Further Readings8.Models for Cross-Classified Multiple Membership Data8.1 Models for Crossed Designs with Overlapping Sets of Interchangeable Raters8.2 Models for Crossed Designs with a Combination of Structurally Different and Overlapping Sets of Interchangeable Raters8.2.1 Application of the cross-classified CTC(M-1) model: Teaching quality and engagement8.3 Extensions of the models to other measurement designs8.4 Chapter Summary8.5 Suggested Further Readings9. Models for Longitudinal Multirater Data9.1 Basic Decomposition of Observed Variables9.2 Basic Multistate-Multirater (MSMR) Model9.2.1 Testing for measurement invariance across time9.2.2 Longitudinal Modeling Strategy9.2.3 Application of the MSMR Model: Loneliness and flourishing9.2.4 Summary and extensions9.3 The Correlated States-Correlated (Methods – 1) [CSC(M – 1)] Model9.3.1 Application of the CSM (M-1) Model with Indicator-Specific Factors: Loneliness and Flourishing9.4 The CSC(M-1) Change Model9.4.1 Application of the CSC(M -1) change model: Loneliness and flourishing9.5 The Multimethod LST Model9.5.1 Application of the MM-LST model: Loneliness and flourishing9.6 Longitudinal Multirater Models for a Combination of Structurally Different and Interchangeable Raters9.6.1 The latent state combination of methods (LS-COM) model9.6.2 Application of the LS-COM model9.6.3 The latent state-trait combination of methods (LST-COM) model9.6.4 Application of the LST-COM model9.7 Chapter Summary9.8 Suggested Further Readings10. Advanced Topics in Multitrait-Multirater Analysis10.1 CFA-MTMR Models with Categorical Observed Variables10.1.1 Practical recommendations10.1.2 Application of CFA-MTMR models with categorical observed variables10.2 Explaining Latent Trait and Rater-Specific Factors in CFA-MTMR Models without Bias10.2.1 Mean structure10.2.2 Covariance structure10.3 Predicting External Criterion Variables based on CFA-MTMR Models10.3.1 Models with only structurally different raters10.3.2 Models with interchangeable raters10.3.3 Application10.4 Chapter Summary10.5 Suggested Further Readings11. Recommendations and Outlook11.1 Choice of a Design11.1.1 Type

"This book provides a comprehensive yet accessible overview of the analysis of data from multiple raters. It introduces readers to the most basic models for analyzing multiple rater data, while also providing in-depth coverage of more advanced models that can be used to analyze more complex designs and answer more complex questions."--Richard E. Lucas, PhD, MSU Foundation Professor, Department of Psychology, Michigan State University"This is a great book for advanced graduate students who are well grounded in statistics and seek a better understanding of how models for multiple raters work. The book does a good job of introducing each of the models and how they are estimated."--Sara Tomek, PhD, Department of Educational Psychology, Baylor University"Written by leading scholars, this book fills a gap in the literature by providing a coherent, concise account of models for analyzing multiple rater data within the CFA framework. The authors present an up-to-date, self-contained introduction to the field. I would recommend this book to students and colleagues and consider it for teaching courses on CFA modeling."--Thomas Eckes, PhD, TestDaF Institute (retired), Bochum, Germany"The extensive coverage will make this the go-to reference on the analysis of multiple rater data."--David Kaplan, PhD, Hilldale Professor and Patricia Busk Professor of Quantitative Methods, Department of Educational Psychology, University of Wisconsin–Madison-