Design, Evaluation, and Analysis of Questionnaires for Survey Research

WILLEM E. SARIS, PHD, is Emeritus Professor in Methodology of the University of Amsterdam and the Universitat Pompeu Fabra, Barcelona. He is Laureate of the 2005 Descartes Prize for “Best Collaborative Research” as member of the Central Coordinating Team of the European Social Survey (ESS) and Recipient of the World Association of Public Opinion Research’s “Helen Dinerman Award” in 2009 for his lifelong contribution to the methodology of Opinion Research. Dr. Saris also received the “2013 Outstanding Service Prize” of the European Survey Research Association.IRMTRAUD N. GALLHOFER, PHD, is a linguist and was senior researcher on projects of the ESS, Research and Expertise Centre for Survey Methodology at the Universitat Pompeu Fabra, Barcelona. She is Laureate of the 2005 Descartes Prize for “Best Collaborative Research” as a member of the Central Coordinating Team of the ESS.

• Preface to the Second Edition xiiiPreface xvAcknowledgments xviiIntroduction 1I.1 Designing a Survey 4I.1.1 Choice of a Topic 4I.1.2 Choice of the Most Important Variables 4I.1.3 Choice of a Data Collection Method 5I.1.4 Choice of Operationalization 6I.1.5 Test of the Quality of the Questionnaire 8I.1.6 Formulation of the Final Questionnaire 9I.1.7 Choice of Population and Sample Design 9I.1.8 Decide about the Fieldwork 10I.1.9 What We Know about These Decisions 10I.1.10 Summary 11Exercises 12Part I The Three-Step Procedure to Design Requests for Answers 131 Concepts-by-Postulation and Concepts-by-Intuition 151.1 Concepts-by-Intuition and Concepts-by-Postulation 151.2 Different Ways of Defining Concepts-by-Postulation through Concepts-by-Intuition 191.2.1 Job Satisfaction as a Concept-by-Intuition 191.2.2 Job Satisfaction as a Concept-by-Postulation 201.3 Summary 27Exercises 282 From Social Science Concepts-by-Intuition to Assertions 302.1 Basic Concepts and Concepts-by-Intuition 312.2 Assertions and Requests for an Answer 322.3 The Basic Elements of Assertions 332.3.1 Indirect Objects as Extensions of Simple Assertions 362.3.2 Adverbials as Extensions of Simple Assertions 372.3.3 Modifiers as Extensions of Simple Assertions 372.3.4 Object Complements as Extensions of Simple Assertions 382.3.5 Some Notation Rules 382.4 Basic Concepts-by-Intuition 392.4.1 Subjective Variables 402.4.2 Objective Variables 472.4.3 In Summary 492.5 Alternative Formulations for the Same Concept 492.6 Extensions of Simple Sentences 512.6.1 Adding Indirect Objects 512.6.2 Adding Modifiers 522.6.3 Adding Adverbials 522.7 Use of Complex Sentences 532.7.1 Complex Sentences with No Shift in Concept 542.7.2 Complex Sentences with a Shift in Concept 542.7.3 Adding Conditions to Complex Sentences 562.8 Summary 56Exercises 573 The Formulation of Requests for an Answer 603.1 From Concepts to Requests for an Answer 613.2 Different Types of Requests for an Answer 633.2.1 Direct Request 633.2.2 Indirect Request 663.3 The Meaning of Requests for an Answer with WH Request Words 693.3.1 “When,” “Where,” and “Why” Requests 703.3.2 “Who” Requests 703.3.3 “Which” Requests 703.3.4 “What” Requests 713.3.5 “How” Requests 723.4 Summary 74Exercises 75Part II Choices Involved in Questionnaire Design 774 Specific Survey Research Features of Requests for an Answer 794.1 Select Requests from Databases 794.2 Other Features Connected with the Research Goal 814.3 Some Problematic Requests 834.3.1 Double-Barreled Requests 834.3.2 Requests with Implicit Assumptions 844.4 Some Prerequests Change the Concept-by-Intuition 854.5 Batteries of Requests for Answers 864.5.1 The Use of Batteries of Stimuli 874.5.2 The Use of Batteries of Statements 884.6 Other Features of Survey Requests 924.6.1 The Formulation of Comparative or Absolute Requests for Answers 924.6.2 Conditional Clauses Specified in Requests for Answers 934.6.3 Balanced or Unbalanced Requests for Answers 934.7 Special Components within the Request 954.7.1 Requests for Answers with Stimulation for an Answer 954.7.2 Emphasizing the Subjective Opinion of the Respondent 954.8 Summary 96Exercises 965 Response Alternatives 985.1 Open Requests for an Answer 995.2 Closed Categorical Requests 1015.2.1 Nominal Categories 1035.2.2 Ordinal Scales 1045.2.3 Continuous Scales 1085.3 How Many Categories are Optimal? 1115.4 Summary 112Exercises 1146 The Structure of Open-Ended and Closed Survey Items 1156.1 Description of the Components of Survey Items 1156.2 Different Structures of Survey Items 1186.2.1 Open-Ended Requests for an Answer 1196.2.2 Closed Survey Items 1206.2.3 The Frequency of Occurrence 1246.2.4 The Complexity of Survey Items 1256.3 What Form of Survey Items Should Be Recommended? 1266.4 Summary 127Exercises 1287 Survey Items in Batteries 1307.1 Batteries in Oral Interviews 1317.2 Batteries in Mail Surveys 1347.3 Batteries in CASI 1387.4 Summary and Discussion 142Exercises 1448 Mode of Data Collection and Other Choices 1468.1 The Choice of the Mode of Data Collection 1478.1.1 Relevant Characteristics of the Different Modes 1488.1.2 The Presence of the Interviewer 1498.1.3 The Mode of Presentation 1518.1.4 The Role of the Computer 1528.1.5 Procedures without Asking Questions 1558.1.6 Mixed-Mode Data Collection 1558.2 The Position in the Questionnaire 1568.3 The Layout of the Questionnaire 1588.4 Differences due to Use of Different Languages 1588.5 Summary and Discussion 159Exercises 160Part III Estimation and Prediction of the Quality of Questions 1639 Criteria for the Quality of Survey Measures 1659.1 Different Methods, Different Results 1669.2 How These Differences Can Be Explained 1739.2.1 Specifications of Relationships between Variables in General 1739.2.2 Specification of Measurement Models 1759.3 Quality Criteria for Survey Measures and Their Consequences 1789.4 Alternative Criteria for Data Quality 1819.4.1 Test–Retest Reliability 1819.4.2 The Quasi-simplex Approach 1829.4.3 Correlations with Other Variables 1839.5 Summary and Discussion 184Exercises 185Appendix 9.1 The Specification of Structural Equation Models 18710 Estimation of Reliability, Validity, and Method Effects 19010.1 Identification of the Parameters of a Measurement Model 19110.2 Estimation of Parameters of Models with Unmeasured Variables 19510.3 Estimating Reliability, Validity, and Method Effects 19710.4 Summary and Discussion 201Exercises 202Appendix 10.1 Input of Lisrel for Data Analysis of a Classic MTMM Study 205Appendix 10.2 Relationship between the TS and the Classic MTMM Model 20511 Split-Ballot Multitrait–Multimethod Designs 20811.1 The Split-Ballot MTMM Design 20911.1.1 The Two-Group Design 20911.1.2 The Three-Group Design 21011.1.3 Other SB-MTMM Designs 21111.2 Estimating and Testing Models for Split-Ballot MTMM Experiments 21211.3 Empirical Examples 21311.3.1 Results for the Three-Group Design 21311.3.2 Two-Group SB-MTMM Design 21511.4 The Empirical Identifiability and Efficiency of the Different SB-MTMM Designs 21811.4.1 The Empirical Identifiability of the SB-MTMM Model 21811.4.2 The Efficiency of the Different Designs 22111.5 Summary and Discussion 221Exercises 222Appendix 11.1 The Lisrel Input for the Three-Group SB-MTMM Example 22212 MTMM Experiments and the Quality of Survey Questions 22512.1 The Data from the MTMM Experiments 22612.2 The Coding of the Characteristics of the MTMM Questions 22912.3 The Database and Some Results 23012.3.1 Differences in Quality across Countries 23112.3.2 Differences in Quality for Domains and Concepts 23412.3.3 Effect of the Question Formulation on the Quality 23512.4 Prediction of the Quality of Questions Not Included in the MTMM Experiments 23712.4.1 Suggestions for Improvement of Questions 23912.4.2 Evaluation of the Quality of the Prediction Models 24012.5 Summary 241Exercises 242Part IV Applications in Social Science Research 24313 The SQP 2.0 Program for Prediction of Quality and Improvement of Measures 24513.1 The Quality of Questions Involved in the MTMM Experiments 24613.1.1 The Quality of Specific Questions 24613.1.2 Looking for Optimal Measures for a Concept 25013.2 The Quality of Non-MTMM Questions in the Database 25213.3 Predicting the Quality of New Questions 25613.4 Summary 261Exercises 26214 The Quality of Measures for Concepts-by-Postulation 26314.1 The Structures of Concepts-by-Postulation 26414.2 The Quality of Measures of Concepts-by-Postulation with Reflective Indicators 26414.2.1 Testing the Models 26514.2.2 Estimation of the Composite Scores 26814.2.3 The Quality of Measures for Concepts-by-Postulation 27014.2.4 Improvement of the Quality of the Measure 27414.3 The Quality of Measures for Concepts-by-Postulation with Formative Indicators 27614.3.1 Testing the Models 27814.3.2 Estimation of the Composite Score 28114.3.3 The Estimation of the Quality of the Composite Scores 28214.4 Summary 283Exercises 284Appendix 14.1 Lisrel Input for Final Analysis of the Effect of “Social Contact” on “Happiness” 284Appendix 14.2 Lisrel Input for Final Analysis of the Effect of “Interest in Political Issues in the Media” on “Political Interest in General” 28515 Correction for Measurement Errors 28715.1 Correction for Measurement Errors in Models with only Concepts-by-Intuition 28715.2 Correction for Measurement Errors in Models with Concepts-by-Postulation 29215.2.1 Operationalization of the Concepts 29215.2.2 The Quality of the Measures 29415.2.3 Correction for Measurement Errors in the Analysis 29715.3 Summary 298Exercises 299Appendix 15.1 Lisrel Inputs to Estimate the Parameters of the Model in Figure 15.1 300Appendix 15.2 Lisrel Input for Estimation of the Model with Correction for Measurement Errors using Variance Reduction by Quality for all Composite Scores 30116 Coping with Measurement Errors in Cross-Cultural Research 30216.1 Notations of Response Models for Cross-Cultural Comparisons 30316.2 Testing for Equivalence or Invariance of Instruments 30716.2.1 The Standard Approach to Test for Equivalence 30716.3 Problems Related with the Procedure 30916.3.1 Using Information about the Power of the Test 30916.3.2 An Alternative Test for Equivalence 31516.3.3 The Difference between Significance and Relevance 31716.4 Comparison of Means and Relationships across Groups 31816.4.1 Comparison of Means and Relationships between Single Requests for Answers 31816.4.2 Comparison of Means and Relationships Based on Composite Scores 31916.4.3 Comparison of Means and Relationships between Latent Variables 32116.5 Summary 324Exercises 325Appendix 16.1 The Two Sets of Requests Concerning “Subjective Competence” 326Appendix 16.2 ESS Requests Concerning “Political Trust” 327Appendix 16.3 The Standard Test of Equivalence for “Subjective Competence” 328Appendix 16.4 The Alternative Equivalence Test for “Subjective Competence” in Three Countries 329Appendix 16.5 Lisrel Input to Estimate the Null Model for Estimation of the Relationship between “Subjective Competence” and “Political Trust” 331Appendix 16.6 Derivation of the Covariance between the Composite Scores 333References 336Index 352