Designing and Conducting Business Surveys

GER SNIJKERS, PhD, is Senior Researcher and Advisor in Business Survey Data Collection Methodology at Statistics Netherlands, after having held various research and project management positions. From 2006-2011, he held the position of Professor of Business Survey Methodology at Utrecht University.GUSTAV HARALDSEN, Mag. Art., is Senior Methodological Advisor in the Department of Data Collection and Methods at Statistics Norway. He previously headed the Division for Interviewing Surveys and the Division of Data Collection Methods.JACQUI JONES, BA (Hons), MSc, is Deputy Director of the Business Indicators and Balance of Payments Division in the UK Office of National Statistics (ONS). Prior to this, she worked in the Methodology Directorate at ONS.DIANE K. WILLIMACK, MS, MA, is Chief of the Response Improvement Research Staff at the Economic Programs Directorate of the U.S. Census Bureau. She previously served as a mathematical statistician and survey methodologist at the National Agricultural Statistics Service (U.S. Department of Agriculture).Additional contributors include Paul Smith (ONS), Deirdre Giesen (Statistics Netherlands), Li Chun Zhang (Statistics Norway), and Mike Hidiroglou (Statistics Canada).

• Preface xixContributors xxiiiAbbreviations xxv1. Surveys and Business Surveys 1Jacqui Jones, Ger Snijkers, and Gustav Haraldsen1.1. The Objective of this Book, 11.2. The Statistical Production Process, 21.3. Surveys, 41.3.1. Definition of Surveys, 41.3.2. Survey Constraints, Considerations, and Tradeoffs, 51.3.3. A Brief Historical Look at Surveys, 81.4. Types of Business Data Outputs, 101.4.1. Official Statistics, 101.4.2. Other Types of Business Data Outputs, 141.5. Use of Business Data Outputs, 161.5.1. National, Regional, and Local Economic Performance, 171.5.2. Individual Business Monitoring and Policymaking, 181.5.3. Everyday Decisions, 181.6. Obtaining Business Data, 191.6.1. Business Data Collection Methods, 191.7. Business Surveys, 221.7.1. Types of Business Surveys, 221.7.2. The Business Survey–Output Production Process, 231.7.3. Perspectives of the Business Survey Process, 261.8. Overview of the Book, 281.8.1. The Audience, 281.8.2. Organization of the Book, 291.8.3. Chapter Summaries, 29Acknowledgments, 33Appendix, 332. The Business Context and its Implications for the Survey Response Process 39Diane K. Willimack and Ger Snijkers2.1. The Business Context from the Perspective of the Organizational Sciences, 402.1.1. Business Goals and Behaviors and Implications for Survey Response, 402.1.2. Dimensions of Social Behavior in Organizations Pertinent to the Survey Response Task, 452.1.3. The Response Process Viewed as Work, 492.2. A Comprehensive Approach Integrating the Business Context and the Survey Response Process, 512.2.1. The Decision to Participate, 542.2.2. Performing Response Tasks, 602.2.3. Release of the Questionnaire, 752.2.4. Role of the Survey Organization in the Business Survey Response Process: Implications for Survey Design, 752.3. Summary, 79Acknowledgments, 823. Quality Issues in Business Surveys 83Gustav Haraldsen3.1. Survey Quality from a User/producer Perspective, 843.1.1. Users and User Evaluations of Business Surveys, 863.1.2. The Total Survey Error Approach, 883.2. Sample-Related Quality Issues, 963.2.1. A Glimpse into Transaction Studies, 963.2.2. Specifying Observational Units, 973.2.3. The Convenience Sample Element, 993.2.4. Sampling Error Issues, 1013.2.5. Reporting Units and Informants, 1023.2.6. Response Rate Issues, 1033.2.7. Quality Effects of Postsurvey Adjustments, 1063.3. Measurement-Related Quality Issues, 1063.3.1. Validity Challenges during Survey Specification, 1073.3.2. Sources of Measurement Error in Business Surveys, 1083.3.3. Measurement Issues Summarized, 1183.4. Sample and Measurement Issues Combined, 120Acknowledgment, 1254. Planning the Survey 127Ger Snijkers, Gustav Haraldsen, and Jacqui Jones4.1. Introduction, 1274.2. Business Surveys and Project Management Principles, 1284.3. Specifying and Planning the Survey, 1314.3.1. Specifying the Survey, 1324.3.2. Planning the Survey in Detail: The Planning Process, 1354.3.3. Planning the Fieldwork, 1424.3.4. Planning the Designing, Building, and Testing of the Survey Components, 1434.3.5. Planning the Data Analysis, Dissemination, and Archiving, 1514.3.6. Planning the Survey Evaluation, 1524.4. Required Resources, 1534.5. The Timetable, 1534.6. Planning Managing, Controlling, Monitoring, and Evaluation of the Survey Project, 1544.7. Risk Management, 1564.8. Finalizing the Project Plan and Getting Started, 1624.9. Summary and Progress Status Overview, 162Acknowledgment, 1635. Sampling and Estimation for Business Surveys 165Paul Smith5.1. Basic Principles, 1655.2. Registers and Frames, 1675.2.1. Business Registers, 1675.2.2. Constructing a Frame (without a Business Register), 1745.2.3. Register and Frame Quality, 1755.2.4. Dual-Frame Surveys, 1765.3. Sample Design, 1775.3.1. Some Basic Ideas, 1775.3.2. Stratified Sampling, 1785.3.3. Probability Proportional to Size (PPS) Sampling, 1955.3.4. Cutoff Sampling, 1965.3.5. Sample Rotation and Overlap, 1975.3.6. Special Sample Designs, 1995.4. Estimation, 2025.4.1. Horvitz–Thompson Estimation, 2025.4.2. Some Basic Ideas, 2035.4.3. Ratio Estimation, 2055.4.4. Combined Ratio Estimation, 2075.4.5. More Complex Estimators, 2085.4.6. Accuracy of Auxiliary Information, 2085.4.7. Optimal Uses of Auxiliary Information, 2095.4.8. Estimating Discontinuities, 2095.5. Outliers, 2105.5.1. Inspection and “Surprise Strata”, 2115.5.2. Winsorization, 2125.5.3. Other Approaches, 2135.6. Variance Estimation, 2145.7. Small Area Estimation, 2165.8. Summary, 217Acknowledgments, 2186. Understanding and Coping with Response Burden 219Gustav Haraldsen, Jacqui Jones, Deirdre Giesen, and Li-Chun Zhang6.1. A Cost versus a Cost Benefit Approach, 2206.1.1. Politicians’ Bird’s-Eye View, 2216.1.2. Commercial Worries, 2226.1.3. Perceived Response Burden, 2236.2. Measuring and Monitoring Business Response Burden, 2266.2.1. Measuring and Calculating Actual Response Burden, 2296.2.2. Measuring Perceived Response Burden, 2326.2.3. Calculations, 2346.3. An Analytical Approach to Response Burden, 2356.3.1. Disentangling Some Sources of Response Burden, 2386.3.2. Suggesting a Curvilinear Correlation between Response Burden and Response Quality, 2396.4. Measures to Reduce Response Burden, 2416.4.1. Survey Reduction Measures, 2416.4.2. Sample Coordination, 2436.4.3. Survey Communication Measures, 2486.4.4. Questionnaire Improvement Measures, 2496.4.5. Tailored Evaluations of Response Burden Reduction Measures, 251Acknowledgments, 2527. Methods for the Development, Testing, and Evaluation of Data Collection Instruments 253Diane K. Willimack7.1. A Framework for Questionnaire Development, Testing, and Evaluation, 2547.1.1. Business Survey Response Model and Measurement Error, 2567.1.2. Constraints and Limitations to Questionnaire Development, Testing, and Evaluation in the Business Survey Context, 2587.2. Developing Content, Measurements, and Survey Questions, 2607.2.1. The Roles of Subject Area Specialists, Stakeholders, and Data Users, 2627.2.2. Methods for Content Development and Specifying Measurements, 2627.3. Testing Questions and Questionnaires, 2677.3.1. The Response Model as a Framework for Testing, 2677.3.2. Methods, 2697.4. Evaluating Survey Questions and Questionnaires, 2767.4.1. Methods Used during Data Collection, 2767.4.2. Methods Used after Data Collection, 2787.4.3. Methods for Identifying Questionnaire Problems in Recurring Surveys, 2807.5. Development, Testing, and Evaluation of Electronic Data Collection Instruments, 2827.5.1. Development Strategies for Electronic Instruments, 2827.5.2. Usability Testing, 2857.5.3. Evaluating Web Survey Data Collection Instruments, 2867.6. Analyzing Qualitative Data, 2877.6.1. Data Capture, 2887.6.2. Types of Analysis, 2887.6.3. Data Reduction, 2897.6.4. Data Display, 2907.6.5. Drawing Conclusions and Verification, 2907.7. Multimethod Approaches to Development, Testing, and Evaluation, 2907.8. Organization and Logistics, 2937.8.1. Survey Organizational Structures for Instrument Development, Testing, and Evaluation, 2937.8.2. Pretesting Logistics, 2947.9. Summary, 295Acknowledgments, 298Appendix 7.A.1 Example of Conclusions from Qualitative Data Analysis: Public Trust in Statistics: A ConceptualModel of Trust Determinants, 298Appendix 7.A.2 Case Study: Logistics of Questionnaire Pretesting for Economic Survey Programs at the US Census Bureau, 2998. Questionnaire Communication in Business Surveys 303Gustav Haraldsen8.1. Communication Modes, 3038.2. Elements in Concert, 3068.3. Tailoring for Businesses, 3118.3.1. Setting the Scene, 3138.3.2. Common Access to Several Web Surveys, 3178.3.3. Layout and Functional Tailoring for Internal Data Collection, 3258.4. Phrasing Questions in Business Surveys, 3328.4.1. From Requirements to Inquiries, 3328.4.2. Clarifying Terms and Tasks, 3358.4.3. Listening to Questionnaires, 3408.5. Minimizing Response Burden, 3468.5.1. Time Saving Measures, 3478.5.2. Reducing the Cognitive Burden, 3518.6. Concluding Reflections, 357Acknowledgments, 3579. Business Survey Communication 359Ger Snijkers and Jacqui Jones9.1. Introduction, 3599.2. The Role, Objectives, and Process of Business Survey Communication, 3609.2.1. Role, 3609.2.2. Objectives, 3619.2.3. Process, 3629.3. Tailoring to the Business Context, 3739.3.1. Actors and Their Roles in the Response Process, 3739.3.2. Business Size and Economic Sector, 3749.3.3. Factors Affecting the Response Process, 3759.3.4. Questions and Complaints, 3779.4. The Survey Organization Context, 3779.4.1. Staff in the Communication Process and Their Roles, 3779.4.2. Survey Organization Policies and Procedures, 3839.4.3. Survey Communication in Perspective, 3889.5. Designing an Effective Business Survey Communication Strategy, 3899.5.1. Planning and Designing the Communication Strategy, 3909.5.2. Effectiveness of Communication Modes, 3989.5.3. Effectively Motivating and Facilitating, 4009.6. Business Survey Communication Measures, Practices, and Guidelines, 4079.6.1. Prefield Communication Measures, 4089.6.2. Field Communication Measures, 4129.6.3. Special Cases: Large Businesses and Mixed-Mode Designs, 4219.7. Summary and Example of a Communication Strategy, 426Acknowledgment, 43010. Managing the Data Collection 431Ger Snijkers and Gustav Haraldsen10.1. Introduction, 43110.2. Implementing the Survey, 43210.3. Active Fieldwork Management, 43410.4. Paradata, 43610.5. Monitoring the Quality of the Response, 44010.5.1. A Classification of Data Collection Outcome Codes, 44110.5.2. Response and Return Rates at Unit Level, 44510.5.3. Weights of Businesses, 44710.5.4. Distribution of the Response: Representativeness, 44810.5.5. Quality of the Measurements, 44910.6. Monitoring the Survey Production Process, 45110.7. Survey Evaluation and Quality Report, 45410.8. Summary, 456Acknowledgment, 45711. Capturing, Coding, and Cleaning Survey Data 459Jacqui Jones and Mike Hidiroglou11.1. Introduction, 45911.2. Data Capture, 46111.2.1. Step One: Initial Data Capture, 46111.2.2. Minimizing, Measuring, and Monitoring Errors in Initial Data Capture, 46211.2.3. Step Two: Data Transfer, 46411.2.4. Step Three: Data Capture Processing, 46511.2.5. Capturing Data from Paper Questionnaires, 46611.2.6. Electronic Scanning, 46811.2.7. Design of Paper Self-Completion Questionnaires for Electronic Scanning, 46811.2.8. Instructions for Completing a Paper Self-Completion Questionnaire to be Scanned, 47111.2.9. Minimizing, Measuring, and Monitoring Errors in Data Capture Processing, 47311.3. Data Coding, 47511.3.1. Coding Descriptions to Classifications, 47811.3.2. Questionnaire Design Implications, 47811.3.3. Two Examples of Data Coding Tools, 48011.3.4. Minimizing, Measuring, and Monitoring Errors in Data Coding, 48111.4. Editing and Imputation, 48311.4.1. An Overview of Editing and Imputation in Business Surveys, 48311.4.2. Microediting, 48511.4.3. Types of Microdata Error, 48511.4.4. Methods for Identifying Microerrors, 48911.4.5. Treatment of Microdata Errors, 49211.4.6. Identifying Errors at the Macroediting Stage, 49611.4.7. Imputation, 49811.4.8. Minimizing, Measuring, and Monitoring Errors in Editing and Imputation, 50011.4.9. Editing Strategies, 50211.5. Concluding Comments, 503Acknowledgments, 50412. From Survey Data to Statistics 505Jacqui Jones12.1. Introduction, 50512.2. Analysis, 50512.2.1. Analysis and Different Types of Statistical Outputs, 50712.2.2. Analytical Technique Considerations, 51012.2.3. Analytical Techniques for Improving Statistical Interpretation, 51312.3. Preparing for Dissemination, 52712.3.1. Preparing Statistical Tables and Datasets, 52812.3.2. The Importance of Maintaining Confidentiality, 53012.3.3. Disclosure Control Methods, 53212.3.4. Presenting the Statistics, 53612.4. Dissemination, 53912.4.1. Evaluating Statistical Dissemination, 53912.5. Archiving, 54012.6. Concluding Comments, 542Acknowledgments, 543References 545Index 597