Research Methods in Human-Computer Interaction

Dr. Jonathan Lazar is an Associate Professor of Computer and Information Sciences at Towson University, where he serves as head of the Computer Information Systems Program, and is the founder and director of the Universal Usability Laboratory.Dr. Lazar has authored over 70 refereed publications, on the topics of web usability, assistive technology, user error and frustration, and user-centered design methods. He has previously authored two books and edited two books. His most recent authored book is Web Usability: A User-Centered Design Approach, published by Addison-Wesley in 2006, and his most recent edited book is Universal Usability: Designing Computer Interfaces for Diverse User Populations published by John Wiley in 2007.Dr. Lazar is a founding member and currently serves as Chair of the Association for Computing Machinery (ACM) SIGCHI U.S. Public Policy Committee. He was also recently named a Distinguished Speaker of the ACM. He has served on a number of conference committees, including CHI, ASSETS, INTERACT, and HCII.Dr. Jinjuan Heidi Feng is the author of Research Methods in Human-Computer Interaction, published by Wiley.Dr. Harry Hochheiser is the author of Research Methods in Human-Computer Interaction, published by Wiley.

• About the Authors xviiAcknowledgments xviiiPreface xix1 Introduction 11.1 Changes in topics of HCI research over time 31.2 Shifts in measurement in HCI 51.3 Inherent conflicts in HCI 91.4 Interdisciplinary nature of HCI research 111.5 Communicating your ideas 131.6 Research and usability testing 14Summary of Chapters 14Discussion Questions 15Research Design Exercise 16References 162 Experimental Research 192.1 Types of behavioral research 202.2 Research hypotheses 222.2.1 Null hypothesis and alternative hypothesis 232.2.2 Dependent and independent variables 252.2.3 Typical independent variables in HCI research 252.2.4 Typical dependent variables in HCI research 262.3 Basics of experimental research 272.3.1 Components of an experiment 272.3.2 Randomization 282.4 Significance tests 302.4.1 Why do we need them? 302.4.2 Type I and Type II errors 322.4.3 Controlling the risks of Type I and Type II errors 342.5 Limitations of experimental research 34Summary 36Discussion Questions 37Research Design Exercises 38References 383 Experimental Design 413.1 What needs to be considered when designing experiments? 433.2 Determining the basic design structure 443.3 Investigating a single independent variable 453.3.1 Between-group design and within-group design 463.3.2 Choosing the appropriate design approach 493.4 Investigating more than one independent variable 533.4.1 Factorial design 533.4.2 Split-plot design 543.4.3 Interaction effects 553.5 Reliability of experimental results 573.5.1 Random errors 573.5.2 Systematic errors 573.6 Experimental procedures 63Summary 64Discussion Questions 65Research Design Exercises 65References 664 Statistical Analysis 694.1 Preparing data for statistical analysis 704.1.1 Cleaning up data 704.1.2 Coding data 714.1.3 Organizing data 734.2 Descriptive statistics 734.2.1 Measures of central tendency 734.2.2 Measures of spread 744.3 Comparing means 744.4 T tests 764.4.1 Independent-samples t test 764.4.2 Paired-samples t test 764.4.3 Interpretation of t test results 774.4.4 Two-tailed t tests and one-tailed t tests 784.5 Analysis of variance 784.5.1 One-way ANOVA 794.5.2 Factorial ANOVA 804.5.3 Repeated measures ANOVA 824.5.4 ANOVA for split-plot design 834.6 Assumptions of t tests and F tests 864.7 Identifying relationships 864.8 Regression 894.9 Nonparametric statistical tests 914.9.1 Chi-square test 924.9.2 Other non-parametric tests 94Summary 94Discussion Questions 95Research Design Exercises 96Team Exercises 96References 965 Surveys 995.1 Introduction 1005.2 Benefits and drawbacks of surveys 1015.3 Goals and targeted users for survey research 1025.4 Probabilistic sampling 1035.4.1 Stratification 1055.4.2 Response size 1065.4.3 Errors 1065.5 Non-probabilistic sampling 1075.5.1 Demographic data 1075.5.2 Oversampling 1085.5.3 Random sampling of usage, not users 1095.5.4 Self-selected surveys 1095.5.5 Uninvestigated populations 1095.6 Developing survey questions 1115.6.1 Open-ended questions 1115.6.2 Closed-ended questions 1125.6.3 Common problems with survey questions 1135.7 Overall survey structure 1135.8 Existing surveys 1155.9 Paper or online surveys? 1165.10 Testing the survey tool 1185.11 Response rate 1195.12 Data analysis 120Summary 121Discussion Questions 121Research Design Exercise 122References 1226 Diaries 1256.1 Introduction 1266.2 Why do we use diaries in HCI research? 1276.3 Participants for a diary study 1306.4 What type of diary? 1326.4.1 Feedback diary 1326.4.2 Elicitation diary 1336.4.3 Hybrid feedback and elicitation diary 1346.5 Data collection for the diary study 1346.6 Letting participants know when to record a diary entry 1366.7 Analysis of diaries 137Summary 138Discussion Questions 138Research Design Exercise 138References 139Appendix A Frustration Experience Form (Time Diary) 140Appendix B Excel Time Diary Form 1417 Case Studies 1437.1 Introduction 1447.2 Observing Sara: a case study of a case study 1457.3 What is a case study? 1477.3.1 In-depth investigation of a small number of cases 1477.3.2 Examination in context 1477.3.3 Multiple data sources 1487.3.4 Emphasis on qualitative data and analysis 1497.4 Goals of HCI case studies 1507.4.1 Exploration 1507.4.2 Explanation 1517.4.3 Description 1527.4.4 Demonstration 1547.5 Types of case study 1567.5.1 Intrinsic or instrumental 1567.5.2 Single case or multiple cases 1567.5.3 Embedded or holistic 1607.6 Research questions and hypotheses 1617.7 Choosing cases 1637.8 Data collection 1647.8.1 Data sources and questions 1647.8.2 Collecting data 1657.9 Analysis and interpretation 1677.10 Writing up the study 1687.11 Informal case studies 170Summary 172Discussion Questions 174Research Design Exercises 174References 1758 Interviews and Focus Groups 1778.1 Pros and cons of interviews 1788.2 Applications of interviews in HCI research 1808.2.1 Initial exploration 1808.2.2 Requirements gathering 1848.2.3 Evaluation and subjective reactions 1868.3 Who to interview 1878.4 Interview strategies 1898.4.1 How much structure? 1898.4.2 Focused and contextual interviews 1918.5 Interviews vs focus groups 1928.6 Types of question 1948.7 Conducting an interview 1978.7.1 Preparation 1978.7.2 Recording the responses 1988.7.3 During the interview 1998.8 Electronically mediated interviews and focus groups 2038.8.1 Telephone 2048.8.2 Online 2048.9 Analyzing interview data 2068.9.1 What to analyze 2078.9.2 How to analyze 2088.9.3 Validity 2128.9.4 Reporting Results 212Summary 213Discussion Questions 214Research Design Exercises 214References 2159 Ethnography 2179.1 Introduction 2189.2 What is ethnography? 2199.3 Ethnography in HCI 2219.4 Conducting ethnographic research 2249.4.1 Selecting a site or group of interest 2259.4.2 Participating: choosing a role 2279.4.3 Building relationships 2309.4.4 Making contact 2319.4.5 Interviewing, observing, analyzing, repeating, and theorizing 2329.4.6 Reporting results 2369.5 Some examples 2379.5.1 Home settings 2379.5.2 Work settings 2389.5.3 Educational settings 2399.5.4 Ethnographies of mobile and ubiquitous systems 2409.5.5 Virtual ethnography 241Summary 246Discussion Questions 246Research Design Exercises 247References 24810 Usability Testing 25110.1 What is usability testing? 25210.2 How does usability testing relate to traditional research? 25410.3 Types of usability testing or usability inspections 25610.3.1 Expert-based testing 25610.3.2 Automated usability testing 25810.4 User-based testing 26010.4.1 Types of usability testing 26010.4.2 Stages of usability testing 26210.4.3 How many users are sufficient? 26310.4.4 Locations for usability testing 26410.4.5 Task list 26810.4.6 Measurement 27010.4.7 The testing session 27110.4.8 Making sense of the data 27410.5 Other variations on usability testing 275Summary 276Discussion Questions 276Research Design Exercise 277References 27711 Analyzing Qualitative Data 28111.1 Introduction 28211.2 Stages of qualitative analysis 28211.3 Grounded theory 28311.4 Content analysis 28511.4.1 What is content? 28611.4.2 Why do we need to collect text or multimedia information? 28611.4.3 Questions to consider before content analysis 28711.5 Analyzing text content 28911.5.1 Procedure 28911.5.2 Identifying coding categories 29011.5.3 Coding the text 29211.5.4 Ensuring high-quality analysis 29411.6 Analyzing multimedia content 300Summary 301Discussion Questions 302Research Design Exercise 303References 30312 Automated Data Collection Methods 30712.1 Exploiting existing tools 30812.1.1 Web logs 30912.1.2 Stored application data 31512.2 Using software to observe and record 31712.2.1 Web proxies 31712.2.2 Instrumented software 32112.2.3 Custom-built software 32412.2.4 Handling stored data 32712.2.5 Keystroke and activity loggers 32812.2.6 Analyzing log files 32912.3 Hybrid data collection methods 33012.4 Automated interface evaluation 33312.5 Challenges of computerized data collection 333Summary 336Discussion Questions 337Research Design Exercises 338References 33913 Measuring the Human 34313.1 Eye tracking 34413.2 Physiological tools 35013.2.1 Physiological data 35113.2.2 Challenges in data collection and interpretation 35613.3 Examples of physiological research in HCI 359Summary 361Discussion Questions 362Research Design Exercise 363References 36314 Working with Human Subjects 36714.1 Identifying potential participants 36814.1.1 Which subjects? 36914.1.2 How many subjects? 37114.1.3 Recruiting participants 37314.2 Care and handling of research participants 37614.2.1 Protecting participants 37614.2.2 Informed consent 38114.2.3 Institutional review boards 38414.2.4 Potentially deceptive research? 38714.2.5 General concerns 38814.3 Online research 38914.3.1 Appropriate topics for online research 38914.3.2 Recruiting 38914.3.3 Study design 39114.3.4 Ethical concerns 39114.3.5 Data collection 392Summary 393Discussion Questions 394Research Design Exercises 395References 39615 Working with Research Participants with Impairments 39915.1 Introduction 40015.2 How many participants? 40115.2.1 Small sample sizes 40115.2.2 Distributed research 40115.2.3 In-depth case studies 40215.3 Proxy users 40315.4 Multi-Population Studies 40415.5 Recruiting users through community partners 40515.6 Pilot studies 40715.7 Scheduling users with impairments 40815.8 Documentation for users with impairments 40915.8.1 Human subjects forms 40915.8.2 Research documentation 41015.9 Differing levels of ability 41215.10 Bringing extra computer parts 41315.11 Payment 415Summary 415Discussion Questions 415Research Design Exercise 416References 416Index 419