HCI International 2011

T12: Practical Statistical Methods for User Research

Half Day Tutorial

James R. Lewis (short bio)
IBM Software Group, USA

Jeff Sauro (short bio)
Oracle, USA

Objectives:

If you don’t measure it you can’t manage it. Usability analysis and user-research is about more than rules of thumb, good design and intuition: it’s about making better decisions with data. Is Product A faster than Product B? Will more users complete tasks on the new design? Did we meet our goal of a 75% completion rate? What sample size should we plan on for a survey, or for comparing product? Will five users really find 85% of all problems? Learn how to conduct and interpret appropriate statistical tests on small and large sample usability data, then communicate your results in easy to understand terms to stakeholders.

Content and benefits:

• 09:00  Introduction
• 09:15  Did you achieve your testing goal?
  • Using t-test and confidence intervals based on t for continuous usability data
  • Using binomial tests and binomial confidence intervals for discrete usability data
• 10:05  Is there a statistically significant difference?
  • Between- and within-subjects t-tests (continuous data)
  • Break (1030-1100)
  • Chi-square and Fisher tests (discrete data)
• 11:25  How many participants do you need?
  • Sample size estimation for summative usability testing
  • Sample size estimation for formative usability testing
• 12:15  Conclusion

Target audience:

Open to anyone who’s interested in quantitative usability tests. Participants should be familiar with the process of conducting usability tests, have an understanding of basic descriptive statistics such as the mean, median and standard deviation, and have access to Microsoft Excel. Participants should be familiar with the process of conducting usability tests, have an understanding of basic descriptive statistics such as the mean, median and standard deviation, and have access to Microsoft Excel. The attendees should plan to bring laptops (Mac or PC) to get the most out of excel based statistics calculator. Without a laptop participants can still participate in the exercises and download the material after the course.

Relevant links:

• µMeasuring Usability
• UsableΣStats

Brief Biographical sketch(es):

Jeff Sauro is a Usability Engineer and statistical analyst with over a decade of experience conducting quantitative usability and statistical analysis for Oracle, Intuit, PeopleSoft, PayPal, Sage Software and General Electric. Jeff has presented and taught courses at CHI, HCII, HFES and UPA. He was the guest editor for Interactions Magazine dedicated to Quantifying Usability. He holds a Masters from Stanford University from their school of Education specializing in teaching quantitative concepts. His teaching style is to work backwards from practical problems that the usability practitioner is likely to encounter when quantifying usability. From these problems he takes actual usability data and shows how to make better decisions with numbers.

James R. (Jim) Lewis graduated with an M.A. in Engineering Psychology in 1982 from New Mexico State University, and received his Ph.D. in Experimental Psychology (Psycholinguistics) from Florida Atlantic University in 1996. He has worked as a human factors engineer and usability practitioner at IBM since 1981. He has published influential research on the measurement of usability satisfaction, use of confidence intervals, and sample size estimation for usability studies. He is on the editorial board of the International Journal of Human-Computer Interaction and the Journal of Usability Studies, and wrote the chapter on usability testing for the 3rd edition of the Handbook of Human Factors and Ergonomics (2006). From 2004-2005 he chaired the Formative Usability Testing Metrics Workgroup for National Institute of Standards and Technology (NIST). He is a BCPE Certified Human Factors Professional, an IBM Master Inventor, and a member of UPA, HFES, and APA.