User-Based Evaluations
Joseph S. Dumas

1. Introduction
   • The scope of the chapter
     • User-based evaluations have typical or potential users of a product participate in the evaluation
   • The methods covered in the chapter
     • Empirical usability testing
     • User-administered questionnaires
     • Observing users
2. Empirical usability testing
   • A brief history of evolution of empirical usability testing
     • In early days: usability testing as a quality assurance method
     • More recent days: testing iteratively and modern prototyping tools
   • The essential components of valid empirical usability testing
     • The focus is on usability
       • Not marketing issues
       • Not research issues
       • Usability test objectives drive the test plan
     • The participants are end users or potential end users
       • The user profile identifies the participants
       • A small sample size is still the norm
       • Recruiting participants and getting them to show up
     • There is a product or system to evaluate
       • The method is robust no matter what the technology
       • The method is robust throughout the development cycle
       • The Internet has not "changed everything"
         • But, simulating the product environment is often a challenge
     • The participants perform tasks with the product
       • The selection of task is critical and limiting
       • The task is presented as a task scenario
         • A attempt to place the task in the use and user environment
         • Does this make any difference?
       • Note: a usability test is not a focus group discussion
     • The participants are observed and data are recorded and analyzed
       • Thinking aloud is a key component
       • The roles of the test administrator
       • Getting developers/managers to watch tests continues to sell testing
       • The usability lab is now ubiquitous
         • Usability testing to go: the portable lab
       • Testing in the work environment
       • Performance measures and subjective measures
         • The measures have not changed much
         • The emphasis on qualitative measures
       • Analysis and interpretation of results
         • Most problems are uncovered during testing not afterwards
         • Triangulation of measures is critical
         • Assigning a priority to the problems is an art
         • Inferential statistics are not part of problem diagnosis
     • The results of the test are communicated to an audience
       • The evolution of informal reporting
       • Proposing solutions - going beyond evaluation
       • When to use a highlight tape
   • Important variations on the essentials
     • When "informal" really means "invalid"
       • "Quick and clean" not "quick and dirty"
     • Variations on focus and objectives
       • Comparing the usability of products
         • The test design migrates toward the research design
       • Creating a usability baseline
     • Variations on users
       • The one user case study
       • More than one user at a session
       • Including users with disabilities
     • Variations on product
       • The special requirements of voice-based systems
       • Systems vs. stand-alone products
     • Variations on tasks
       • Walking through tasks vs. unrestricted task performance
       • Letting the users do what they want - allowing free exploration
     • Variations on observations, recording, and analysis
       • Where the observations are made
         • Bringing the use environment to the test
         • Remote testing, reaching the unreachable users
       • The measures
         • Automatic data collection
   • Challenges to the validity of empirical usability testing
     • How do we evaluate empirical testing?
       • If its not research, what criteria do we apply?
     • Is usability testings' face validity deceiving?
     • Why don't observers see the same problems?
     • Are the test administrator's roles in conflict?
     • Why can't we map usability measures to UI components?
     • Is the products' operational environment being ignored?
   • Additional issues
     • How do we evaluate ease of use?
     • How does usability testing compare to other evaluation methods?
       • The comparison studies and Gray and Salzman's criticism
       • A key question: Does usability testing find unique problems?
     • Is it time to standardize usability testing methods?
       • The NIST effort
   • Final note: the boundaries are still fuzzy but confidence is high
3. User-administered questionnaires as stand-alone evaluations
   • SUS, QUIS, SUMI, etc
4. Observing users as an evaluation method
   • The limitations of observation
   • Private camera conversation