Formal Usability Testing Martin Frank April 1, 1997 draws from Jakob Nielsen, Usability Engineering, Academic Press, 1993 draws from Newman and Lemming, Interactive System Design, Addison-Wesley, 1995 What's usability? ----------------- Five attributes of usability: learnability - easy to learn efficiency - productivity of expert users (draw learning curve of airline ticket kiosk vs. travel agent reservation system: x - time of use y - productivity, explain "having a steep learning curve") memorability - if you haved used the system for a year and come back, then what's your efficiency? errors - no irreversible errors satisfaction - subjective measure of how plasant the UI is to use the "usability" thing became important because: relative cost of computing time to human time ->justifies minimizing human time rather than computing time (an ever-increasing percentage of computing power is dedicated to the human interface) What's (formal) usability testing? ---------------------------------- "Formal usability testing" means you observe people using your system or a prototype thereof, without interfering. The aim is statistical validity and re-producability (if you would do the same experiment on the same kind of subject you would get the same results) Why do formal usability testing? -------------------------------- can improve bottom line: - discover that users may not need contemplated features (when testing on a prototype) (relate Siemens story) - usability is an important element in getting good product reviews - product may sell better - savings in calls to the company's hot-line - time (=cost) savings in internal products - usability criteria may be part of a contract (or may not improve the bottomline: - of course: too much testing - diminishing returns) Justification slogans: "Your best guess is not good enough" - nothing is sillier than a bunch of computer scientists arguing how e.g. air traffic controllers would react to a piece of software "Designers are not users" - once you know about a design you cannot be a usability subject (because humans cannot willfully forget) "Vice Presidents are not users" - many top executives now realize that usability is becoming a competitive parameter - downside: may be imvolved in design (everyone has an opinion about the user interface, as opposed to say the networking protocols used) but: "Users are not designers" - can't replace usability testing with simply having the users design the system in the first place (lack of dialog design and layout knowledge, lack of exposure to user interfaces design alternatives) "Less is more" - usability testing helps you find a minimal design, rather than throwing every imaginable feature, option and design alternative at the user "Details matter" - e.g. color scheme The position of usability engineers within the organization ----------------------------------------------------------- Inherent tension between programmers and designers. UI groups tend to be created as a company grows over a certain size. Boeing, CHI'97, page 85 Netscape, CHI'97, page 87 Responsibility for usability engineering normally rests with the development manager rather than with the manager for product maintenace and technical support. (Possible solution: shift responsibility up in the management hierarchy.) read from CHI'95, page 531 read from CHI'95, page 540 read from CHI'95, page 553 read from CHI'96, page 481 How to conduct formal usability testing --------------------------------------- Reliability: would one get the same result if the study were repeated -> statistical analysis, e.g. confidence intervals Validity: does the result actually reflect the issues you wanted to test? Ethics Considerations - - - - - - - - - - - User interface testing is non-desctructive testing: "deep respect for subject's well-being and emotions" - communicate that you are testing the system, not the user - say that the system is experimental and has many problems (as to reinforce the point that the system is the problem studied here) - make the subject aware of any logging technique used (video etc.) - communicate that the results of the study are anonymous (so e.g. their managers will not see them!) (also, videotapes of subjects should not be shown without their explicit permission) - design the test so that the subject has an early success experience - minimize the number of observers - don't interfere with the user (first, that invalidates the study, but it also makes the test user feel stupid) - (mrf) sit back, pretend not to be watching, read a newspaper - end by stating that the test user has helped you find areas where the system needs to be improved Test Goals and Test Plans - - - - - - - - - - - - - goal: formative evaluation vs. summative evaluation (find problems and improvements to an interface vs. compare overall suitability of an interface - often to compare two) enumerate what you want to learn (example: ThomasPro study) plan: What do you want to achieve? Budget? How long is a test? Who are the users? etc. Test Tasks - - - - - - - representative of system use - cover most of the interface dialog - small enough for a lab setting, but not trivial Experiment Design - - - - - - - - - *testing something against something else (two or more design alternatives) Grizzly Bear "programmer" study book example of pie menus vs. pull-down menus - eliminate all factors other than the menu factor *testing something by itself ThomasPro study Grizzly Bear "non-programmer" study materials for testing: ---------------------- description of what the goals are and how the experiment is designed pre-experiment questionnaire (if there is one) instructions for experiment (for experimenter and/or subject) time sheet to keep track of how long it takes the user to accomplish tasks post-experiment questionnaire presentation of results - maybe a table, summary techniques for testing: ----------------------- pilot studies verbal protocol ("thinking aloud")