Investigation 4: Usability testing

Assigned

Friday, 23 April 2021

Summary

In this assignment you will practice designing, running, and interpreting a small, informal, think-aloud usability study. You should gain experience both facilitating and observing.

Collaboration

You will work in a team of two students. You may also consult others about your work.

Preparation

Before you start this assignment, you should read about usability testing on Usability.gov and work on your reading journal.

Assignment

• Identify a Web site or mobile application to test. For example, you might choose https://iowapublicradio.org.
  • If you prefer, you may use a Web site or app you have built.
  • Please do not use a particularly popular Web site or app, such as Amazon, Facebook, or Google, unless you plan to focus on functionality that you expect is rarely used. The goal of this assignment is to see how people work with less familiar material and when people use a function often, they tend to find ways to work around usability problems or just take them for granted.
• We would normally have you recruit three participants. However, we will instead use the other members of class as participants and conduct the usability studies during class time.
• Develop three small but realistic tasks that someone might want to do using the Web site. These should be relatively high-level tasks; please don’t give clues as to how to achieve them. And please don’t choose tasks that require the exchange of personal information or money. For example, here are some scenarios for IPR.
  • Determine which frequencies we should be able to use to hear IPR on the radio in Grinnell. (Yes, we could listen on our computers, but we’ll ignore that.)
  • I like to listen to “The Folk Tree” but always forget when it is. When can I listen?
  • I heard a show I liked Tuesday at 11 am. What was it?
• Sketch out a script for the facilitator. I recommend that you start with the sample scripts provided by Steve Krug or Usability.gov.
• Pilot-test your secnarios and script by doing a practice session within your team or with Mai. Make sure the scenarios are clear and that the tasks can be completed in 15 minutes or less. Revise as necessary.

• Carry out your usability tests (in class). Assign each team member a role, ensuring that there is a facilitator and at least one observer. Every team member should experience each role at least once.

  Remember that it is ok to provide a hint if the user is clearly stuck. You should abort the test if you run out of time or things otherwise go really wrong. If you find after your first participant that your tasks are unclear, you may revise them.

• Debrief on the test sessions. Your goal is to identify the three most serious usability problems you saw.
• Finally, prepare your report as a team. (Sorry, that won’t happen in class.)

Report

Write a short report structured as follows. You are encouraged to include sketches, screen shots, short video clips, or other representations.

Method (5 points)

Document your approach to the usability test.

• Name the site or app you tested and explain why you chose it.
• Present your scenarios (a bulleted list is fine) and briefly explain why you chose them.
• Describe your participants. (Do not identify them by name.)
• How did you collect data from the sessions? Did you record the sessions? If so, with what tools?

Usability problems (12 points)

Identify the three most serious usability problems you saw. For each problem, address the following questions.

• What was the problem? Describe it as clearly and objectively as possible.
• How many participants experienced the problem?
• How severe was the problem? (Your scale might range from “a minor annoyance” to “prevented participant from completing the task”.)
• What do you recommend the site developers do to resolve the problem?

Reflection (3 points)

Briefly reflect on three of the following questions (whichever you find most interesting).

• Did you create the site/app you tested, or did someone else?
  How did that affect your ability to conduct the usability test?
• How do you think the use of other students in the class as test subjects affected the process?
• How did you revise your scenarios after the pilot? Why? (Do not say “We didn’t”; if that’s the case, chose another question.)
• What did you find most challenging about each role (facilitatior and observer)?
• Did you discover additional problems by observing the second and third participants? Do you think you would have learned more by
  observing more participants?
• Are your proposed solutions the absolute best solutions you can think of (“the perfect”) or the least work possible to make things better (“the good”)? Why did you make that choice?

Acknowledgements

This assignment is closely based on one Janet Davis wrote for the Spring 2015 section of HCI, although it has been adapated to the complexity of an online accelerated term. That assignment was based on one Dr. Davis has previously written for CSC-105, The Digital Age.