Assignment 2: Usability Evaluation
Digital Humanities 110 (S’22)
James Yoon
What is Usability Testing (UT)?
Usability testing uses actual users (ideally from the target demographic group) to collect data on an interface’s effectiveness, efficiency, and satisfaction.
| Parameter | Definition |
|---|---|
| Effectiveness | Whether users are able to perform a specified task and achieve a goal. |
| Efficiency | How quickly the user can complete this task. |
| Satisfaction | How the user feels after completing this task; the attitudes associated with the task. |
In addition to the three parameters above, UT can also provide insight into learnability (how easy it is for new users to learn tasks), memorability (whether users can perform these tasks after a break), and usefulness (a combination of usability and utility - whether it provides information these users would need or are interested in). By using UT, we can create user centered design by iterating the design process depending on the UT results.
It is often useful to couple heuristic analysis by experts with UT. Heuristic analysis can find more severe errors and can prime the issues you want to focus on in following UT runs, while UT uncovers more global usability problems and predicts end-user problems better than heuristic analysis. These two methods have little overlap, so it’s good to use both when identifying problems with interfaces.
For my heuristic analysis of Ultimate-Guitar, I found that there were quite a few problems with consistency (namely what “Shots” in the main menu refers to), flexibility (that it’s too tailored to advanced users and guitar players), and aesthetics (too much information on the homepage).
To accomodate this, I designed four tasks for UT (as this assignment entailed at least 3 tasks for our pilot UT):
| Task | Description |
|---|---|
| 1 | Navigating the Courses page (ensures that beginners can still access appropriate material) - Heuristic 7 |
| 2 | Transposing and reading a guitar tab (ensures that users can access important guitar tab changes) - Heuristic 2/6/usability as a whole |
| 3 | Watching and interacting with Shots (addresses consistency in navigation, interacting with multimedia) - Heuristic 4 |
| 4 | Entering and interacting with recent forum threads (partly aesthetics and flexibility - can a user navigate to forums without being distracted by the other modules?) - Heuristic 8 |
This UT will serve as a pilot. By analyzing how well this test goes, we can improve these UT tasks for an official round later this quarter.
Materials
UT entails the following:
- A moderator: James Yoon
- Participant: Chelsey Wang (fourth year at UCLA, she/her)
- Website in question: Ultimate-Guitar.com
- Consent Form + Survey: The link to the survey can be found here.
- Place: portable minimalistic lab setting (UCLA classroom)
- Recording software: Zoom
Video Recording
If the embedded video does not work, the video can also be found here.
Note that in the video, the participant’s face was not recorded. This was due to a number of factors:
- My participant could not take off their mask.
- The Zoom meeting was recorded locally and under the “Shared Window” layout, which may have potentially also disabled the camera recording. According to Zoom, the face camera only operates when the shared screen does not take up the full screen. Since this UT entailed sharing one’s full screen due to both the survey and the website being on the Desktop at the same time, this may have been why. If I were to conduct another UT, I would record to the Cloud, as there is more flexibility in the layout there.
However, in the future, official UT will use the facecam to identify key facial expressions during the test. Instead, she verbalized what she was thinking, and some of the inflections help us understand how she was feeling at the time.
Reflections
The usability test was a useful pilot run. It didn’t go perfectly, but I feel like I learned a lot about how to conduct these tests and how to be a better moderator. This was also my first time being a moderator (I was the participant during the in-class activity), so it was a learning process as well.
What went well!
One thing that went smoothly was that my participant was very vocal. She described what she was thinking as she was performing the task, which made it easy to see where she focused her attention on and why certain tasks might have been easier/harder for her. One observation she made was that the videos on Ultimate-Guitar are similar to the controls on TikTok - this connects with the consistency heuristic and highlights how one can make design more learnable.
Furthermore, there were times when she laughed or where how she was feeling was evident from her tone of voice, which helps identify what specific emotions users may feel when prompted to navigate through the website. This made up for the lack of video, but I do want to use video + hopefully no masks later in my official UT so that I can couple these inflections with facial expressions.
Another strength was that I got more comfortable with moderating as the testing began. By the third and fourth task, I got the hang of how much I should talk and encourage my participant, although it was a bit awkward in the first two tasks. I definitely improved in starting to redirect questions!
What to improve on!
There were quite a few ways in which I could’ve improved my testing.
-
First, the Google form I created forced all respondents to fill out “For what purpose do you use Ultimate-Guitar.com?” and “Which type of device do you usually use to access Ultimate-Guitar.com?” on the Background Questions section, even if the respondent had never used Ultimate-Guitar before. I had my participant put “N/A” and the device they were using at that moment, respectively, but I went back afterwards and made those questions optional.
-
In regards to the first two tasks, there were two places when I wasn’t sure what to do. The first was during Task 1, when my participant asked whether she should start watching the videos on the Courses page to find the answer. I was nervous that this would take too much time (it took me quite a bit of time when I designed this survey, and I wanted to be respectful of her time), so I told her she could continue with the next task or start watching the videos to find the answer. Part of my rationale was that I intentionally designed that task to take a lot of time because I think locking the videos until you watch the previous one makes for bad flexibility, but I wasn’t too sure how to proceed with the physical and time constraints of UT.
-
On a related note, time! The tasks went a bit shorter than expected because of Task 1. I said that she could skip on to the next task without watching videos, since I wanted to be respectful of her time. If I did encourage her to start watching the videos to find the answer in Unit 2, the UT would’ve taken up to 1 hour, which was also the point of that task - I wanted to show that this information wasn’t readily available and was hidden behind locked videos. I should either decide on guidelines for how to approach this task or redesign this task on finding the guitar string names entirely.
-
The second was during Task 2 when she asked what an alternate guitar fingering was. She has never played guitar before, so it was expected that she didn’t know what it meant. However, if I were to try to find people in my desired demographic group (i.e. middle-aged men interested in learning guitar), I’m not sure whether guitar fingering charts/finger placements would be jargon for them as well.
What was interesting was that I could use my understanding of the 10 usability heuristics to inform my UT questions. I’m not sure if I did this the best (e.g. asking what the “Shots” tab would do in the pre-test questionnaire, designing Task 1 to take such a long time), but this data does confirm some of the potential flaws that I identified. In addition, since she can be considered a beginner guitar player, this pilot test supports my suspicion that Ultimate-Guitar is a bit too tailored for intermediate and advanced guitar players. It was a fun learning experience, and I’m excited to conduct more UT in the future.