About the Test Design

Printable version of the Test Design (.pdf).

The Test Design will involve the creation of the Test Plan, a document with multiple sections, many of which you will have drafted in the Background Analysis section, including: Scope, Purpose, Audience Analysis, Web Site Analysis, Scenarios, Questions (including pre- and post-questions), Data to be Collected, Roles, and a few others. A Note-taking Guide and IRB documents, including questionnaire and consent form, will also be prepared and handed in. A pilot test will also be conducted. The goal of this section of the Usability Study is to prepare the test and its associated documents.

The Test Plan

The Test Plan is a single document which contains the following sections. See Usability.gov’s test plan discussion for more detail on each section, but remember that the length of our study is considerably less than the one imagined by usability.gov. (This document is adapted from the Test Plan described at the above link and Clay Spinuzzi’s):

Scope
Indicate what you are testing: Give the name of the Web site and the section you are going to focus on. Also specify how much of that section the test will cover.

Purpose
Identify the concerns, questions, and goals for this test. These can be quite broad; for example, "Can users navigate to important information from the prototype’s home page?" They can be quite specific; for example, "Will users easily find the search box in its present location?"

Audience Analysis
See Background Analysis

Web Site Analysis
See Background Analysis

Scenarios and Tasks
See Background Analysis. Total of 4 Scenarios (very short stories that motivate participants to attempt tasks) and associated tasks participants will accomplish.

Pre-test Questions
These questions establish background data for your participants and allow them time to become comfortable with the test facilitator(s) and setting. The typically ask participants about their web use, their technology skill level, their comfort with computers and other technologies, their gender, age, ethnicity, and so forth. Adapt from usability.goc test screen questions and other resources (see right sidebar).

Post-test Questions
We are going to be using the System Usability Scale for post-test questions. Please come up with 10 questions based on the ones in the article "SUS – A quick and dirty usability scale" by John Brooke (Word version; PDF version).

Data to be Collected
What will you measure? Think in terms of:

• things that you can measure during each task and/or the interview
• that are related to the purpose
• and that are relevant to the scope

Roles
Who will do what in the usability test? Think in terms of your group members’ responsibilities. Common roles include (you can combine these roles and/or create new roles if applicable):

• Briefer: brief’s participant on the test, describes IRB documents, and administers pre- and post-test questionnaire; ensures that all tasks are completed.
• Narrator: leads the participant through the scenarios, tasks, and is familiar with Think Aloud protocols for usability tests
• Note-taker: takes notes on the participant’s actions during the test
• Technology Coordinator: ensures that all the applications are running; starts and ends the screen recorder, saves file with meaningful names in the group’s folder in the open area.

Note-taking Guide

The Note-taking Guide is perhaps the most important document in a usability study as it will allow the Note-taker to record all of the participant’s actions, including their progress, the routes they take to complete tasks, and if tasks are actually completed. It will complement the screen recording that we get (assuming that everything will be ready in time) and guide your findings. Please use and adapt the note-taker guide template created by Usability.gov (.doc).

IRB Documents

Institutional Review Board (IRB) Documents are completed once the entire study is designed. They include the Review Application and the Consent Form. For more information see Rowan’s IRB site. Download and use the Review Application, Checklist, and Informed Consent forms. Please complete the forms as if you were going to have them review by the IRB board. Because you are students, however, the professor must be listed as a Primary Investigator noting parenthetically that he is the faculty sponsor. Each group should determine the Primary Investigator and the remaining group members will each be listed as a Co-Primary Investigator. There is really no distinction in status between the Primary and the Co-Investigators; the Primary is essentially the person who would be contacted if there were any IRB Board questions. View a sample IRB application (.pdf, note the templates used might be different than what appears on the current Rowan site).

Due Dates

Week 5: 2/21
Draft of Note-taking Guide, Scenarios, Pre- and Post-test Questions

Week 6: 2/28
Complete Draft of Test Plan

Week 7: 3/6
Final Test Plan, Note-taking Guide, and IRB Documents (please bring 8 printed copies of the consent form only)