Screening

Prior to subject selection, we screen all potential participants to ensure they meet the set of required qualifications for testers. Here, we can also ask them to sign an NDA or other agreement before beginning.

Pre-test arrangements

During this scrfeening process, we have the participant:

• Fill out their personal information
• Set up their screen recording software

Introduction to the session

Next, we instruct the participant to follow the following guidelines throughout the testing process:

• Explain their process
• Encourage thinking aloud
• Deliver the test objective

Test Execution

Participants will start at a url you provide and are instructed to build an application.

As developers attempt the task, our software records all activity on their screen, application usage, code changes, and application usage. It also transcribes their voice through their computer’s microphone as they narrate their thoughts along the way, providing insight into how they approach the development process, what they care about, and anything that potentially confuses them.

After the task is completed (or the specified time runs out), the participant will upload the recorded video and respond to a questionnaire which asks quantitative and open-ended questions about their experience working with your product.

Post-test debriefing

During the debriefing process, we gather both qualitative and quantitative information about their experience working with your product. During this debriefing, we:

• Ask broad questions to gather information about their preferences and other qualitative data.
• Have them fill out a quantitative survey on their experience.
• Have them report any suspected bugs found in the software.
• Request that they upload their video.
• Request that they describe any particular problems that arose during the testing process.