Tutorial: Usability Testing With Pongo

This is a basic tutorial on using Pongo to conduct usability tests.

Designing your test

Usability testing usually involves asking one or more research partipants to complete a series of tasks using the software being tested.

Decide which tasks you will ask the participant(s) to complete.

Ensure that the tasks are logically ordered and follow naturally from one to another.
It is a good idea to start with an easy task - this will make the participant more relaxed and hence act more naturally.

Select a method:

Thinking aloud protocol
- This involves getting the participant to continuously describe what they are doing while the test is running.
Co-discovery method
- Get two participants to do the tasks together.
Question asking protocol
- Prompt the participant with a series of questions. Example: 'How would you check the news?'.

The aim of the briefing should be to:

Ensure that the participant(s) is fully informed about the purpose of the test what will happen to the results.
Provide the participant(s) with the information they require in order to complete the test.
Put the participant(s) at ease - so that they behave as naturally as is possible.

Important things to do in the briefing:

Explain that the research is for the GNOME Project. Explain what GNOME is.
- Example: 'The research is for the GNOME Project. GNOME is a collection of programs which is made by volunteers collaborating over the Internet. Every everything that you will see and interact with on the screen is part of GNOME.'
Describe the purpose of the test. Emphasise that it is the software that is being tested, not the participant(s).
- Example: 'The idea is to find out how to make the software easier to use, and for that we need to see how different people use it. This isn't a test of your computer skills. The purpose is to test the software.'
Describe what is going to happen with the test.
- Tell them that they will be given a series of tasks to perform with the computer.
- Explain the method. For thinking aloud protocol, instruct the participant to talk about what they are doing while they do it, for example.
- Explain that the participant(s) can withdraw from the test at any time.
Explain what will happen to the results of the test:
- State that the results will be analysed alongside the results of other tests.
- Explain that the results will be sent to the programmers who make the software so that they can make improvements.
- Describe where the video will be stored and how it will be circulated. Check with the participant(s) whether they are happy for this to happen, and offer to anonymise the desktop recording.
Ask if the participant(s) has any questions before the test.

Run pongo in a terminal.
Do the test with the participant(s).
- You may want to take discrete notes while the test is running - these can be useful when debriefing and may help with the analysis.
  - Things you could note down:
    - Note down any activities that the participant(s) found particularly difficult.
    - Anything surprising or unexpected.
- If using thinking aloud protocol, it might be necessary to prompt the participant(s) to keep talking. Examples: 'What are you doing now?', 'What are you trying to do?'.
When the participant(s) has completed all the tasks bring up the terminal again and stop Pongo with ctrl+c.

Thank the participant(s)!
It is sometimes appropriate to ask some questions after the test has finished (remember to record the answers).
- Example questions:
  - 'How did you find the exercise?'
  - 'Where there any tasks that were particularly difficult?'
  - 'Where there any aspects of the software that you especially liked?'
You may also want to review the recording of the session in order to ask the participant(s) questions about what happened during the test. There may have been moments where you did not understand what the(y) were trying to do, for example, and it could be useful to go back and ask them what was happening.
Get some basic information from the participant(s):
- Age, gender.
- Length of time using a computer.
- Previous experience with computers - which operating systems.
- Common computing activities - what do they use their computer for?
- Patterns of computer usage - at work or at home? How often? For what lengths of time?
- Remember to note the distro and version, as well as the hardware you have been using.
Ensure that the participant(s) knows how to contact you in the future, should they have any questions or requests about the test and resulting data.
You may want to give the participant(s) information about GNOME distros. A CD or a sticker could also be nice.

Publish the video and accompanying notes on the web somewhere and share it with your friendly GNOME hackers.