Writing Regression Tests for Orca

This page provides information for creating the *.py regression testing files that live under each ./keystrokes/* directory. These contain keystrokes recorded using the Macaroon utility (see below) and then hand-edited to add test assertions. In addition, there may be an optional *.settings file for each *.py file. The *.settings file is an Orca settings file to use specifically for the *.py file, and is used to help test specific Orca features such as word echo, key echo, etc. There may also be a *.params file. If it exists, the contents of the *.params file will be appended to the command line used to start the application to test.

Prerequisites

See Orca/RegressionTesting.

Regression Test Overview

The basic idea behind each test is to do the following:

  1. Assume the application is already running and is in a known state
  2. Play a keystroke and wait for something to happen (e.g., a window is activated, a component has focus, etc.)
  3. Repeat the play/wait step until you're where you want to test something
  4. Tell Orca to record output via the sequence.append(utils.StartRecordingAction()) command from the harness's utils.py file

  5. Do more play/wait operations to conduct the test
  6. Assert that Orca presented the right thing via the sequence.append(utils.AssertPresentationAction(test_name, expected_output) command from the harness's utils.py file

  7. Repeat more tests - tell Orca to record, play/wait, assert
  8. Leave the application in a known state by waiting/playing more keystrokes

With each test, you should assume the application you are testing with (e.g., gtk-demo, gedit, etc.) is already up and running in a well known state, and the test should leave the application back in this well known state. The tests themselves should not start or stop the application -- let the harness do that.

Macaroon

Macaroon is an application and a library to allow you to record keystrokes and other AT-SPI event information to a Python script. The resulting Python script contains instructions to playback the keystrokes you've typed, with optional pauses inserted to wait for things such as windows being activated, objects getting focus, as well as other AT-SPI events.

To record a test, you should first start the application to be tested and leave it in its initial startup state. For example, if you want to write a test for gedit, just start 'gedit' and don't do anything in it yet.

Then, run the macaroon command and record keystrokes while you interact with the application. It's very subtle, but when you run macaroon a red dot will appear in the top panel. This red dot is your interface to controlling macaroon. When you press the red dot, the red dot will turn into a gray square and macaroon will start recording keystrokes. To stop recording, press the gray square and a macaroon window called "Macro preview" will appear containing a Python listing of all the keystrokes you've typed. The window will also contain other interesting information, such as AT-SPI events for things such as window activation and object focus.

You can copy text from the macaroon "Macro preview" window and paste it into your favorite editor for editing. This will form the core of your test file. Please also refer to other test files for examples.

Level 2 Recording

Macaroon has two levels of recording keystrokes: Level 1 and Level 2. Level 1 records at a very fine granularity -- each key press and release is recorded along with timing and keycode information.

Here's an example of what "Level 1" looks like:

# Level 1 recording.  Avoid writing tests that do this.
#
# To get Level 2 recording, right click on the Macaroon icon (a very
# subtle red dot in the panel) and select Level 2.  Newer versions of
# Macaroon should automatically come up in Level 2 recording mode.
#
sequence.append(KeyPressAction(8162, 64, "Alt_L")) # Press Alt_L
sequence.append(KeyPressAction(624, 41, "f")) # Press f
sequence.append(KeyReleaseAction(143, 41, "f")) # Release f
sequence.append(KeyReleaseAction(71, 64, "Alt_L")) # Release Alt_L
sequence.append(KeyPressAction(399, 9, "Escape")) # Press Escape
sequence.append(KeyReleaseAction(104, 9, "Escape")) # Release Escape
sequence.append(KeyPressAction(0, 46, "l")) # Press l
sequence.append(KeyReleaseAction(96, 46, "l")) # Release l
sequence.append(KeyPressAction(159, 26, "e")) # Press e
sequence.append(KeyReleaseAction(112, 26, "e")) # Release e
sequence.append(KeyPressAction(135, 55, "v")) # Press v
sequence.append(KeyReleaseAction(80, 55, "v")) # Release v
sequence.append(KeyPressAction(256, 26, "e")) # Press e
sequence.append(KeyReleaseAction(56, 26, "e")) # Release e
sequence.append(KeyPressAction(183, 46, "l")) # Press l
sequence.append(KeyReleaseAction(88, 46, "l")) # Release l
sequence.append(KeyPressAction(175, 65, "space")) # Press space
sequence.append(KeyReleaseAction(88, 65, "space")) # Release space
sequence.append(KeyPressAction(200, 10, "1")) # Press 1
sequence.append(KeyReleaseAction(103, 10, "1")) # Release 1

Because it works as a very low level, Level 1 recording is something you should avoid. Instead, you should use Level 2 recording, which helps record the typing of a string of text as well as key combinations such as Alt+F1.

Here's an example of what Level 2 recording looks like, performing nearly the same keystrokes as the Level 1 example above -- the only difference is that it types "level 2" instead of "level 1":

# Level 2 recording.  Strive to write tests that do this.
#
sequence.append(KeyComboAction("<Alt>f"))
sequence.append(KeyComboAction("Escape"))
sequence.append(TypeAction("level 2"))

Macaroon should come up by default with Level 2 recording enabled by default. To enable Level 2 recording, right click on the Macaroon icon (e.g., the subtle red dot that appears in the panel) and select "Level 2".

An auto-generated API document could be found here

Format of a Test File

A macaroon test consists of packaging up a sequence of actions in a MacroSequence and then telling that sequence to start. The following sections describe the various sections of a test file:

Preamble

All tests will have a similar preamble:

"""Test of menu accelerator label output using the gtk-demo UI Manager
   demo.
"""

from macaroon.playback import *
import utils

sequence = MacroSequence()

Wait for Application to Have Focus

The harness starts the application, which may take a variable amount of time. To make sure the application has focus before playing keystrokes, you can add a WaitForWindowActivate action to the macro sequence:

########################################################################
# We wait for the demo to come up
#
sequence.append(WaitForWindowActivate("GTK+ Code Demos"))

Perform an action, wait for something to happen. Repeat.

The next step is to get the application to a state where you're ready to test something. For example, you might want to tab to a specific checkbox or text area before running the actual test. Once the test is sure the application has focus, you can perform an action, which is typically playing a keyboard event using the KeyComboAction(keyComboString) or TypeAction(stringToType) actions. After you perform an action, you typically want to wait for something to happen in the application before performing another action. For example, the following types Ctrl+f, and then waits for a ROLE_TEXT object to get focus before typing "UI Manager":

########################################################################
# Once gtk-demo is running, invoke the UI Manager demo.
#
sequence.append(KeyComboAction("<Control>f"))
sequence.append(WaitForFocus(acc_role=pyatspi.ROLE_TEXT))
sequence.append(TypeAction("UI Manager", 1000))

Tell Orca to Start Recording Output

Once you're at a point where you want to test something, you can tell Orca to start recording it's output:

########################################################################
# Once gtk-demo is running, run through some debugging commands.
#
sequence.append(utils.StartRecordingAction())

Conduct a Test

After you've told Orca to start recording its output, conduct a test. This one tests the debugging commands and tells Orca to output script information:

sequence.append(KeyPressAction(0, None, "KP_Insert"))
sequence.append(KeyComboAction("<Control><Alt>Home"))
sequence.append(KeyReleaseAction(0, None, "KP_Insert"))

Assert the Output is as Expected

You can now assert that Orca output the expected test results:

sequence.append(utils.AssertPresentationAction(
    "Report script information",
    ["BRAILLE LINE:  'SCRIPT INFO: Script name='gtk-demo [(]module=orca.default[)]' Application name='gtk-demo' Toolkit name='GAIL' Version='[0-9]+[.][0-9]+[.][0-9]+''",
     "     VISIBLE:  'SCRIPT INFO: Script name='gtk-de', cursor=0",
     "SPEECH OUTPUT: 'SCRIPT INFO: Script name='gtk-demo [(]module=orca.default[)]' Application name='gtk-demo' Toolkit name='GAIL' Version='[0-9]+[.][0-9]+[.][0-9]+''"]))

The AssertPresentationAction takes two parameters:

  1. A name for the test -- you make this up to give someone an idea of what the test is
  2. The expected output as a list. You can get this by setting orca.debug.debugLevel = orca.debug.LEVEL_INFO in your ~/.orca/user-settings.py file.

NOTE: WDW - the above has some regular expression syntax in it. We *may* end up using it. For now, however, the expected results are really a list equality comparison. If you come across a difference that is in a very gray area, you can embed KNOWN ISSUE - reason for issue at the beginning of the expected results. The harness will recognize KNOWN ISSUE and not count it as a failure. These are to be used very conservatively and are used for handling things such as differences in output when running in asynchronous mode (normal operating mode) and synchronous mode (testing mode).

Do Another Test

You can have multiple tests in a single file. Each test really consists of:

  1. Play/wait keystroke/event operations to get to where you want to test
  2. Telling Orca to start recording
  3. Play/wait keystrokes/event operations to do the test
  4. Assert the output is as expected

Reset the application to a known state

Once you're done testing what you want to test, you should put the application back into the state in which you started. The reason for this is that the harness will not always kill the application before starting another test. So, you want to make sure you leave the application the same as it was before your test started.

Here's an example that resets the gtk-demo back to the point where the "Application main window" menu item is selected. Note that instead of waiting for the "GTK+ Code Demos" window to get focus, the test actually waits for something inside the window to get focus (a ROLE_TREE_TABLE). In addition, we also see use of a generic WaitAction that waits for the table to change state ("object:active-descendant-changed") before moving on:

########################################################################
# Go back to the main gtk-demo window and reselect the
# "Application main window" menu.  Let the harness kill the app.
#
#sequence.append(WaitForWindowActivate("GTK+ Code Demos",None))
sequence.append(WaitForFocus(acc_role=pyatspi.ROLE_TREE_TABLE))
sequence.append(KeyComboAction("Home"))

sequence.append(WaitAction("object:active-descendant-changed",
                           None,
                           None,
                           pyatspi.ROLE_TREE_TABLE,
                           5000))

Pause a moment as the last action in the sequence

By waiting just a little extra before ending a sequence, you give the system a chance to deliver events to Orca and allow Orca to process them. This helps with "repeatability" of the test results -- we want the same run of the same test using the same version of Orca on the same system to produce the same results.

# Just a little extra wait to let some events get through.
#
sequence.append(PauseAction(3000))

Postamble

Start the sequence. This is the end of the test file.

sequence.start()

A complete test can be found here.


Tips and Tricks for Writing Tests

As people like to say, "The devil is in the details." Writing a test is not as simple as recording a macaroon script and saving it. Instead, you actually need to put some thought into the process and need to spend time hand editing the macaroon output. Here's a few tips and tricks for writing tests:

Look at Other Tests

A complete test can be found here.

Copy from the Macaroon "Macro preview" window and paste it into your favorite editor

As mentioned, you can copy from the macaroon "Macro preview" window that appears when you press the gray square macaroon icon in the top panel. Don't just save the macro away and assume you are done. Instead, you should analyze the macaroon output to make sure it is doing what you think it is doing, and you might also need to insert extra lines to optimize the test behavior.

Enable "Record focus events" in Macaroon

Right click on the macaroon icon in the panel and select "Record focus events". This will instruct macaroon to listen for focus change events on objects and embed WaitForFocus actions in the script. These WaitForFocus actions are very useful to help instruct macaroon to wait for the application to get into a good state before playing more keyboard events.

Remove extra stuff Macaroon gives you

Try to put only the meaningful stuff in the file and delete stuff that is not necessary. Earlier versions of macaroon used to put extra whitespace and other parameters in the output. The WaitForFocus lines particularly had lots of extra stuff in them. This has been improved in recent versions of macaroon, where WaitForFocus events now consist of a more acceptable form:

sequence.append(WaitForFocus("Open...", acc_role=pyatspi.ROLE_MENU_ITEM))

Favor focus events over window activate events

Windows can sometimes come up before their contents do. As a result, playing keyboard events before the window is truly ready can result in keyboard events being ignored. When possible, it is usually better to wait for something inside the window to get focus. It is helpful documentation to leave a WaitForWindowActivate action in a script, however, to give the reader an idea of what to expect. Here, we see that the "GTK+ Code Demos" window should be active, but we're really waiting for the ROLE_TREE_TABLE object inside it to get focus before moving on:

########################################################################
# Go back to the main gtk-demo window and reselect the
# "Application main window" menu.  Let the harness kill the app.
#
#sequence.append(WaitForWindowActivate("GTK+ Code Demos",None))
sequence.append(WaitForFocus(acc_role=pyatspi.ROLE_TREE_TABLE))

Here's another example where we are waiting for a ROLE_PUSH_BUTTON labeled "close" to get focus in a window called "UI Manager" before moving on:

########################################################################
# Once the UI Manager window is up, open the file menu, arrow down
# over the menu items, and then close the menu.
#
#sequence.append(WaitForWindowActivate("UI Manager"))
sequence.append(WaitForFocus("close", acc_role=pyatspi.ROLE_PUSH_BUTTON))

Listen for other events

Sometimes, the event you want to wait for is not a focus event. It might be something else, such as a caret moved event. You can listen for those as well, and you typically find what you're supposed to be looking for by analyzing Orca's debug logs:

sequence.append(WaitAction("object:text-caret-moved",
                           None,
                           None,
                           pyatspi.ROLE_TEXT,
                           5000))

Avoid parts of the application that change as a result of remembered activity or system state

Some applications, such as gedit, keep track of recently opened files. You should avoid navigating menu items that run over these kinds of things. Avoidance maneuvers can be as simple as pressing the Up arrow to go to the last menu item from the first menu item instead of Down arrowing through all the menu items.

Other applications and application dialogs can expose system state. The gnome-terminal application, for example, can expose path information in the title and in its prompts. These should be avoided, and the test harness tries to make changes to the gnome-terminal settings to prevent this kind of behavior.

Use *.params files to open files

The "Open" file dialog is sensitive to directory contents and can thus present different information depending upon what is in the directory where the test happens to be running. For applications such as gedit and oowriter, it is sometimes better to create a *.params file containing a path to a file to open. With the exception of ending in params instead of py, the name of the *.params file is the same as the name of the Python test script and it also lives in the same directory as the Python script. If it exists, the harness will append the contents of the *.params file to the command line used to start the application.

For example, if you were creating a test called test/keystrokes/gedit/say_all.py that wanted to work with the test/text/SayAllText.txt file in your Orca source directory, you would create test/keystrokes/gedit/say_all.params that contained the following line:

../../../text/SayAllText.txt

The path name is relative to where the test is being run.

Debugging Tests

Here's some quick notes on debugging what might be going wrong with your test file.

  • Run the test without Orca running:
    1. Run the app to test from one shell
    2. Run the test ("python mytest.py") from another shell
    3. Quickly Alt+Tab to the application to test and observe it
    4. If it doesn't run as expected, you've got a problem in your test
  • Run the test with Orca running:
    1. Run Orca from a shell, making sure the debug level is LEVEL_ALL and output is sent to a file
    2. Run the app to test from another shell
    3. Run the test ("python mytest.py") from another shell
    4. Quickly Alt+Tab to the application to test and observe it
    5. If it doesn't run as expected, Orca might be causing some change in behavior in the app
    6. Look at the Orca debug output for stack traces
  • Use runone.sh to run the test
    1. ./runone.sh ../keystrokes/gtk-demo/role_accel_label.py gtk-demo 0
    2. Output files are saved in the directory where you ran runone.sh. Look at them, including the debug output.
  • Use runall.sh to test all the files in one directory
    1. ./runall.sh -a pwd/../keystrokes/gtk-demo

    2. Run it twice and diff the results
    3. Check your tests for repeatability
    4. Avoid doing things in your tests that make the results unrepeatable


Projects/Orca/RegressionTesting/WritingTests (last edited 2013-11-22 19:22:26 by WilliamJonMcCann)