<<TableOfContents: execution failed [Argument "maxdepth" must be an integer value, not "[1]"] (see also the log)>>

Tutorial: Writing a Basic Gaim Perk

Brett Clippingdale ( dbclippi@us.ibm.com ), Peter Parente ( pparent@us.ibm.com )
Last updated for LSR version 0.5.0

Introduction

LSR scripts (called Perks) provide alternative user interfaces for applications that expose information using the GNOME Accessibility ToolKit (ATK) and the Assistive Technology Service Provider Interface (AT-SPI). One potential use for LSR scripts is to improve non-visual access to desktop applications. For instance, the default LSR script (DefaultPerk) does a decent job of reporting information about accessible applications using text-to-speech synthesis. However, the basic user experience created by the DefaultPerk for the the Gaim instant messaging client suffers from some accessibility and usability problems:

1. Events outside the focused control are not announced automatically by the DefaultPerk. Therefore, incoming chat messages are not read unless the message history has focus.

2. No hotkey exists in a Gaim chat window to switch focus to the message history. It must be mouse clicked or given focus programmatically so that the user may browse its content with the text caret.

This tutorial explains how to write a basic LSR script to solve both of these problems using speech output only. First, the tutorial introduces the use of the lsr command line options to generate and register a new Perk template. Next, it covers the topic of registering Tasks to handle accessibility events, particularly events indicating a new message has been received. Then, the tutorial sidetracks to discuss the use of the at-poke tool as a way of learning about the structure of an accessible application. Finally, the tutorial puts the information gleaned from at-poke to use in defining a hotkey that switches focus between the message history and message composition text boxes.

By the end of this tutorial, you should be familiar with the following:

1. Basic lsr command line options for generating, installing, and associating Perks.

2. Portions of the TaskTools API, a set of methods provided by LSR to simplify the writing of scripts.

3. The use of at-poke for exploring the structure of accessible applications.
4. The use of LSR hotkey for refreshing and announcing loaded Perks.
5. The use of the monitors packaged with LSR that aid in the debugging of Perks.

The completed Perk that you will develop in this tutorial can be downloaded as a reference: GaimPerkBasic.py

Prerequisites

This tutorial assumes you have installed LSR version 0.3.0 or higher by following the getting started instructions and version 1.5 of the Gaim client.

Though not required, you may also find it useful to read the overview of the LSR architecture in the LSR workbook before starting.

Note that the accessibility implementation of Gaim 2.0 is different than that of 1.5. Some of the techniques used here to read or manipulate Gaim 1.5 will not work with 2.0. See the code in the GaimPerk packaged with the LSR core for ideas about supporting multiple versions of Gaim. The advanced Gaim Perk tutorial explains that code in more detail.

Also note that LSR's TaskTools API is still in development. We expect the Perk writing experience to become easier over time. The methods and techniques used in this tutorial are based on the 0.3.0 version of the API, and, therefore, may not include the latest LSR abilities.

Generating a Perk Using a Template

To begin, you'll first want to remove the GaimPerk that ships with LSR so it doesn't interfere with the example Perk you're about to write. You can remove it with the following command:

lsr -d GaimPerk --profile=developer

Now you'll create a new script and register it with LSR to load when the Gaim chat client is activated. The lsr command line script can be used to generate (--generate or -g) a new boilerplate Perk. In your home directory, do the following:

lsr -g GaimPerkBasic.py perk --app=gaim -p developer

The name is the name of the Perk class and its identical Python file name (the .py extension will be added, of course!). The kind specifies it to be a Perk extension. The optional app argument states that the Perk should only be associated with the Gaim application given its process name (i.e. gaim all lowercase). This command sets the new Perk to load only when the developer profile is used.

Now run the Gaim application. Because Gaim does not initialize support for accessibility on its own like most gtk apps, you must run it with a special environment flag set:

GTK_MODULES=gail:atk-bridge gaim &

Whenever you run LSR using the developer profile and switch to the Gaim application, your new Perk will be activated. Try it:

lsr -p developer

When you give focus to a Gaim window, you should hear, "GaimPerkBasic handling a focus change." The generated Perk code defines an event handler called a Task that responds in this manner to focus events. All that remains to enable the desired features is to customize the Perk code with new, more useful Tasks. ^{1 2}

At this point, you should leave LSR running while continuing to read this document. If LSR is jabbering, press the right Ctrl key to silence it temporarily. If its speech becomes an incessant annoyance, press the left and right Ctrl keys simultaneously to mute LSR indefinitely. It can be unmuted later by pressing the same keys again.

Overview of a Perk

You should now briefly examine the code of GaimPerkBasic.py. You won't understand it all yet, but the rest of this tutorial will explain it in detail. For the moment, just look at the code and see if you can follow this overview:

1. The GaimPerkBasic class is a subclass of the Perk base class. It is instantiated when Gaim is activated.

2. The init method in the Perk class is called after instantiation. It registers an instance of the HandleFocusChange class as an event handler.

3. The ReadPerkName class is a subclass of the Task base class. It is instantiated by the GaimPerkBasic instance.

4. Its execute method is called whenever the use presses the key combo, CapsLock-A, bound to this instance by the GaimPerkBasic init method.

5. The HandleFocusChange class is a subclass of the Task base class. It is instantiated by the GaimPerkBasic instance.

6. Its executeGained method is called whenever a widget in the Gaim application receives focus. It performs the following actions:
   1. Prevents the next call to stop output.

   2. Speaks the phrase, "GaimPerkBasic handling a focus change."

   3. Returns nothing (or True) to allow other Perks which have registered for focus events to handle this event.

   1 '''
   2 <insert purpose of Perk here>
   3 
   4 @author: <author's name>
   5 @organization: <author's organization (or None)>
   6 @copyright: Copyright (c) <year and copyright holder>
   7 @license: <license name>
   8 
   9 All rights reserved. This program and the accompanying materials are made 
  10 available under the terms of the <license name> which is available at
  11 U{<license URL>}
  12 '''
  13 # import useful modules for Perks
  14 import Perk, Task
  15 from POR import POR
  16 from i18n import bind, _
  17 
  18 # metadata describing this Perk
  19 __uie__ = dict(kind='perk', tier='gaim', all_tiers=False)
  20 
  21 # to support translation of strings in this Perk, uncomment the following line
  22 # and provide the proper domain and path to your translation file
  23 # _ = bind(domain, locale_dir)
  24 
  25 class GaimPerkBasic(Perk.Perk):
  26   '''
  27   Example Perk template. Use this file as a way to get started writing your 
  28   own Tasks and creating your own keyboard bindings. The getDescription method
  29   should be changed to return a translated description of this Perk. The 
  30   description should include a statement of which Tiers this Perk applies to
  31   by default.
  32   '''
  33   def init(self):
  34     self.registerTask(HandleFocusChange('read test focus'))
  35     self.registerTask(ReadPerkName('read perk name'))
  36     kbd = self.getInputDevice('Keyboard')
  37     self.addInputModifiers(kbd, kbd.AEK_CAPS_LOCK)
  38     self.registerCommand(kbd, 'read perk name', True, 
  39                          [kbd.AEK_CAPS_LOCK, kbd.AEK_A])
  40                          
  41   def getDescription(self):
  42     return _('Applies to gaim by default.')
  43 
  44 class ReadPerkName(Task.InputTask):
  45   '''
  46   Example Task that announces the name of this Perk.
  47   '''
  48   def execute(self, **kwargs):
  49     self.stopNow()
  50     self.sayInfo(text='GaimPerkBasic handling an input gesture')
  51 
  52 class HandleFocusChange(Task.FocusTask):
  53   '''  
  54   Example Task that reports when some component gains the focus.
  55   '''   
  56   def executeGained(self, por, **kwargs):
  57     # stop current output
  58     self.mayStop()
  59     # don't stop this output with an immediate next event
  60     self.inhibitMayStop()
  61     self.sayInfo(text='GaimPerkBasic handling a focus change')

Reading Incoming Chat Messages

Now you'll fix an accessibility shortcoming in Gaim: the reading of incoming chat messages. The Gaim client can play a sound when a remote buddy sends a chat message, but a user with a visual impairment will likely want to hear the message spoken aloud. The solution to this problem is to create and register a new Task to detect changes in the message history. Such a Task can be defined as follows:

   1 class ReadIncomingMessage(Task.CaretTask):
   2   '''
   3   Reads text inserted into a text control starting with a newline.
   4   Usually, a message received.
   5   '''
   6   def executeInserted(self, por, text, **kwargs):
   7     if text.startswith('\n'):
   8       self.stopNow()
   9       self.sayItem(text=text)

The class derives from CaretTask which means it will monitor caret events when it is registered. The executeInserted method will be called when text is inserted into a widget. The code in the method will stop all output immediately and speak the inserted text whenever that text starts with a newline character.

After defining the event handler, you must register an instance of it so it will be notified about caret events. The following code should be appended to the init method of the GaimPerkBasic class:

   1 self.registerTask(ReadIncomingMessage('caret test', tier=True))

The tier=True flag indicates that the Task will execute in response to caret change events in unfocused controls when the Gaim application is active. That is, the Task will announce incoming messages when Gaim is active, but the focus is on another control (e.g. the message composition text box, or one of the other Gaim windows like "Preferences"). You may also specify the background=True flag to register a Task to handle caret events when the Gaim application is in the background or focus=True to register the Task to handle caret events in the focused control only. Any combination of these flags may be used when registering a Task.

Before continuing, you should disable the previously-defined focus event announcement by removing the line of code that registers the focus Task. Comment out the first line of the Perk init method and the announcement should go away:

  def init(self):
    #self.registerTask(HandleFocusChange('read test focus'))

Save your work and unmute LSR if you silenced it earlier (i.e. press left and right Ctrl togeter). You can refresh Perks loaded for an application by navigating to that application and pressing Alt-CapsLock-K. Alternatively, you can always kill the lsr process using Ctrl-C and run it again later. Confirm that the GaimBasicPerk reloaded properly by making a Gaim window active, pressing Alt-CapsLock-J, and listening for the name of the Perk. If it is missing from the list, then it most likely did not load because of a Python syntax error. Check the console where you started LSR for exceptions, or kill LSR and run it with the following switch to see all error output

lsr -l debug -p developer

Now, when a remote buddy sends you a message, the message will automatically be read aloud. By default, Gaim prepends all messages with the remote buddy's user name and a timestamp, and so that information will be read as well. Try IM'ing yourself to test it.

If all is well, you now have a complete solution to the first problem identified in the introduction. If you would like a deeper understanding of the concepts discussed in this section such as definitions of Tier, Perk, and Task; the types of Tasks available to handle events; how Tasks are registered; and so on, see the LSR workbook.

At this point, you should kill LSR by pressing Ctrl-C in the terminal in which you ran it. We will run it again later after a brief sidetrack into the use of Accerciser.

Exploring Gaim Accessibility

The LSR TaskTools API will shield you from the gritty details of the GNOME accessibility architecture and how individual applications make use of it. In order to query and control a program in non-trivial ways, however, you may need to explore how its user interface is exposed via AT-SPI. One way to learn about a UI is to explore it using a tool called Accerciser. The tools in Accerciser provide a wealth of information about the accessible widgets in an application including their properties, relationships, and potential actions. As a general rule, any information shown by Accerciser is available for use in a LSR Perk.

To get started with Accerciser, follow the installation directions on the GNOME Live! Accerciser page. The instructions for building Accerciser from Subversion are reproduced below:

While Gaim is running (remembering to define GTK_MODULES for Gaim, as discussed in "Generating a Perk Using a Template" above), run Accerciser:

accerciser

You should now see the main window of Accerciser:

Scroll down the accessible tree and start unfolding the Gaim application node to see its widget hierarchy. You will see a tree of Gaim frames and windows along with their respective titles. The following example screenshot shows the Buddy List window and the chat window, with the name of the active conversation as its title. When you click on the Buddy List node in the tree, the outline of the buddy list window should flash on the desktop. If you are ever unsure of what widget is associated with a tree node, just click on it and see what flashes.

Now expand the Buddy List node by clicking the arrow on the left of the name/title. Unfolding the node will show its one child accessible: an object with no name, a role of filler, and ? children. When you expand this filler object, it should show a child of its own, again with no name or description but now with a role of menu bar. If you expand this node, you'll see the Buddy List menu items as its three children (Buddies, Tools, and Help).

A number of plug-in tabs are located to the right of the accessibility tree view in Accerciser. Select the Interface viewer tab to see the AT-SPI interfaces supported by the selected accessible in the tree view. Unfold the various panels in the interface viewer to get more information about the attributes in that interface.

Continue to explore the GUI with Accerciser as it is reported by AT-SPI. Remember that if an application is not fully compliant with GNOME's ATK accessibility guidelines, some of its key GUI components will not be meaningfully described (i.e. missing name and description data) or may be missing from the accessibility hierarchy altogether.

You may find it interesting to see how AT-SPI reports GUI events. In Accerciser, select the Event monitor tab. Unfold the bottom panel and select the events to log (e.g. focus). Now watch as events are delivered by AT-SPI as you navigate the GNOME desktop. These are the events that trigger Task execution in gaim.

Toggling Chat Composition/History Focus

Because Gaim does not define a hotkey to give focus to the chat history text box in a chat window, it is difficult (though not impossible) for a LSR user to review the message history. To improve usability of the message history, we will define a LSR hotkey, CapsLock-T, that switches focus between the history text box and the message composition text box in a Gaim chat window.

First, you must find a way to refer to these two accessible components using Accerciser. Run Accerciser now if you closed it:

accerciser

Start a Gaim conversation with someone. Then select the gaim application in Accerciser and start unfolding it.

The chat window is of interest. Find the tree node labeled with the title of the open chat window. Note that this accessible has a role defined as frame indicating it is a top-level window. Click it and make sure the outline of the window on the desktop flashes. If it does, you have found the correct accessible in the Accesserciser tree and should now unfold it.

You should now see one new child with a role of filler and no other useful information. Unfold this node. It should have two children of its own, with no names or descriptions, but with roles menu bar and page tab list. Again, you can expand the menu bar to see the accessible menu items in the conversation window:

Next you should expand the page tab list node to reveal its page tab. These children represent the panels containing the widgets for each conversation in progress:

Notice the path from the frame to the page tab list. Using zero-indexing, walk from the frame to its first child (the filler) and then on to the second grandchild (the page tab list). This path (0,1) can be used by the TaskTools method getAccFromPath to get a reference to the page tab list. This is a portion of the full path from the selected accessible to the top level application shown in the Accerciser status bar.

Now you need to define a path from the active conversation tab to its message history and composition text boxes. As before, navigate to the page tab list in Accerciser, expand it, and pick any one of its children. (All chat tab subtrees are similarly structured.) Expand the page tab child, expand the next first child (filler), the next first child (split pane), the next first child (the first of two fillers), the next first child (scroll pane), and finally the next first child (text). You have arrived at the chat history area following the path (0,0,0,0,0) rooted at the page tab accessible. Click the text accessible in Accerciser and make sure correct widget flashes in the Gaim GUI.

Use a similar method to find the path from a conversation page tab to its chat message composition text panel. The path should be (0,0,1,1,0,0,0).

You can now use all of these identified paths to reference message history and composition text boxes in Gaim. With these references, your Perk can move focus back and forth between the history and composition areas for the active conversation. The pseudocode to accomplish this feat is the following:

1. Define a new subclass of Task.InputTask called ToggleChatPanel.

2. Define a method called execute that will run when the user gives the input command to change focus.
3. Get a reference to the active frame.
4. Try to get a reference to the page tab list using the path (0,1).
5. Make sure the accessible is a page tab list. If not, a chat window probably isn't active.
6. Get the selected (active) conversation tab.
7. Locate the history and composition text boxes using paths (0,0,0,0,0) and (0,0,1,1,0,0,0) respectively.
8. Check if the composition text area has focus.
   1. If so, give the message history focus.
   2. If not, give the composition box focus.
9. Return True to allow other Perks handle this input event if it is of interest.

This algorithm can be implemented in the GaimPerkBasic by replacing the ReadPerkName class in the template with the code listed below. Of course, the code could be refactored to place the paths in variables, reusable code in separate methods, etc. See the code in the GaimPerk packaged with LSR for an example of refactoring.

   1 class ToggleChatPanel(Task.InputTask):
   2   '''
   3   Toggles focus between chat composition and chat history panels.
   4   '''
   5   def execute(self, **kwargs):
   6     # get the frame
   7     frame = self.getRootAcc()
   8     # get the page tab list
   9     tab_list = self.getAccFromPath(frame, 0,1)
  10     # do a sanity check to make sure we're refering to the correct component
  11     if self.hasAccRole('page tab list', tab_list):
  12       # get active conversation (i.e. the selected tab)
  13       tab = self.getFirstSelected(tab_list)
  14       history = self.getAccFromPath(tab, 0,0,0,0,0)
  15       compose = self.getAccFromPath(tab, 0,0,1,1,0,0,0)
  16       # is composition text box currently focused?
  17       if self.hasAccState('focused', compose):
  18         # if so, change focus to chat history panel
  19         self.setAccFocus(history)
  20         # stop current speech
  21         self.stopNow()
  22         # inhibit the next stop
  23         self.inhibitMayStop()
  24         # use _() to support internationalization of text
  25         self.saySection(text=_('History,'))
  26       else:
  27         # otherwise, change focus to chat compose panel
  28         self.setAccFocus(compose)
  29         self.stopNow()
  30         self.inhibitMayStop()
  31         self.saySection(text=_('Compose,'))

The code that announces whether the message history or composition text area is now focused deserves some explanation. After switching the focus programmatically, the code calls the stopNow method to cease any current output. This method is called to prevent the next announcement from being queued. Next, the code calls inhibitMayStop. Without this call, the very next event (i.e. the focus event) would stop the announcement made by this Task. Finally, the code says which panel received focus using a localized string if it is available. The call to sayTitle respects the user settings for how he or she would like names and titles to be announced.

All that remains is binding the InputTask you wrote above to a input gesture that will trigger it. The Task can be set to execute on the release of CapsLock-T on a standard keyboard by changing the command registration code in the init method of the GaimPerkBasic. The init method should now read:

   1   def init(self):
   2     #self.registerEventTask(HandleFocusChange(self))
   3     self.registerTask(ReadIncomingMessage('caret test', tier=True))
   4     self.registerTask(ToggleChatPanel('chat panel toggle'))
   5     # get the Keyboard device and set CapsLock as a modifier
   6     kbd = self.getInputDevice('Keyboard')
   7     self.addInputModifiers(kbd, kbd.AEK_CAPS_LOCK)
   8     # register CapsLock-S as the key to trigger the input task
   9     self.registerCommand(kbd, 'chat panel toggle', False, 
  10                          [kbd.AEK_CAPS_LOCK, kbd.AEK_T])

The registerTask() call associates an instance of the ToggleChatPanel class with a name unique across all Perks loaded for the Gaim application. ³ Here, GaimPerkBasic is prefixed to help ensure uniqueness. The call to registerCommand sets the given sequence of input gestures to trigger a Task with the given name. The keyboard device supports one step sequences only (i.e. one key combo only, not a sequence of key combos). In this case, CapsLock must be pressed simultaneously with T to trigger the Task. ⁴

Save your work and run lsr again with the developer profile. Open or switch to a Gaim 1.5 chat conversation. Press CapsLock-T to toggle the focus between composition and history. While in the chat history, use the arrow keys to read the text of past chat messages. The chat history is now much more conveniently accessible to the user.

Congratulations! You have now completed your first basic Perk.

Conclusion

The code developed in this tutorial is a good start, but it will fail in certain situations. For instance, it will not work with Gaim 2.0 and the hotkey behavior is undefined when the chat window is not active. A polished GaimPerk should do better and provide many more accessibility and usability improvements. If you look at the GaimPerk included with LSR, you will see it has a great deal of extra functionalilty. Although we will not explain its many functions here, you can use what you've learned so far to grasp its overall structure:

1. Perk class: Registers new event handlers and key bindings in init(). Provides helper methods that can be called from any Task in the module.
2. Task classes: Handle focus, view, caret, table, and property events to report buddy arrival and departure, conversation status change, etc. Define new queries to report the status of all converations.

The advanced Gaim Perk tutorial continues where this tutorial ends by discussing these additional features.

Now that you've completed this tutorial, you have a solid foundation for getting started on your own Perks.

Troubleshooting

Refer to the questions and answers in this section if you are having trouble completing this tutorial.

GaimPerkBasic doesn't do anything new.

If you don't hear the GaimPerkBasic handling focus event message each time you give a Gaim window focus, you may be using a speech synthesis driver that does not permit audio mixing. Try turning off Gaim application sounds to prevent a conflict. Navigate to Buddy List, Preferences, Interface, Sounds, and uncheck all Sound Options checkboxes.

If that doesn't help, there may be a problem with your installation of LSR or its dependencies. Be certain you have downloaded, built, and installed LSR properly and can run it using lsr without any command line arguments. See the getting started guide for assistance.

How do I unregister my new Perk?

Type the following to disassociate your new Perk from the developer profile:

lsr -d GaimPerkBasic -p developer

It can later be associated with the developer profile (or another profile) by typing the following:

lsr -a GaimPerkBasic -p developer

To restore the GaimPerk that ships with LSR, do the following:

lsr -a GaimPerk -p developer

Footnotes