Python Powered Accessibility

This article introduces key accessibility concepts and programs on GNOME, along with providing a tutorial exploring accessibility with accerciser. While it is aimed at developers, the earlier sections are also of interest to less technical users as well. It is hoped this will get you started with using these tools to enable your project to reach more users and to have improved quality.

Python is playing a vital role in improving the accessibility of desktop programs and the web. Infrastructure and applications created by the GNOME Desktop accessibility community are providing Python-powered accessibility, enabling people with disabilities to more easily operate programs and access information. In addition, accessibility features provide an effective mechanism for automating and testing GUIs. Having been adopted by the Open Accessibility Group, these techniques are also appearing on other platforms, including KDE and Windows.

• Disabilities and Assistive Technologies Accessibility infrastructures Accessibility on GNOME pyatspi features Using Accerciser to explore accessibility Code example Testing Application accessibility Other Open Source Assistive Technologies Summary Getting involved in accessibility Resources

1. Disabilities and Assistive Technologies

Ensuring accessibility can be viewed as a form of good usability design. Everyone has unique requirements and preferences when using computers, and while much user configuration is provided by modern operating systems and programs, accessibility requires catering to more extreme user variation. Disabilities occur across a wide spectrum, from a mild difficulty with performing a task to severe inability to interact. Milder forms of disabilities can be met with suitable configuration options such as themes, but more severe forms usually require specialized Assistive Technology (AT) that adapts the computer to the user's abilities.

To understand accessibility and why it is important, it is essential to understand the audience that these tools and techniques are trying to reach. An understanding of the disabilities people may have will help us to discover how programs can be adapted to make them more accessible.

Disabilities can be present from birth, or appear as part of the natural aging process, and are sometimes the result of accident. While disabilities can appear in complex combinations, we can at least create useful categories to help us provide appropriate accessibility features:

• Visual Motor and speech Hearing Cognitive

Visual impairments are common, and we can all expect some degradation as we age. Mild difficulties can be compensated for by using large displays or fonts, high contrast themes or special color schemes and sounds. More pronounced acuity problems require magnification or zoom facilities. Users with little or no sight need the services of a screen reader AT that provides output in speech or Braille (raised dots that can be felt) and has mechanisms to allow navigation of the screen contents, plus notification of changes. Braille is the only form of accessibility suitable for people who are both deaf and blind. Audible equivalents for visual media can be provided by adding descriptive text for synthetic speech or recorded narration.

Motor impairments manifest themselves as an inability to reliably control movement, whether from lack of motion, excessive or random movements, or shaking. Sometimes, there are differing abilities for gross and fine movements. Mild motor impairments can be compensated for with settings for the mouse and keyboard.

Figure 2.

These include options to ignore false activations, provide automatic activations, or alternatives such as using the keyboard to simulate a mouse. Specialized mice and keyboards can also be used, and there are alternative input devices such as joysticks and eye gaze trackers.

Some users can only operate simple mechanical switches, perhaps with particular parts of the body like their head or perhaps with breath control. These users benefit from intermediary UIs, sometimes called "On Screen Keyboards" (OSKs), that allow control of applications and text input from switches or pointers by presenting a set of cells that can be more easily selected using various "scanning" modes to move between cells. Speech recognition is useful for users who have speech, or else other vocal gestures may be usable.

People who find speech difficult can use synthetic speech. One common use of this is in Alternative and Augmentative Communication, AAC, which provides speech built from phrases constructed by selecting from word sets or symbols, often with a switch.

Hearing impairments include various degrees of deafness which can be supported by providing visual cues for sound notifications and events. Videos can be made accessible by adding text captions or subtitles.

Cognitive impairments include reading disorders and difficulty with comprehension or composition. Reading support can be provided by automatic speech and highlighting of text. Language tools such as spelling and grammar checkers or a thesaurus are often useful. Symbols or pictograms help communication with sentence construction and automatic conversion between text and symbols along with speech support. Careful design of user interfaces and choice of language are important to avoid cognitive overload. It is fair to say that people with cognitive impairments are the least well served by current accessibility provision.

2. Accessibility infrastructures

Most interfaces are designed using a narrow range of human-computer interface devices: namely screen, keyboard and mouse. Developers are used to working with these devices as they are explicitly supported by operating systems and UI frameworks. Thus, users are well catered to if they use these devices. But what of people who cannot use these devices such as those with visual or motor impairments? Or what of environments or platforms that make them unsuitable such as mobile applications and devices?

Whilst some input devices have hardware or drivers that mimic a mouse or keyboard, others do not and so need specific support in applications. For example games often support joysticks and game pads, with frameworks like pygame (based on SDL) providing common platform services. However it is unrealistic to expect developers to add specific support for a wide range of alternative devices. The use of synthetic speech and Braille output, for example, is complicated by the need to manage the mismatch between their serial natures and the 2 dimensional, random access of a pointer and screen. Assistive Technologies provide this specialist input and output operation. Acting as an intermediary between the application and the user of alternative access devices, they monitor and control the programs being run. The Assistive Technologies connect to the applications through a dedicated accessibility infrastructure via dedicated APIs.

While application developers need to develop additional code to to provide support for this infrastructure, once done, the application will be accessible from any Assistive Technology and so can reach more users. The accessibility infrastructure provides interprocess communication between the application and AT in a loosely client-server relationship (the application playing the role of server, in this case). Software APIs at both ends simplify and standardize access to the raw RPC or message passing protocols, and wrappers like pyatspi (which we'll discuss further, shortly) giving 'pythonic' access.

In the traditional architecture, applications expose rich information about the state of the GUI through a set of predefined properties and methods. This is done on a per application and widget basis, thereby maintaining an object oriented architecture. UI elements appear as a tree of accessibility objects which usually closely mirrors the UI structure, though variation can occur. Events are used to signal application state changes. Standard widgets have default implementations provided by accessibility services, though fine tuning for a particular application is possible. Custom widgets require that accessibility is specifically added by developers, though libraries can ease the task. Applications without a GUI can also expose their state or text-based UI to make them accessible.

Assistive technologies navigate and interrogate the tree of accessibility objects exposed by the applications and subscribe to events in order to respond to application state changes. Actions can be invoked to alter the UI state, but this is restricted to invoking application behavior in ways similar to when the user drives the UI, thus ensuring that the application does not lose track of its own UI state.

A by-product of this architecture is that it provides an ideal way to test or automate (or automate testing of) GUI applications, something that is otherwise difficult to achieve. The slight indirection between GUI and accessibility API is nevertheless small enough for most testing and automation purposes.

Naturally, having good accessibility is only part of good user interface design, but the process of adding it can help improve the overall quality of a project by helping to highlight problem areas. Stock widgets have an advantage over custom behaviours as they are well tested and accessibility is already provided. Custom widgets need careful design for both usability and accessibility.

3. Accessibility on GNOME

The infrastructure that connects Assistive Technology and applications is called Assistive Technology Service Provider Interface or AT-SPI. Although it is designed to be an integral part of the desktop, it is currently off by default (though this should change soon) and must be enabled by the user with the 'enable assistive technologies' option.

Figure 7.

In addition to providing this 'glue', AT-SPI also features a central registry process (or daemon) that routes events and provides ATs with various discovery mechanisms. The following shows how this architecture works on GNOME.

Figure 6.

Applications use the GNOME Accessibility Toolkit (ATK) library to provide accessibility and isolate them from the raw AT-SPI. For applications that use the GTK+ GUI toolkit, the GAIL module provides default accessibility functionality for stock widgets. Thus, there is often little for developers to do in order to make applications accessible, and many accessories bundled with GNOME are fully accessible. Applications with custom widgets (or without any widgets) can use the ATK library and papi is a Python wrapper to the ATK library.

Several non-GNOME applications also support AT-SPI. OpenOffice.org for example uses its own architecture to support AT-SPI (and incidentally its native Open Document Format has features to ensure accessibility of documents). Mozilla Firefox 3.0 makes both its UI and web documents accessible via AT-SPI. Support for Web Accessiblity Initiative/Accessible Rich Internet Applications (WAI ARIA) techniques gives web authors fine control over how ATs see pages, something that is critical for Web 2.0 and AJAX applications (and available on Windows since Firefox 1.5).

Assistive Technologies developers find pyatspi greatly simplifies access to AT-SPI and handles all inter-process proxying and serialization (marshaling), as ATK does for applications. It is a very thin wrapper, using clever runtime mixin code to create a more pythonic interface to the raw objects instantiated with "pyorbit"'s ability to process the CORBA interface descriptions (IDL). Python's introspection facilities promote useful interactive exploration. Though it's not shown on the above diagram, pyatspi sits between the Platform and AT layers.

AT-SPI, currently runs on ORBIT, the Linux CORBA implementation, but work is ongoing to port it to DBUS (which also makes it more attractive to other platforms). CORBA is component oriented, and AT-SPI is, likewise, organised as components (or objects) that expose various interfaces.

4. pyatspi features

pyatspi wraps AT-SPI in a pythonic shell providing access using common Python idioms. The key concepts are accessible objects with their role, states and relations, the registry and events.

The AT-SPI model of an application is a tree where each node is an accessible object representing a UI element or sub-element. The root represents the application and an object node can have 0 or more children. In addition an object may also have relations that specify 'links' to other objects that are not direct children. Each object has a role that defines what it does, what states it can have and what operations can be performed on it through various methods. States represent object properties that may change over time such as STATE_VISIBLE or STATE_CHECKED. In addition attributes may set as application-specific name-value pairs. Related methods and properties are grouped together into interfaces and an object only implements those interfaces that make sense for it's role. For example an object that represents a piece of text will have a role of ROLE_TEXT and implements the Text interface among others. The Accessible interface is implemented by all objects and provides basic accessibility information such as the role, state and children. The Component interface describes UI component properties such as position on screen. The Action interface lets you programmatically invoke actions such as 'button' role object allowing the simulation of user operation through a 'click' action.

When using pyatspi you obtain an object from a traversal or search of the hierarchy or from an event. This python object has attributes for the methods and properties of the Accessible interface. Other interfaces are accessed by using the appropriate 'queryXXX' method (e.g queryComponent) which returns a new object, again with appropriate attributes. Objects are collections of child objects and can indexed or iterated to access the children (i.e getitem and iter are implemented) .

Events are generated in response to the application changing the user interface in some way, usually in response to a user action. There is a wide range of events covering actions such as window creation and activation, focus change, object state changes and changes to the tree structure. Events are grouped into those relating to Windows, Focus, Objects and Documents and there are facilities to subscribe to individual events or an entire group.

The registry is a Singleton object that provides access to open applications on the desktop and thus each application's tree of objects. It provides services for subscribing to specific events and for generating synthetic mouse and key events.

pyatspi also provides utility functions for finding objects that match a condition and a class for handling StateSets (which are server-side objects).

5. Using Accerciser to explore accessibility

Accerciser provides flexible tools for exploring application accessibility on GNOME. It is a pygtk application with an open, pluggable architecture (only the the main tree view is not a plugin). The integral IPython shell can be used to explore interactively, and provides an ideal platform for some experiments in accessibility. It uses pyatspi, making it a good reference application, and is useful for learning how to perform a particular task.

The following is offered as a whistle-stop tour of the basics. The accerciser web page has more details and webcasts are available on the website.

In order to follow this tutorial you need a recent GNOME (2.20 includes accerciser). Ubuntu Gutsy LiveCD is one possibility though you will need to install the accerciser package as it is labeled a developer tool. Once you've confirmed that you have the required bits installed, perform the following steps:

• Enable accessibility: By default AT-SPI infrastructure is off so you need to enable it (though this is set to change in future releases). In Ubuntu, choose system -> preferences -> universal accesses -> accessibility options -> enable assistive Technologies.

  Launch Gedit ( e.g. applications -> accessories -> text editor). This is our sample accessible application. Run accerciser

The pane on the left is a tree that shows all of the accessible applications that are running. Accerciser uses the Registry object to get all accessible applications, and then as you expand the nodes it descends through the node tree, inquiring about basic info: name, role, number of children. The selected node is the context for the plugins on the right which show information for each node. A useful tip is that the ctrl+alt+? key combination will cause accerciser to show information about the widget in another applicaition that is under the pointer.

• select gedit in the tree.

To the right you will see a number of tabs depending on which plugins are installed. "API Browser" details the interfaces supported by the current node (select the interface in the combo box) and lists the available methods and properties for the selected. This is implemented by showing the Python object attribute names and data attribute values. "Interface Viewer" provides detailed inspection of each interface supported by the node. "Event Monitor" provides monitoring of selected events and "Script Recorder" provides record and playback; neither are used in the tutorial.

Figure 8.

• look at the "interface viewer" tab

You'll see "Accessible" and "application" interfaces are implemented, and if you expand them in "interface Viewer" you'll see key information listed.

Below you'll see the ipython interactive interpreter, where you can use the variable named 'acc' to reference the currently selected accessible node and the tab key gives command line completion.

• type "acc.childCount" type "acc.getRole()"

These will return the values you see in the tabs and are what accerciser is doing to display those items. Note that "Accessible" is the default selected interface in a node (and so acc) so these work without the need to query the interface first.

Figure 9.

Python introspection features, including dir() and help(), are useful:

• type "dir(acc)"

You see a lot of private attributes prefixed with a '_' plus attributes you can use such as getRole. This information is also shown in the 'Interface Browser' tab.

• expand gedit in the tree and select the frame directly under it.

Note that a flashing dashed rectangle appeared briefly around part of Gedit showing the screen location of the selected node. The selected frame implements the component interface which describes the nodes screen location, as you can see in the "Interface Viewer" tab. To get at methods or properties on another interface we call the correct query function which returns a new object with attributes for them.

• type "ic=acc.queryComponent()" type "ic" type "acc"

You'll see that "ic" is a component whereas "acc" is an accessible. These objects represent the interfaces defined in the IDL for AT-SPI.

Figure 10.

• type "ic.getSize()"

and you'll get the size of the frame component on the screen as show in the "Interface Viewer" under Component.

• navigate to the New menu item in the node tree.

Note the "0 0 0 0 1" shown in the status bar. This represents the path to the node in the tree with each digit being the child index at that level. The Path is often useful in specifying how to get to a specific node.

• in "Interface Viewer expand the action interface select "Click" and then click the "Perform Action" button.

Figure 1.

A new empty tab should appear in gedit as the //new// action has been invoked, just as if the user had used that menu item.

• spend some time exploring accerciser's tabs and using the iPython console. The "Event Monitor" and "Script Recorder" are useful for exploring the dynamics of an application's GUI and running tests.

6. Code example

Listing 1 demonstrates typical uses of the features of pyatspi. When run it prints information about a few nodes and then monitors window activations events until a keyboard interrupt occurs (Ctrl+C).

Toggle line numbers

• 1 import pyatspi 2 3 # get the Registry singleton 4 reg = pyatspi.Registry() 5 6 # and desktop 7 desktop = reg.getDesktop(0) 8 9 # show the number of and name of each running app

• 10 print len(desktop) 11 apps = [app.name for app in desktop if app is not None] 12 print apps 13 14 #find gnome-panel 15 gpanel = pyatspi.findDescendant(desktop, 16 lambda x: x.name == 'gnome-panel', 17 breadth_first=True) 18 print gpanel.name, gpanel.getRole() 19 20 #child nodes 21 print gpanel.childCount 22 print ["name %s, position:%s" % (panel.name, panel.queryComponent().getPosition(0)) 23 for panel in gpanel] 24 25 26 # events 27 def eventCallback(event): 28 print event 29 30 what = ['window:activate', 'window:deactivate'] 31 print 'Monitoring window events so select some applications on the desktop' 32 reg.registerEventListener(eventCallback, *what) 33 34 try: 35 pyatspi.Registry.start()

  36 except KeyboardInterrupt: 37 pass 38 39 #cleanup 40 pyatspi.Registry.stop() 41 reg.deregisterEventListener(eventCallback, *what) 42 43 print 'Done'

7. Testing Application accessibility

Ad hoc exploration with accerciser is one way to ensure the quality of your project and complements formal testing with one of the frameworks based on pyatspi. Accerciser also has a validation plugin that can produce an interactive report listing potential a11y issues.

Applications should also be tested with various desktop accessibility options selected and accessed with the different Assistive Technologies. However there is no substitute for testing by real users of Assistive Technologies who will often exercise programs in ways that may not be obvious.

8. Other Open Source Assistive Technologies

In addition to the desktop accessibility options such as the mouse and keyboard features and high-contrast themes, there are a number of key Open Source Assistive Technology projects.

Orca enables people with little or no vision to successfully interact with applications, including programs with a graphical user interfaces. It provides speech and Braille output along with various interactive "review" mechanisms that give effective navigation. Magnification services are also controllable with Orca and allow enlargement of areas of the screen. Orca is written in Python and also provides for customisation of the behaviour for individual applications and user preferences through scripts.

Figure 3.

GOK provides access for user with physical disabilities by supporting switch input for applications control and text input. In addition to basic OSK features and various scanning modes it has more advanced features like dynamically creating layouts from application UIs. It is implemented in C but there are discussions about rewriting it in Python (perhaps collaborating with a Python ATs like OnBoard or Jambu).

Figure 4.

OnBoard is an OSK that comes from the Ubuntu Linux distribution. It has some scanning and switch features and is written in Python.

Dasher provides a highly alternative and unique text input method which is controllable with 2 input states (e.g. up, down) and vision. Users "steer" through a dynamic field of letters in order to spell out text. It uses word prediction to ease the process. Many languages are supported and a version of dasher supports Bliss symbols. Dasher is an independent cross platform AT project and is written in C.

Figure 5.

On Windows the NVDA screen reader is becoming popular with users who have visual impairments and who value an Open Source solution. Written in Python NVDA uses IAccessible2 for accessibility access and takes advantage of Python packages such as win32all and comtypes for access to the platform APIs.

AutoHotKey is a powerful scripting environment for Windows which controls applications using simulated mouse and keyboard events. It allows automation of many tasks and improved accessibility. It is a C++ application.

Also on Windows the Special Access to Windows, SAW, provides OSK and other related features. SAW is developed in C++.

FireVox is screen reader extension for the Firefox Web browser that provides screen reading services for the web. CLiCk, Speak is a sibling program that provides reading support. Both use the services of a Java Script library that is available for other developments. As a Firefox add on, FireVox is available on the 3 main platforms.

Jambu aims to improve access with alternative input devices. Currently providing SVG-based OSK creation, work is in progress on providing 'in- application' navigation and control. Jambu is being developed in Python by the author.

More programs can be found on the OATSoft.org repository website.

9. Summary

So that's the basics of accessibility. We've seen the architecture and how Python is playing an important part with wrappers like pyatspi, tools like accerciser and assistive technologies like Orca and OnBoard. We've also seen how easy it is to explore application accessibility using GNOME's accerciser tool. While we've concentrated on Linux and Solaris, Windows accessibility is very similar, especially when using IAccessible2, thanks to the work of the Open Accessibility Group.

10. Getting involved in accessibility

Accessibility offers a good introduction to contributing to Open Source projects. If you are interested in getting involved then you can approach one of the project communities listed in resources for more information. The communities are friendly with a focus on user needs and will help with questions, offer advice and welcome contributions. Most projects run mailing lists and irc channels where you can lurk to get a feel for what is going on. Mozilla, GNOME and Orca are particularly active at the moment. The annual GNOME and Mozilla Accessibility summits run together in November and are a great way to meet the people working on open accessibility.

Other sources of ideas include the Open Accessibility Group's resources, mail lists and teleconferences. Linux distributions like Ubuntu have accessibility projects as do applications like OpenOffice.org. OATSoft is resource and community for finding Open Source Assistive Technology. It provides an online repository of accessibility software plus a developer area and aims to help developers and users hook up.

Key areas for involvement include application accessibility, desktop accessibility or specific Assistive Technology projects. There's plenty of scope for learning and applying all sorts of skills including documenting, testing, graphics design, coding, community work and marketing. User's are particularly welcome.

Accessibility can make excellent student projects and several important features have been developed this way. Google's Summer of Code can help with funding and Mozilla Accessibility Grants enable people to work on projects when they would not otherwise be able to (Jambu started this way).

11. Resources

The Open Document Format Accessibility Guidelines have a great introduction to various disabilities, suitable AT and how to provide effective software features. http://docs.oasis-open.org/office/office-accessibility/v1.0/

Eitan Issacson has written a great article on Making your Application Accessible with accerciser which includes general good design principles for application writers.

The GNOME Accessibility Project has some documents that are 'must-read' for making applications accessible. GNOME Accessibility Project: http://live.GNOME.org/GAP

There's a good presentation on the AT-SPI architecture at http://www.GNOME.org/~billh/ArchitecturalOverview.odp. Be sure to read the notes tab for each slide.

AT-SPI:

http://www.GNOME.org/~billh/at-spi-idl/html/index.html and GAP above

pyatspi:

http://live.GNOME.org/GAP/PythonATSPI

Accerciser:

http://live.GNOME.org/Accerciser

papi:

http://ocemp.sourceforge.net/papi.html

Open Accessibility Group:

http://www.linux-foundation.org/en/Accessibility

Mozilla Firefox accessibility:

http://www.mozilla.org/access/

OpenOffice.org accessibility:

http://ui.openoffice.org/accessibility/

Orca:

http://live.GNOME.org/Orca

GOK:

http://www.gok.ca/