Accessibility, Wayland, Flatpak

This page will be used to collect the status of accessibility in GNOME in general, with a particular focus on problems with Wayland and Flatpak.

Components of Accessibility

Accessibility is an umbrella term that covers a number of different functionalities worth discussing separately.

• Keyboard accessibility: AccessX features (Stickykeys, Slowkeys, Bouncekeys), Key repeat
• On-screen keyboard
• Mouse accessibility: Mousekeys, Click assist, Double-click delay, Cursor locator
• Audible feedback: Soundkeys, input event sounds
• Visible feedback: Visible bell, Modifier status display, Caret blinking, Cursor size, Zoom, Large text, High-contrast, other color tweaks
• Screen reader

A bit outside of this, accessibility infrastructure is also used for automated UI testing. Important aspects for this:

• Determining global positions and stacking order of windows and widgets
• Inject events into applications

Keyboard accessibility

Status: Historically, AccessX features and key repeat have been implemented in the X server (as part of XKB). Under Wayland, key repeat is currently implemented client-side in GTK+, and other AccessX features are not implemented.

Plans: A client-side implementation of the other AccessX features is doable (Matthias did a prototype once), but problematic. This should really be implemented in the input event routing layer of the compositor. A slight complication arises for X clients, where we need to disable the XKB implementation in Xwayland, or somehow make it sync state with the compositor.

Update: Oliver Fourdan points out that the AccessX implementation in Xwayland can be disabled by a simple commandline switch, -accessx, which mutter can use when launching Xwayland. Patches: 788564

On-screen keyboard

Status: Our compositor has an implementation of an on-screen keyboard, but it is not up to par with other platforms. The OSK needs some information that is provided by at-spi for focus tracking. This relies on global coordinates and does not work properly under Wayland in its current form.

Plans: An OSK is also needed for touch, and to implement expected features there such as (tailored completion and suggestions, emoji, etc), we need something like the Wayland text protocol, which has not landed upstream yet. Some new protocol will be needed for focus tracking.

Update: Wayland text protocol is landing in 3.28, in both mutter and GTK. A first round of OSK improvements are landing too.

Mouse Accessibility

Status: Double-click is implemented on the client-side, both under X and Wayland, and the delay is customizable via a setting. Mousekeys have traditionally been implemented in the X server (as part of XKB), this is currently missing under Wayland. Click-assist has traditionally been provided by the mousetweaks client. This approach does not work under Wayland. Cursor location has been implemented by a client that is part of gnome-settings-daemon under X. This also does not work under Wayland.

Plans: Mousekeys, click-assist and cursor location can all be implemented inside the compositor. issue, issue

Audible feedback

Status: Historically, Soundkeys have been implemented in the X server (XKB). Other input event sounds are implemented client-side in GTK+. Input event sounds work fine under Wayland.

Plans: We may just make input event sounds cover anything that might be missing from Soundkeys, and get away without compositor-side Soundkeys support.

Visible feedback

Status: The visible bell has traditionally been implemented in the window manager, and it is now implemented in the compositor. It works under Wayland, with the limitation that flashing just the titlebar does not work with client-side decorations, and is reinterpreted as "flash window". Modifier status display has been realized as an applet long ago and was lost with the move to gnome-shell. Caret blinking an cursor size are implemented client-side and work equally well under X and Wayland. Large text and high-contrast is realized client-side as theme and work both under X and Wayland. Zoom and color tweaks are implemented in the compositor and work equally well under X and Wayland.

Plans: The wording of the visible bell setting should be adjusted to say "Flash window" and not talk specifically about the titlebar. Modifier status display should be implemented in the compositor, or as a shell extension. 787132

Screen reader

Status: Orca is the one component of accessibility that makes extensive use of the at-spi exported object model. For the most part, this works as well under Wayland as it does under X, with some extra complications:

• Wayland clients can't use a root window property to locate the a11y bus
• Parts of the atk object model that are making assumptions about global coordinates can not be implemented under Wayland

The bigger problem with at-spi is that we can not give sandboxed applications to the a11y bus. There bus protocol is bidirectional, and there is not access control, and all clients are visible to each other on this bus. It is trivial to kill the desktop session by sending some bad requests to gnome-shell via the a11y bus, which is not something that we want to let sandboxed apps do.

Also, just like input methods, the a11y bus sends all input to the bus, and allows clients to inject input events into each other.

Plans: In the short term, the worst problems can be alleviated by changing the a11y bus from a bus to a peer-to-peer connection between the sandboxed app and an a11y service, but even this may require significant changes to the way the at-spi protocol works. In the longer-term, the approach of exporting a full object model of the application on the bus should be reconsidered altogether. In particular, an object model that was frozen in the late 90s and does not match todays user iterface paradigms and requires app-specific workarounds in orca anyway.

Update: We have the short-term fix in place now, so orca will work with sandboxed apps.

Testing

Status: UI tests are majorly affected by the lack of global positions and the inability to inject input events.

Plans: gnome-shell / mutter can expose a D-Bus api for this, which would be separate from the a11y interfaces. The D-Bus api will be privileged and not available to normal applications. issue

References

• Accessibility/Wayland