Orca Audio Support

On this page you will find details about orca audio support: playing sounds on specific events and using that to provide users with a better experience. For now this is just a draft

Introduction

To provide feedback for what is going on the desktop orca Relies on speech, braille and magnification. To complement these means, it can be used pure audio interfaces (what some call sound icons) to provide other ways of communication. This page describes possible ideas to use these kind of interfaces and all the details and developments.

Use Cases and Ideas

These are just initial use cases and ideas to integrate audio support in orca. If you have another idea please put here! Some things here were borrowed from other accessibility products like nVDA, JAWS, etc. Orca does not pretend to be a clone of anything so we must not implement the same things same way unless that is the right way.

Beep on Progress bar Updates

  • Description: on progress bar updates, inform the user of the relative percentage using beeps of various frequencies.
  • Details: One great idea implemented in the nVDA screen reader is the beep on progress bar updates functionality. When the current application as a progress bar control and that control changes the value NVDA beeps informing the user of the completed percentage on the progress bar. Higher the beep frequency, higher the progress bar value. This is really useful when copying or downloading files, installing packages, etc.

Provide Mouse Localization using Sound

  • Description: when the user moves the mouse, provide mouse localization in screen using sound.
  • Details: The current mouse review feature in orca provides a way to inform the user of the object under the mouse pointer. Using sound, it is also possible to inform the location of the object on screen using pan (stereo) and pitch. Pan provides the horizontal coordinate and pitch the vertical one. For example, if the user moves the mouse to the screen top left corner, orca plays a high pitch beep on the left speaker, on the bottom right corner it plays a low frequency beep on the right speaker. This feature is also present on the NVDA screen reader.
  • Notes: Because this feature is useful outside Orca a standalone application can be created for this porpose. Controling it from Orca could be a good enhancement but for all flexibility a separate process is the best way.

Audio cues when Crossing line boundaries

  • Description: When in flat review, when navigating a text area, and in "say all", Orca should provide an option to present an audio cue to the user when they cross line boundaries.
  • Details: I don't recall if I saw this feature on emacspeak but it seems really interesting from an usability standpoint

Report miss-spelled Words using Sound

  • Description: use a sound icon to report miss-spelled words instead of using speech
  • Details: Currently in firefox for example, miss-spelled words are reported using speech (the synth inform the user "miss-spelled word"). Using a sound icon the user can be informed of the mistake without interrupting the normal text flow. This must be synchronized with the speech flow.

Sound when entering and exiting interactive controls in HTML Documents

  • Description: When entering or exiting interactive controls while navigating HTML, the user could be informed using a sound icon so he can now if it is in navigation mode or in interactive mode (can use navigation keys or not).
  • Details: This is a feature present on the NVDA screen reader and, I think, in VoiceOver, we can use those for a starting point.

Sound on page loaded in Firefox

  • description: When a page finishes loading orca anounces it via speech. We can use a sound icon for this porpose
  • Details: I think this overlaps a bit with beep on progress bar updates but can be a good idea

Spatial location of controls when Navigating the Desktop

  • Description: When the user navigates the desktop and enters some control, before speaking the control information we can use a sound icon informing the spatial location of the control
  • Details: This is present in VoiceOver from apple. We can use sterio position and pitch to provide information but have to check further how apple implemented it.

Use sound to inform the value of range based controls

  • Description: When interacting with range based controls like sliders, scroll bars, etc. We could use sound cues to inform the user of the relative value
  • Details: We could use pan or pitch like most cases above.
  • Notes: Does the at-spi provide ways to find minimum and maximum values for these kind of controls?

Sound on bodytyping in pidgin and empathy

  • Description: When a body is typing in pdigin (or some other IM client) inform this fact using sound instead of saying "body is typing"
  • Details: This is a feature heavily used by the JAWS screen reader users and seems good from an usability perspective.
  • Nots: With the integration of telepathy in Gnome this application must be also considerated. On the Pidgin side I think a pidgin plugin can be a better option then scripting just Orca, maby the same for Empathy.

Audio support configuration

  • Description: The user should be able to configure audio support in orca
  • Details: some users don't like sounds going around, some other pref ere them to play lower, some must turn the off on some occasions. There must be a way to turn sounds off, control there overall volume, possibly change the sound card where they are played, Etc. There must be also a good default configuration for most users, for example use gnome settings for Te sound card output.

Configuring audio Support

TODO

implementation Developments

Developments on the implementation are also being discussed and posted on bugzilla:

Gstreamer

The audio support is implemented with the Gstreamer Multimedia Framework. Due to python gstreamer integration problems in some platforms (OpenSolaris for example) the current implementation uses an os.system call to run the gst-launch tool to play a sound. When things are solved with this we will use the python gstreamer bindings.

Documentation

Here it is some gstreamer documentation I found useful:

Audio Files and Formats

The current implementation only allows playing of wav files specified by absolut path (or maby relative to $PWD I guess). We must define what formats should be supported (wav, vorbis, etc.) and a standard location for audio files. On other subject, if the interface will use sounds and will depend on sound files, we may have to distribute some sound files with the source, it must be defined what and how to distribute.


The information on this page and the other Orca-related pages on this site are distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.


CategoryAccessibility

Orca Audio Support

On this page you will find details about orca audio support: playing sounds on specific events and using that to provide users with a better experience. For now this is just a draft

Introduction

TODO

Use Cases and Ideas

These are just initial use cases and ideas to use audio support in orca. If you have another idea please put here! Some things here were borrowed from other accessibility products like nVDA, JAWS, etc. Orca does not pretend to be a clone of anything so we must not implement the same things same way unless that is the right way.

Beep on Progress bar Updates

  • Description: on progress bar updates, inform the user of the relative percentage using beeps of various frequencies.
  • Details: One great idea implemented in the nVDA screen reader is the beep on progress bar updates functionality. When the current application as a progress bar control and that control changes the value NVDA beeps informing the user of the completed percentage on the progress bar. Higher the beep frequency, higher the progress bar value. This is really useful when copying or downloading files, installing packages, etc.

Provide Mouse Localization using Sound

  • Description: when the user moves the mouse, provide mouse localization in screen using sound.
  • Details: The current mouse review feature in orca provides a way to inform the user of the object under the mouse pointer. Using sound, it is also possible to inform the location of the object on screen using pan (stereo) and pitch, pan provides the horizontal coordinate and pitch the vertical one. For example, if the user moves the mouse to the screen top left corner, orca plays a high pitch beep on the left speaker, on the bottom right corner it plays a low frequency beep on the right speaker. This feature is also present on the NVDA screen reader.

Audio cues when Crossing line boundaries

  • Description: When in flat review, when navigating a text area, and in "say all", Orca should provide an option to present an audio cue to the user when they cross line boundaries.
  • Details: I don't recall if I saw this feature on emacspeak but it seems really interesting from an usability standpoint

Report miss-spelled Words using Sound

  • Description: use a sound icon to report miss-spelled words instead of using speech
  • Details: Currently in firefox for example, miss-spelled words are reported using speech (the synth inform the user "miss-spelled word"). Using a sound icon the user can be informed of the mistake without interrupting the normal text flow. This must be synchronized with the speech flow.

Sound when entering and exiting interactive controls in HTML Documents

  • Description: When entering or exiting interactive controls while navigating HTML, the user could be informed using a sound icon so he can now if it is in navigation mode or in interactive mode (can use navigation keys or not).
  • Details: This is a feature present on the NVDA screen reader and, I think, in VoiceOver, we can use those for a starting point.

Sound on page loaded in Firefox

  • description: When a page finishes loading orca anounces it via speech. We can use a sound icon for this porpose
  • Details: I think this overlaps a bit with beep on progress bar updates but can be a good idea

Spatial location of controls when Navigating the Desktop

  • Description: When the user navigates the desktop and enters some control, before speaking the control information we can use a sound icon informing the spatial location of the control
  • Details: This is present in VoiceOver from apple. We can use sterio position and pitch to provide information but have to check further how apple implemented it.

Use sound to inform the value of range based controls

  • Description: When interacting with range based controls like sliders, scroll bars, etc. We could use sound cues to inform the user of the relative value
  • Details: We could use pan or pitch like most cases above.
  • Notes: Does the at-spi provide ways to find minimum and maximum values for these kind of controls?

Sound on bodytyping in pidgin

  • Description: When a body is typing in pdigin (or some other IM client) inform this fact using sound instead of saying "body is typing"
  • Details: This is a feature heavily used by the JAWS screen reader users and seems good from an usability perspective.

Audio support configuration

  • Description: The user should be able to configure audio support in orca
  • Details: some users don't like sounds going around, some other pref ere them to play lower, some must turn the off on some occasions. There must be a way to turn sounds off, control there overall volume, possibly change the sound card where they are played, Etc. There must be also a good default configuration for most users, for example use gnome settings for Te sound card output.

Configuring audio Support

TODO

implementation

Developments on the implementation are also being discussed and posted on bugzilla:

Gstreamer

The audio support is implemented with the Gstreamer Multimedia Framework. Due to python gstreamer integration problems in some platforms (OpenSolaris for example) the current implementation uses an os.system call to run the gst-launch tool to play a sound. When things are solved with this we will use the python gstreamer bindings.

Documentation

Here it is some gstreamer documentation I found useful:

Audio Files and Formats

The current implementation only allows playing of wav files specified by absolut path (or maby relative to $PWD I guess). We must define what formats should be supported (wav, vorbis, etc.) and a standard location for audio files. On other subject, if the interface will use sounds and will depend on sound files, we may have to distribute some sound files with the source, it must be defined what and how to distribute.


The information on this page and the other Orca-related pages on this site are distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.


CategoryAccessibility

Projects/Orca/Audio (last edited 2013-12-28 19:17:57 by WilliamJonMcCann)