Vocal Joystick
http://ssli.ee.washington.edu/vj/The goal of this project is to develop a novel system that we call the Vocal Joystick (VJ). This device will enable individuals with motor impairments to use vocal parameters to control objects on a computer screen (buttons, sliders, etc.) and ultimately electro-mechanical instruments (e.g., robotic arms, wireless home automation devices).
Standard spoken language can be quite inefficient for such continuous control tasks and is often recognized poorly by automatic speech recognizers. The VJ system, in contrast, will allow users to exploit a large and varied set of vocalizations for both continuous and discrete motion control, and its selection will be optimized for high discrimination ability and low communication bandwidth. Furthermore, the users are able to perceive visualized feedbacks from the system and make adjustments on the fly. This may include regular speech sounds, such as vowels and consonants, but the primary focus will be on the variation of individual acoustic-phonetic parameters like pitch, energy, vowel quality and voice quality.
Standard spoken language can be quite inefficient for such continuous control tasks and is often recognized poorly by automatic speech recognizers. The VJ system, in contrast, will allow users to exploit a large and varied set of vocalizations for both continuous and discrete motion control, and its selection will be optimized for high discrimination ability and low communication bandwidth. Furthermore, the users are able to perceive visualized feedbacks from the system and make adjustments on the fly. This may include regular speech sounds, such as vowels and consonants, but the primary focus will be on the variation of individual acoustic-phonetic parameters like pitch, energy, vowel quality and voice quality.
The diagram above shows the mapping of the vowel sounds recognized by the Vocal Joystick engine to the radial direction resulting in a mouse pointer movement. The VJ engine also captures loudness and pitch information, which can be used to control the speed of the pointer movement.
Publications
|
Longitudinal study of people learning to use continuous voice-based cursor control Susumu Harada, Jacob O. Wobbrock, Jonathan Malkin, Jeff Bilmes and James A. Landay Conference on Human Factors in Computing Systems, 2009. Full Paper (PDF) |
|
The VoiceBot: A Voice Controlled Robot Arm Brandi House, Jonathan Malkin and Jeff Bilmes Conference on Human Factors in Computing Systems, 2009. Full Paper (PDF) |
|
The Vocal Joystick: Evaluation of Voice-based Cursor Control Techniques Susumu Harada, James A. Landay, Jonathan Malkin, Xiao Li and Jeff Bilmes ACM SIGACCESS Conference on Assistive Technologies, 2006. Full Paper (PDF) |