A New Paradigm for Human-AI Interaction
While I was a Post Doc Researcher in Brain-Computer Interface Group at Microsoft Research, I had the opportunity to initiate my own research directions. As a PhD in music technology (where I learned how to BCI), I was inspired by the ideas of music theorists such as Pauline Oliveros, R. Murray Schafer, and John Cage, and who had cultivated noise as a valuable part of music.
Noise happens to be one of the primary challenges of building a brain-computer interface (BCI) for the real-world. To measure brainwaves, the sensors had to be extremely sensitive (brainwaves are usually measured in the micro-volts!). But in the real-world, even the small, imperceptible movements of the neck, jaw, or even eyes can produce signals much stronger than those produced by the brain.
I was curious how I could put the natural occurrence of noise in BCIs to good use. If I could show how useful the noise was, I reasoned, perhaps I could help get the sensors into everyday devices, where they could also be used for measured brainwaves. With my previous work on heartbeats, brainwave synchrony, and cancer sonification, I was curious about other ways to make the body make sound. Although I couldn’t hear it yet, I realized that when I hit my tongue against my teeth my whole head would shake. If I could make this into a sound, then it could be used as a signal.
This was the theory behind my work in Tongue Gesture Recognition. The tongue is a hidden, muscular organ responsible for essential activities like eating and talking. If sensitive EEG electrodes can easily pick up noise artifacts, then maybe they can measure the motion of the tongue as well.
Pioneering Research
To test this hypothesis, I worked with Tan Gemicioglu to develop a controlled laboratory study where we asked participants to complete eight different tongue gestures while wearing a HP Reverb G2 and Muse 2 headset. These headsets contained multiple types of sensors including EEG, PPG, IMU, Eye-Tracking and Head Tracking. We recorded these signals while participants performed the tongue gestures, and then trained an ML model to classify the signals. To our surprise, we achieved >90% accuracy, and >80% accuracy just using the IMU sensor!
Accessibility in Mind
Having made the classifier, we wanted to see how it performed in real-world. We imagined using the tongue as a a virtual “mouse-click,” but while we had a means to click, we didn’t have a way to point. For that purpose, we combined tongue gestures with eye-tracking technology specifically a Tobii eye-tracker, which enabled us to point at objects on a screen, and then to select them by tapping the tongue. This paradigm can be used for anything a mouse can do on a screen, but was hands-free and voice-free, making an private and accessible alternative to using hands or a mouse. We teamed up with
Another Door Opens
We published our work in three papers, and posted our dataset and model online to share with others and advance the field. The following fall, after the dust had settled, I used our results to apply for an internal grant to collaborate with Microsoft Mixed Reality and help get these results into a product. The result was my subsequent research on Silent Speech — a private and accessible alternative to voice!
Patents
Winters, R. M., Gemicioglu, T., Gable, T. M., Wang Y. T., Tashev, I. J. (2024). Inertial Sensing of Tongue Gestures. US Patent 18075786.
Publications
Gemicioglu, T., Winters, R. M., Wang Y. T., Gable, T. M., Tashev, I. J. (2023). TongueTap: Multimodal Tongue Gesture Recognition with Head-Worn Devices. ICMI ’23: International Conference on Multimodal Interaction, 564-573
Gemicioglu, T. Winters, R. M. Wang, Y. T., Gable, T. M., Paradiso, A. Tashev, I. J. (2023). Gaze & Tongue: A Subtle, Hands-Free Interaction for Head-Worn Devices. CHI EA ’23: Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems
Gemicioglu, T. Winters, R. M., Wang, Y. T., Tashev, I. J. (2022). Tongue Gestures for Hands-Free Interaction in Head Worn Displays. 26th Annual International Symposium on Wearable Computers (UbiComp/ISWC ’22 Adjunct), 38-40.
