Over the past few years, I’ve been exploring the intersection of sound technologies, sensor data, and accessible design—a journey that began with my early work on sonification and tongue gestures. One project, in particular, has left a lasting impression on me: our Silent Speech initiative. This work not only pushed the boundaries of non-invasive, hands-free interaction but also provided a fresh perspective on how subtle sensor signals can be transformed into meaningful communication.
Silent Speech Research
During my time as a Post Doc in the Brain-Computer Interface group at Microsoft Research, I had the opportunity to lead a grant that brought together product and research teams. This collaborative effort not only paved the way for innovative breakthroughs but also funded a PhD internship through the Microsoft Research internship program. One of the key contributors was PhD intern Tanmay Srivastava, who focused on leveraging IMU-based silent speech detection. His work was instrumental in capturing and classifying the subtle neuromuscular signals associated with silent, closed-mouth articulations.
Our system was built using off-the-shelf sensors—IMUs, EEG, and PPG—to capture these signals and translate them into commands. While the technical details of sensor fusion and gesture recognition are described in our publication, what I find most fascinating is how this work extends the idea of using sensor data to produce sound—a concept I’ve explored before in my sonification projects.
Integrating with LLMs: A Quiet Revolution
One of the most memorable aspects of the project was our integration with GPT-4. At a time when access to GPT-4 was highly restricted—requiring special APIs—we managed to connect our silent speech interface to this powerful language model. The concept was both straightforward and transformative: enable users to interact with a large language model through silent articulation. In essence, you could “speak” to the system and receive context-aware responses without producing a single audible word.
This integration wasn’t merely a technical milestone; it also signaled a new way of thinking about communication—one that is discreet, accessible, and profoundly inclusive.
Why it Matters
At its core, the Silent Speech project is about more than just pushing technological boundaries—it’s about creating a meaningful impact on everyday communication. By transforming minimal sensor readings into actionable commands, we open up new channels for interaction in environments where traditional voice interfaces may fall short. Whether in noisy public spaces, quiet settings, or for individuals with speech impairments, this technology has the potential to offer a truly accessible alternative.
For me, this work resonates with my broader interests in novel sound technologies and accessible design. Just as my earlier projects on sonification sought to make data audible in innovative ways, this initiative reimagines communication—allowing technology to be both intuitive and inclusive. It demonstrates that by harnessing the quiet potential of our bodies, we can bridge gaps between technology and human need in unexpected and powerful ways.
Final Thoughts
Reflecting on the journey of the Silent Speech project, I am struck by the collaborative spirit that drove its success. From the grant that united product and research teams to the contributions of talented individuals like Tanmay Srivastava, every step of this process reaffirmed my belief that innovation thrives at the intersection of diverse expertise and shared vision.
Integrating our silent speech interface with GPT-4 was not just a technical achievement—it was a conceptual leap that reshaped how I think about communication. In a world where speaking aloud isn’t always possible or desirable, enabling silent, thoughtful interaction opens up countless possibilities. As I continue to explore new frontiers in sound, sensor data, and inclusive design, I remain excited about the future. This project is a testament to the idea that sometimes, the most profound breakthroughs come not from loud proclamations, but from quiet revolutions.
Publications
Srivastava, T., Winters, R. M., Gable, T., Wang, Y. T.; LaScala, T.; Tashev, I. J. (2024). Whispering Wearables: Multimodal Approach to Silent Speech Recognition with Head-Worn Devices. ICMI ’24: International Conference on Multimodal Interaction, 214-223.
