Brain-Controlled Robotic Drumming Arm

As a PhD student in Gil Weinberg’s Robotic Musicianship Group, I worked around a lot of robots. When I arrived, Shimon & Friends were an established group of robotic musicans that could play, listen, improvise and even dance.

In my second year, our group began designing a new robot that would augment a human drummer with a third arm. This idea raised an interesting question: How would a drummer (who is already using both arms ands legs) also control the arm?

We tried computer vision, piezos, head-mounted gyroscopes and EMGs, but the one that was most exciting for me was EEG. With funding from the NSF (IIS EAGER 1550397), we bought a Brain Vision 32-Channel Active Electrode System.

I was given the challenging job of learning how to use the EEG to control the arm. I had no previous experience with EEG, but I learned by reading, asking questions and collaborating with other researchers and labs on campus. Especially helpful were the people in Melody Moore Jackson’s GT BrainLab and the students in my VIP group. Over several months, I learned a a lot through hands-on trial and error. By the end of the semester, we could use the Steady-State Visually Evoked Potential (SSVEP) to send musical control signals to the arm.

This was a demo video made by my students using SSVEP to send control signals to the robotic drumming arm. Curiously, the drummer uses his free hand to shoot a POV video.

SSVEP is a strong signal and only needs a few sensors on the visual cortex. For music performance, this meant that we could use a wireless, dry EEG headband and still capture all of the information we needed.

Here’s a demo video showing me controlling OpenVIBE’s SSVEP simulator using three sensors in the back of the headband.

This wireless, dry headband had an ideal form-factor, and the SSVEP was a strong candidate for realtime brain-music control. However, I wanted to learn more about EEG and to learn how to recognize authentically musical ideas in the brain. This lead me towards an entirely different path in the neuroscience of music performance.