{"id":237,"date":"2018-11-02T22:17:38","date_gmt":"2018-11-02T22:17:38","guid":{"rendered":"https:\/\/mikewinters.io\/?p=237"},"modified":"2019-10-04T19:24:22","modified_gmt":"2019-10-04T19:24:22","slug":"ai-driven-audio-in-social-media","status":"publish","type":"post","link":"https:\/\/mikewinters.io\/?p=237","title":{"rendered":"AI-Enhanced Social Media Images"},"content":{"rendered":"\n

So much of the content in social media platforms is visual: GIFS, photographs, memes, infographics, bitmojis and emojis to name a few. Sound and music are such vital components to the enjoyment of media, but have been largely overlooked.<\/p>\n\n\n\n

I joined Microsoft Research in the summer of 2016 for an internship with researchers in computer vision and accessibility. We explored novel ways of transforming social media—especially images—into rich audio experiences for the blind.<\/p>\n\n\n\n

\"\"
Social media content was analyzed by AI algorithms to retrieve high-level data features for sonification.<\/figcaption><\/figure>\n\n\n\n

Image sonification systems typically use low-level data such as RGB-values as the basis for their audio-mappings. However, using Microsoft Cognitive Services (AI), I was able to extract high-level features such as people, sentiment, emotions, environments and objects. This made it easy to craft soundscapes that actually sounded like the images.<\/p>\n\n\n\n

\"\"
Short sounds play right after the username and longer sounds play in the background of the post text.<\/figcaption><\/figure>\n\n\n\n

I ultimately settled on a design that used speech, sonification, auditory icons, soundscapes and music. I had to be smart about the way I arranged them so that people could hear everything without adding too much extra time to the browsing experience. I hope that these types of technologies start finding their way into apps soon. So much of the visual content on the web is still unaccessible to the blind. <\/p>\n\n\n\n

\nhttps:\/\/www.youtube.com\/watch?v=1W2uQYfvb1I&feature=youtu.be\n<\/div>
A sequence of five social media posts contrast text-to-speech with the AI-driven audio additions.<\/figcaption><\/figure>\n\n\n\n