An icon of an orange juice matched with a happy face means “I like orange juice” Likewise, one with a sad face means “I don’t like orange juice”. This is how people with speech difficulties can now communicate using images via a free app developed by the Industrial Design Centre (IDC) of the Indian Institute of Technology – Bombay (IIT-B).
The app, called Jellow Communicator, can be used by children with cerebral palsy, autism spectrum disorder, Down syndrome and brain injury, and adults who have lost their speech to a stroke.
The idea to develop such an app originated in 2004 for children with cerebral palsy. “Children with cerebral palsy have speech problems, and that affects their ability to communicate and this blocks all channels of learning. It affects social, cognitive and motor development,” said Sudha Srinivasan, post-doctoral fellow, IDC, and paediatric speech therapist, who provided clinical content for the app.
Initially, it was conceptualised as a hardware device with each piece costing around Rs13,000. “But it turned out to be expensive. And with mobile phones getting cheaper, the idea then moved on to developing a free app instead of carrying a device,” said professor Ravi Poovaiah, IDC.
Over the last two years, an interdisciplinary team of therapists, graphic designers, software programmers and animators at the IDC School of Design worked on developing the app. Around 8,200 lines of pre-programmed vocabulary each in English and Hindi enables users to communicate what they’d like to eat for breakfast – from eggs and bread to upma and dosa – or learn about animals and birds through the Google text-to-speech converter app available on playstore. A keyboard feature in the app enables the app to speak out custom-typed sentences. The content, icons and accent of the app are developed bearing in mind the socio-cultural context of India.
The team conceived novel Visual Emotional Language Protocol to enhance the language ability of Jellow, and content was developed following research that involved children, therapists, and parents.
The app can also be used by children who are learning to speak to communicate daily activities such as brushing, going to the toilet and bathing. “Toddlers now are interested in mobiles. So when they explore the app, seeing the images and hearing the sounds associated with it will help them improve their vocabulary,” said Srinivasan.
The team is now collecting data to document the app’s effectiveness, improve it based on user feedback. Next, the vocabulary will be available in Bengali and Marathi. “We are waiting for Google text-to-speech engines to be available in other languages to translate the pre-programmee vocabulary,” said Srinivasan.