Speech Therapy for the Hearing Impaired
Here’s a summary of the highlighted features of Vocle. These featured in the minimum viable product that we built towards the second half of the project.
There is a personalised avatar who guides the child through the stages of the training module and encourages her to practice.
The application provides immediate feedback, akin to a doctor or a therapist. This ensures the exercises are effective.
The child is enticed to practice more with the help of rewards. Here, we have badges and a streak to keep the child engaged.
Imagine this situation: You have your headphones on with your favorite song on maximum volume. Your friend next to you calls out your name and you reply, with your headphones still on. The next thing you know, your fiend yanks your headphones off and asks you to stop shouting.
This is a scenario that most of us would have experienced in our lives. What happened was that your headphones blocked out the feedback from your voice and you didn’t have the cues you normally do to adjust the volume and intensity you spoke at.
Welcome to the world of the hard of hearing.
Observing a speech therapy session for a very young child (photographed with permission)
Hard of hearing kids have improper speech due to the lack of voice feedback. As a result, a lot of them choose to stay silent even when they have properly developed vocal chords.
Children born with a hearing impairment need to be enrolled in speech therapy. Once a hearing implant is fitted into the child’s ear, she is required to attend multiple sessions with a licensed speech therapist. Here’s the ground reality of the state of hard of hearing kids and speech therapy in India:
There are tools and equipment used to diagnose and train children but they are incredibly expensive and inaccessible for a vast majority of Indians.
Being able to speak is a ridiculously complicated ability that sets us apart from other species. While there are many factors that dictate the quality of speech, it comes down to 3 variables:
The pitch levels that convey tone of voice
How loud or soft the speaker is; the volume
A factor of time that conveys emotion
These factors control the intonation of the speaker. Intonation is the last stage of speech therapy that teaches children how to modulate their voice and better convey their feelings. While the importance of intonation is highlighted, it is often ignored by therapists and children alike.
This is where we spotted an opportunity for improvement. We wanted to build a product that would bring speech therapy to the child’s home, free of cost.
design a product that enables hard of hearing kids aged 8-10 years with cochlear implants to improve their intonation through carefully crafted training exercises and realtime feedback on speech, with minimal remote monitoring by the doctor
From the get go, there were a few fundamental design objectives our explorations needed to meet. These acted as a guiding north star for us to evaluate an idea and prioritise features.
To enable the child to train their voice limited external guidance
To incorporate medically-certified training exercises
To provide real-time feedback to the child’s intonation levels
To engage the child with playful gameplays and incentives
Inspired by the lean methodology, we adopted the ethos of Build • Validate • Learn throughout the design process, by constantly iterating and testing our designs ovet the span of 3 months.
How the app and its interface evolved over the course of 4 months.
Here are a few snapshots that wil help you understand the product.
The child logs into the product using an existing ID or creates a new account
She is immediately shown a fun interactive onboarding experience
She lands on the home screen to discover various curated lessons for her.
The trainer works in three stages:
1)The child listens to an Animoji speak out a sentence
2)The child can speak into the phone while recording it
3)She gets realtime feedback with the pitch and intensity curves.
Streaks and badges to incentivise the kids to reinforce positive behaviour.
We recruited software developers to build out an MVP that can be tested with our target audience. The MVP focuses on the core functionality, which is the real-time feedback of intonation levels.
Our MVP was built using a Python library called Praat-Parselmouth, an open source library that records an audio file and charts frequency and intensity graphs. We used this library to map the child’s speech. Prerecorded reference files of the sentences used for training were also run through the script and their corresponding frequency and intensity curves were recorded.
The first step was to extract an array of values corresponding to regular intervals on the frequency and intensity curves. These values were used to compare the reference audio with the child’s recorded speech.
Another crucial step in this process was to accommodate for different speeds in speaking. In reality, different people speak at different speeds and comparing the two curves without accommodating for this owuld lead to inaccurate and ineffective results. Using Dynamic Time Warping, the two curves were compressed or stretched to coincide with time.
We ran audio files of different children to establish the thresholds of good and bad speech.
The final step is for the script to measure the difference in the values of the graphs and different intervals. A cumulative score was generated for each comparison and this dictates how many stars should be awarded to the child for that particular lesson.
We spoke with 7 therapists and speech experts who helped us access users within our target demographic and test out our application. Some objectives driving the testing session included whether the lessons were effective and whether the feedback was appropriate.
All the feedback received from our subject matter experts and users were recorded and incorporated
In the middle of the testing was when the pandemic hit and we went remote. To battle the issue of access to feedback during the lockdown, we used WhatsApp as a common platform to record feedback, test, and rapidly deploy updates.
Screenshots of our conversation with therapists recording the feedback we received from them
When following the Build • Validate • Learn process, we were constantly in touch with subject matter experts, therapists, and most importantly, the hard of hearing kids themselves. The All India Institute of Speech and Hearing, India was our go-to for all things speech and regularly provided us with insights and feedback throughout the process. Their guidance was invaluable and supported us at every step!
A quick overview of the research methods we employed in the course of this project.
The more stakeholders we spoke to, the more we got to refine our target audience, the problem statement, and subsequently the approach to the solution. One of our biggest takeaways from this project is the unique interdisciplinary nature of our interactions. We documented our experience and verbalised it into a simple framework that would help designers working in an interdisciplinary context to structure their approach better.
The Infinity Process: A framework for interdisciplinary problem-solving.
The Infinity Framework was accepted to be published in the Proceedings of the International Conference on Research and Design 2019, a Springer publication.
This was the most gratifying experience for me as a designer, where I got to directly affect someone with an idea that has the potential to change lives! There were many challenges along the way, not the least being the pandemic hitting us in the middle of the project. By pushing ourselves at every step of the way, we achieved so much more than we hoped for and we are incredibly proud.
We officially wound down work on Vocle in May 2020, with each of us pursuing other avenues.