Vocle

Speech Therapy for the Hearing Impaired

A lot of kids born with hearing impairment give up speaking even though they have fully developed vocal chords. Speech therapy is cumbersome and extensive, requiring several years of effort.

In collaboration with the All India Institute of Speech and Hearing, I identified areas of design intervention in the field of speech therapy for children and designed an accessible tool for children to continue therapy at home.

Vocle is a one-of-a-kind speech therapy application that allows any child to train their voice with the help of realtime feedback and captivating gameplay, with minimal to no therapist intervention.

Vocle is designed to make speech therapy fun and engaging

Here’s a summary of the highlighted features of Vocle. These featured in the minimum viable product that we built towards the second half of the project.

Personalised trainers

There is a personalised avatar who guides the child through the stages of the training module and encourages her to practice.

Critical feedback

The application provides immediate feedback, akin to a doctor or a therapist. This ensures the exercises are effective.

Exciting rewards

The child is enticed to practice more with the help of rewards. Here, we have badges and a streak to keep the child engaged.

Speech therapy is expensive, inaccessible and dropped midway for too many kids across India

Imagine this situation: You have your headphones on with your favorite song on maximum volume. Your friend next to you calls out your name and you reply, with your headphones still on. The next thing you know, your fiend yanks your headphones off and asks you to stop shouting.

This is a scenario that most of us would have experienced in our lives. What happened was that your headphones blocked out the feedback from your voice and you didn’t have the cues you normally do to adjust the volume and intensity you spoke at.

Welcome to the world of the hard of hearing.

Observing a speech therapy session for a very young child (photographed with permission)

Hard of hearing kids have improper speech due to the lack of voice feedback. As a result, a lot of them choose to stay silent even when they have properly developed vocal chords.

Children born with a hearing impairment need to be enrolled in speech therapy. Once a hearing implant is fitted into the child’s ear, she is required to attend multiple sessions with a licensed speech therapist. Here’s the ground reality of the state of hard of hearing kids and speech therapy in India:

There are tools and equipment used to diagnose and train children but they are incredibly expensive and inaccessible for a vast majority of Indians.

There are 3 components of voice that are trained by a speech therapist

Being able to speak is a ridiculously complicated ability that sets us apart from other species. While there are many factors that dictate the quality of speech, it comes down to 3 variables:

Frequency

The pitch levels that convey tone of voice

Intensity

How loud or soft the speaker is; the volume

Rhythm

A factor of time that conveys emotion

These factors control the intonation of the speaker. Intonation is the last stage of speech therapy that teaches children how to modulate their voice and better convey their feelings. While the importance of intonation is highlighted, it is often ignored by therapists and children alike.

This is where we spotted an opportunity for improvement. We wanted to build a product that would bring speech therapy to the child’s home, free of cost.

design a product that enables hard of hearing kids aged 8-10 years with cochlear implants to improve their intonation through carefully crafted training exercises and realtime feedback on speech, with minimal remote monitoring by the doctor

Our designs were driven by a need for accuracy and kid-friendly experiences

From the get go, there were a few fundamental design objectives our explorations needed to meet. These acted as a guiding north star for us to evaluate an idea and prioritise features.

01

To enable the child to train their voice limited external guidance

02

To incorporate medically-certified training exercises

03

To provide real-time feedback to the child’s intonation levels

04

To engage the child with playful gameplays and incentives

We refined and redesigned the interface with feedback from SMEs and users

Inspired by the lean methodology, we adopted the ethos of Build • Validate • Learn throughout the design process, by constantly iterating and testing our designs ovet the span of 3 months.

How the app and its interface evolved over the course of 4 months.

We designed key screens to build the minimum viable product of our solution

Here are a few snapshots that wil help you understand the product.

The child logs into the product using an existing ID or creates a new account

She is immediately shown a fun interactive onboarding experience

She lands on the home screen to discover various curated lessons for her.

The trainer works in three stages:
1)The child listens to an Animoji speak out a sentence
2)The child can speak into the phone while recording it
3)She gets realtime feedback with the pitch and intensity curves.

Streaks and badges to incentivise the kids to reinforce positive behaviour.

We worked closely with 4 developers to build a testable Android application

We recruited software developers to build out an MVP that can be tested with our target audience. The MVP focuses on the core functionality, which is the real-time feedback of intonation levels.

Our MVP was built using a Python library called Praat-Parselmouth, an open source library that records an audio file and charts frequency and intensity graphs. We used this library to map the child’s speech. Prerecorded reference files of the sentences used for training were also run through the script and their corresponding frequency and intensity curves were recorded.

The first step was to extract an array of values corresponding to regular intervals on the frequency and intensity curves. These values were used to compare the reference audio with the child’s recorded speech.

Another crucial step in this process was to accommodate for different speeds in speaking. In reality, different people speak at different speeds and comparing the two curves without accommodating for this owuld lead to inaccurate and ineffective results. Using Dynamic Time Warping, the two curves were compressed or stretched to coincide with time.

We ran audio files of different children to establish the thresholds of good and bad speech.

The final step is for the script to measure the difference in the values of the graphs and different intervals. A cumulative score was generated for each comparison and this dictates how many stars should be awarded to the child for that particular lesson.

We showed our MVP to children, speech therapists and parents to validate our solution

We spoke with 7 therapists and speech experts who helped us access users within our target demographic and test out our application. Some objectives driving the testing session included whether the lessons were effective and whether the feedback was appropriate.

All the feedback received from our subject matter experts and users were recorded and incorporated

In the middle of the testing was when the pandemic hit and we went remote. To battle the issue of access to feedback during the lockdown, we used WhatsApp as a common platform to record feedback, test, and rapidly deploy updates.

Screenshots of our conversation with therapists recording the feedback we received from them

We documented and published our interdisplinary design process in ICoRD’19

When following the Build • Validate • Learn process, we were constantly in touch with subject matter experts, therapists, and most importantly, the hard of hearing kids themselves. The All India Institute of Speech and Hearing, India was our go-to for all things speech and regularly provided us with insights and feedback throughout the process. Their guidance was invaluable and supported us at every step!

A quick overview of the research methods we employed in the course of this project.

The more stakeholders we spoke to, the more we got to refine our target audience, the problem statement, and subsequently the approach to the solution. One of our biggest takeaways from this project is the unique interdisciplinary nature of our interactions. We documented our experience and verbalised it into a simple framework that would help designers working in an interdisciplinary context to structure their approach better.

The Infinity Process: A framework for interdisciplinary problem-solving.

The Infinity Framework was accepted to be published in the Proceedings of the International Conference on Research and Design 2019, a Springer publication.

Closing thoughts

This was the most gratifying experience for me as a designer, where I got to directly affect someone with an idea that has the potential to change lives! There were many challenges along the way, not the least being the pandemic hitting us in the middle of the project. By pushing ourselves at every step of the way, we achieved so much more than we hoped for and we are incredibly proud.

We officially wound down work on Vocle in May 2020, with each of us pursuing other avenues.