Engineers across the board are working to improve the quality of life for the deaf and hearing impaired.

A Will Ferrell movie isn’t all that funny when you can’t hear the jokes he’s cracking. By the same token, going to church doesn’t have the same impact without the words of the pastor. And your excitement level at your alma mater’s championship basketball game might be a little lower if you can’t hear the commentary over the loudspeaker.

For the deaf and hard of hearing, these situations are all too common because captions with information audible to the general public are often available only in select movie theaters. But engineers at the Georgia Tech Research Institute (GTRI) are trying to change that by bringing text captions to a wide range of places, from government meetings to places of worship.

But GTRI researchers aren’t the only engineers working to help the almost 30 million Americans who are deaf or hard of hearing. Engineers across the nation are making strides in diagnosing and treating hearing problems as well as creating technologies to make everyday life for the deaf and hard of hearing a little easier.

The GTRI captioning technology, a new wearable captioning system that uses mobile wireless technology, allows users to receive text via a personal digital assistant (PDA) or laptop computer. They can view the captions on the PDA or laptop screen, or they can use a microdisplay that clips on to their glasses or a headband. Although the head display is positioned close to the user’s eyes, the display floats the words in front of the users, who can adjust the location and position the text where they want in their visual field.

A transmitter at a theater, museum or another location sends the captions via standard wireless technology. That technology is already installed in a variety of places, including sports stadiums, restaurants and business districts. Project director and electrical engineer Leanne West says she sees the system being useful at a wide variety of places beyond movie theaters. West says she’s gotten calls from schools interested in using it. The system can transmit multiple text streams, which means it can be used for language translation or to transmit other information. A baseball stadium, for example, could send the statistics of a player as he comes up to bat. “It gives the venue a lot of options as to what they want to send out,” West says.

Captions can be prerecorded, easily done at places like movie theaters, or generated in real time with a shorthand typing method. West says it’s possible to use voice-recognition to create captions for system, but some kinks, including punctuation and accuracy, are still being worked out.

The idea was hatched about five years ago during a conversation West was having with coworkers about litigation, now settled, over the lack of captioning at some movie theaters. “I just thought, well, why can’t you do it everywhere and would that be a useful thing?” West says.

So West and her colleagues took their idea to the Georgia Council for the Hearing Impaired and to Self-Help for Hard of Hearing People to gauge the interest and need for this technology. “We didn’t want to build something people didn’t want, so we asked if it would be something they would want to use,” West says. After hearing strongly positive feedback from the two groups, the researchers moved forward.

With funding from GTRI and a grant from the Wireless Rehabilitation Engineering Research Center at Georgia Tech, which is funded by the National Institute on Disability and Rehabilitation Research, the researchers worked with the Georgia Council and Self-Help groups to gather participants to test the system. The feedback, from two trials with deaf or hard of hearing participants ranging in age from 15 to 75, was overwhelmingly in favor of the system. “People were really excited about it,” West says. “They wanted to know when it would become available.” The participants also offered suggestions for improvement, which West says the researchers tried to incorporate into the prototype. Study participants suggested the importance of being able to customize the font size of the captions and the focus, both of which have been added.

\While the researchers work with a company to license and sell the captioning system, they’re also improving the system in terms of security for the text stream. West says she hopes the system will be available within six months to a year.

Early Detection

At the University of Cincinnati and the Imaging Research Center at the Cincinnati Children’s Hospital Medical Center, engineering and medicine are working together—and getting good results. Scott Holland, McLaurin Scholar and professor of radiology and biomedical engineering at the University of Cincinnati, and Dan Choo, associate professor of otolaryngology head and neck surgery and founder of the Center for Hearing and Deafness Research at the hospital, are researching ways of predicting how much a cochlear implant would help deaf or hearing impaired infants and children. A cochlear implant is a small electronic device that can help provide a sense of sound and a better understanding of speech.

With a grant from the National Institute on Deafness and Other Communication Disorders, Holland and Choo are studying how a technique called functional magnetic resonance imaging (fMRI) could be an early, minimally invasive predictor of the usefulness of a cochlear implant. Formerly, the only way of determining that was by actually performing the surgery.

The fMRI maps brain activity while a patient is being stimulated. In children who are deaf but who could likely benefit from the implant, Holland says there is visible brain activity on the fMRI in the auditory and language areas when the infant is exposed to something language-based.

The study is in the primary stage of collecting brain activation maps from a group of normal hearing infants and two groups of hearing impaired infants. After the data is in, Holland says a certain percentage of the hearing impaired infants will receive the implants, and the researchers will study the children’s hearing and language skills for two years. “At end of the process, we should know how well fMRI does in predicting how well they’d do with the implant,” Holland says. In the children with successful implants, within two years of the surgery, speech and speech interpretation could be within normal range.

Like Holland and Choo, Susan Voss is hoping her research will help improve treatments for the deaf and hard of hearing at an early age. Voss, assistant professor of engineering at Smith College, is researching how problems in the ear alter energy reflectance of sound within the ear canal. In infants, it’s crucial to identify what and where the problem is, Voss says, so that parents and doctors can determine the best course of action as early as possible.

While a majority of states require that newborns be tested for hearing loss, Voss, with a grant from the NIH, is working on non-invasive ways to determine specifically where and what the problem is, based on the reflection and absorption of energy. This could be helpful for families with newborns who have failed the initial hearing screening. Parents want to know as soon as possible what the problem is and how to handle it. If the problem can be pinpointed with this non-invasive method, the infant could get the specific treatment he or she needs early on. A parent might be relieved to find out that the failed hearing test resulted from fluid in the middle ear, which typically resolves itself.

Diagnostic tools to measure energy reflectance using a small earphone and microphone that fit within the ear canal are being developed nationally and are in use for research purposes in some clinics. But there’s work to be done on making sure the measurements are accurate and on interpreting the responses. “I think there’s a lot of basic science remaining to be done, specifically questions like what does a reflectance measurement in a normal ear typically look like versus how would you expect the reflectance measurement to look in a diseased ear,” Voss says.

Back at Georgia Tech, Thad Starner, assistant professor in the College of Computing, is in the primary research stages of developing a mobile American Sign Language (ASL)-to-English translator as well as a videogame to help young deaf children improve their signing skills.

Telesign, Starner’s translator, aims to ease conversations between deaf and hearing people when an interpreter isn’t feasible. With the user wearing wrist bracelets and a hat with a camera to track hand movements, the system tracks the signed phrase and then offers a list of pre-programmed English phrases most closely matching what the user signed. The user selects the desired phrase, and the system speaks it out loud.

Modeled after a phrasebook a traveler would take to a foreign country, the phrases are designed to elicit mostly nonverbal responses, such as “Can you point in the direction of the nearest restroom?” Starner and his research team are beginning tests with Telesign, comparing it with handwritten notes and typing on a PDA, which are common ways of communicating.

Building on a recently formed partnership with the Atlanta Area School for the Deaf, Starner is also creating a child-friendly game called CopyCat to help deaf schoolchildren develop their ASL skills early on. The children sign commands to CopyCat’s main character, Iris the cat, and wrist bracelets and a video camera track the child’s signing. If the child signs poorly, Iris looks confused and the child has to try the phrase again.

Although this system, like Telesign, is in the early stages of development, the researchers hope to test the complete system with 9-year-old subjects early this year. As the first educational software that helps deaf children sign, Starner says this could be a significant improvement considering that many children might not be learning to sign at home. Ninety percent of deaf children are born to hearing parents, who often do not know or have low levels of proficiency in sign language. Pilot studies tell Starner that CopyCat might be a hit. “The children play the same game over and over again. They really get into it.”

At Duke University’s Pratt School of Engineering, Leslie Collins, associate professor of electrical and computer engineering, is working to improve cochlear implant software so that users can better hear not only speech but also music. “Music is one thing that implanted individuals have not been happy with,” Collins says.

The software was tested with acoustic models of cochlear implants in hearing individuals last year, and Collins and her team are in the beginning stages of trying it with implant-wearing subjects. Feedback so far shows speech recognition improving from 30 to 80 percent, and Collins says the improvement in music, which is harder to quantify, shows a “fairly dramatic change.”

As the biomedical field grows, there will be more engineering researchers like Cincinnati’s Holland working to help the hearing impaired. And it is growing. Since 1999, undergraduate and graduate degrees awarded in bioengineering have virtually doubled. But it’s not just biomedical engineers who are developing products that address health and medical issues. In Duke’s electrical and computer engineering department, Collins estimates that two-thirds of her colleagues are working on something related to biomedicine. She sees the pairing of engineering with medical and quality of life issues as growing, which she says forecasts good things for the future. “The more bright minds that you can get focusing on medical issues means these problems are going to get solved that much faster.”

Now, that’s music to everyone’s ears.

Lynne Shallcross is associate editor of Prism.