To the Brain and Back: Speech Emotion Perception, Hearing Loss, and Hearing Aid Use

By Brandon T. Paul, PhD

Spoken language conveys meaning not only through words but also through tone of voice, body language, context, and other dimensions. Emotion is one such feature carried by the voice, and awareness of emotion plays an important role in building and managing social relationships. Hearing joy in someone’s voice helps us share that joy, and hearing sadness can be met with consolation. While emotion is expressed nonverbally through vocalizations such as laughter and crying, speech emotion perception specifically refers to the identification of emotion when someone produces spoken words. If hearing loss makes emotional speech perception difficult, social interactions may suffer, reducing overall psychosocial well-being.

Recently published research [1] has shed new light on the brain basis of speech emotion perception in people with hearing loss who use hearing aids, specifically highlighting the role of brain regions involved in cognition and speech articulation. This article provides a brief overview of speech emotion perception and the brain, then describes this finding and its potential implications for audiology. The focus is on adults, but for interested readers I also reference research on speech emotion perception in children and cochlear implant users.

Speech emotions and the brain

Speech emotions are conveyed through a variety of acoustic patterns over a wide range of time scales, from fast changes in spectrotemporal energy in formant frequencies to slower changes in pitch prosody. For example, speech produced with joy or anger exhibits a higher fundamental pitch, greater pitch variation, greater high-frequency spectral energy, and increased speech rate [2,3]. Sadness, in contrast, is produced with lower fundamental pitch, less pitch variance, and lower speech rate [3].

Once the ear encodes these acoustic details, research shows that brain processing of speech emotion generally parallels that of normal speech. It is worthwhile to quickly review brain pathways for speech to understand the implications of hearing loss and hearing aid use. In brief, speech perception is organized in a “dual pathway” network comprising ventral and dorsal streams (Figure 1) [4]. These pathways can be viewed as a loop in which acoustic information is integrated with motor articulation patterns we use to generate speech, suggesting that speech perception relies on the speech production system. The ventral stream combines auditory signals into increasingly more complex phoneme and word hierarchies in the temporal lobe along a posterior-to-anterior axis (i.e., “mapping sound to meaning”). The dorsal stream consists of sensorimotor integration areas in the inferior parietal lobe and the premotor cortex. This stream matches the fine-grained acoustical speech details with motor articulation templates used to produce them (i.e., “mapping sound to action”). Thus, the perception of speech and speech emotion in the brain involves synthesis of auditory signals and “motor simulation” of these signals.

One difference between normal and emotional speech processing in the brain is the degree of hemispheric activation. Both the left and right hemispheres process speech, but this network is strongly left-lateralized for normal speech [4]. Speech emotion perception is similarly bilateral but may further activate parts of the right temporal lobe that are sensitive to emotion-relevant acoustic features, such as prosody [5,6].

Neuroscience of Speech Emotion Perception In Hearing Loss and Hearing Aid Use

Auditory encoding deficits caused by hearing loss, such as reduced audibility, poorer temporal processing, and broadened auditory filters, disrupt the perception of spectral and prosodic details that convey emotion in the voice [7,8]. Adults with hearing loss are slower [1,9,10] and less accurate at speech emotion identification [10–12], and rate themselves as poorer at hearing emotion in speech compared to adults without hearing loss [11]. Hearing loss also compresses the “extremes” of emotion perceived in auditory signals. One study [13] had participants rate pleasantness and unpleasantness for a variety of non-speech vocalizations and natural sounds. Participants with hearing loss gave lower, more neutral ratings for all sound types, whereas people without hearing loss were more likely to use higher ratings across positive and negative dimensions.

The well-known benefits of hearing aids for speech understanding unfortunately do not extend to speech emotion perception. Research studies repeatedly show that hearing aid use provides little to no benefit for speech emotion identification [11, 14–16], suggesting that typical processing algorithms may not appropriately preserve or amplify speech emotion cues. Speech emotion perception is also troublesome for children and adults who use cochlear implants [17, 18, see 19 for a review], likely because of the device’s extremely limited spectral resolution and reduced dynamic range. These findings affirm that hearing loss impairs emotional communication and available treatment options do not adequately remedy this.

How do speech emotion perception deficits in hearing loss and hearing aid use affect brain activity, and what can we learn from this? An issue with imaging methods like fMRI or EEG is that these technologies detect magnetic or electric fields, and hearing aids are either incompatible with or introduce artifacts into the scans or recordings, making it difficult to interpret the results. A study published this year by Dang et al. [1] sidestepped this problem by using functional near-infrared spectroscopy (fNIRS), a neuroimaging method that uses near-infrared light to measure region-specific changes in cortical blood oxygenation, with increased oxygenation indicating cortical activation. Because fNIRS measures light rather than electromagnetic fields, it is fully compatible with hearing aids and cochlear implants. Participants in Dang and colleagues’ study were older adult hearing aid users, compared with a control group with normal hearing and a similar age. Aided users were instructed to use their preferred processor settings, and they were also tested when unaided. The participants’ task involved listening to lists of lexically identical sentences spoken with different emotions and pressing a button when they noticed that the speech emotion had changed. Consistent with previous research, listeners with hearing loss were slower to notice the emotion switch than normal-hearing listeners, regardless of hearing aid use.

Before an emotion switch, Dang and colleagues recorded participants’ brain activity using fNIRS. This allowed them to examine cortical activation at times when listeners were attending to speech emotion cues. During unaided listening, participants with hearing loss showed elevated activity in frontal brain regions and greater connectivity (i.e., synchronized activation) between the auditory cortex and frontal brain areas compared to normal-hearing control participants. This is consistent with research showing that cognitive or attentional control processes in the frontal lobe are activated when hearing loss impairs speech perception [20], but Dang et al. now demonstrate this frontal involvement during speech emotion perception. This result further corroborated an earlier fMRI study in unaided listeners with mild-to-moderate hearing loss that used non-speech vocalizations [8], together implying that people with hearing loss rely on cognitive strategies for emotion processing.

While aided, Dang et al.’s participants did not show greater recruitment of frontal brain networks as was found for unaided listening. Rather, aided listening was associated with increased activity in the inferior parietal lobe, a key node of the dorsal speech pathway. This could mean that hearing aids restored audibility, partially re-engaging the dorsal speech stream and removing the burden from cognitive control areas. However, notably, inferior parietal lobe activation was associated with reducedconnectivity to frontal brain areas. Dang and co-authors argued that this configuration of brain activity, without an accompanying increase in emotion change detection, reflected sensorimotor integration failure during aided listening: signal processing algorithms in the hearing aid that are designed to boost speech perception, including nonlinear amplitude compression and uneven gain applied across the aided frequency range, may have distorted the spectrotemporal cues that carry speech emotion. This may disrupt the brain’s ability to activate the appropriate articulatory templates. In other words, the hearing aid settings may have artificially enhanced speech signals to a degree that hindered sensorimotor simulation for speech emotion processing. This novel finding opens the door for new research, such as how to design processor strategies that preserve speech emotion information, and whether this enables greater sensorimotor integration in the brain.

Implications for Audiology

Given its importance for interpersonal communication and the management of social relationships, it is important to explicitly discuss speech emotion perception with hearing loss patients. They should be aware that difficulty with speech emotion perception is commonplace, and that hearing aid use is not guaranteed to restore this ability. In everyday settings, it may be helpful for conversation partners to express emotions overtly through words when speaking with hearing aid users or people with hearing loss.

A helpful self-report inventory that can help audiologists characterize speech emotion perception difficulty in patients is the Emotional Communication in Hearing Questionnaire (EMO-CHeQ) [11] EMO-CHeQ is a validated instrument with strong psychometric properties, and its 16-item format is agreeable with tight clinical schedules. Patients can also fill it out before or after appointments. Importantly, EMO-CHeQ scores predict performance on speech perception identification tasks and can distinguish emotional communication handicap between listeners with normal and impaired hearing.

---

Acknowledgements

I thank Carmen Dang and Frank Russo for their comments and feedback on an earlier draft of this article.

---

References

1. Dang C, Singh G, Russo FA. Hearing Aids Reshape Neural Processing of Emotional Speech Without Improving Emotion Perception. Trends in Hearing. 2026 Jul;30:23312165261465515.
2. Scherer KR. Vocal affect expression: a review and a model for future research. Psychol Bull. 1986;99:143–165.
3. Murray IR, Arnott JL. Toward simulation of emotion in synthetic speech: a review of the literature on human vocal emotion. J Acoust Soc Am. 1993; 93:1097–1108.
4. Rauschecker JP. Ventral and dorsal streams in the evolution of speech and language. Frontiers in evolutionary neuroscience. 2012 May 15;4:7.
5. Schirmer A, Kotz SA. Beyond the right hemisphere: brain mechanisms mediating vocal emotional processing. Trends in cognitive sciences. 2006 Jan 1;10(1):24-30.
6. Mauchand M, Zhang S. Disentangling emotional signals in the brain: an ALE meta-analysis of vocal affect perception. Cognitive, Affective, & Behavioral Neuroscience. 2023 Feb;23(1):17-29.
7. Chatterjee M, Kulkarni AM, Combs J, Fitzpatrick D, Pitts A, Hagemann P, Lewis D. Age-related changes in acoustic cue weighting for emotional prosody identification by adult listeners. Ear and Hearing. 2026 Mar 1;47(2):558-75.
8. Buono GH, Crukley J, Hornsby BW, Picou EM. Loss of high-or low-frequency audibility can partially explain effects of hearing loss on emotional responses to non-speech sounds. Hearing research. 2021 Mar 1;401:108153.
9. Husain FT, Carpenter-Thompson JR, Schmidt SA. The effect of mild-to-moderate hearing loss on auditory and emotion processing networks. Frontiers in systems neuroscience. 2014 Feb 4;8:10.
10. Christensen JA, Sis J, Kulkarni AM, Chatterjee M. Effects of age and hearing loss on the recognition of emotions in speech. Ear and hearing. 2019 Sep;40(5):1069.
11. Singh G, Liskovoi L, Launer S, Russo F. The emotional communication in hearing questionnaire (EMO-CHeQ): Development and evaluation. Ear and Hearing. 2019 Mar 1;40(2):260-71.
12. Rigo TG, Lieberman DA. Nonverbal sensitivity of normal-hearing and hearing-impaired older adults. Ear and hearing. 1989 Jun 1;10(3):184-9.
13. Picou EM. How hearing loss and age affect emotional responses to nonspeech sounds. Journal of Speech, Language, and Hearing Research. 2016 Oct;59(5):1233-46.
14. Goy H, Pichora-Fuller MK, Singh G, Russo FA. Hearing aids benefit recognition of words in emotional speech but not emotion identification. Trends in Hearing. 2018 Sep;22:2331216518801736.
15. Waaramaa, T., Kukkonen, T., Stoltz, M., & Geneid, A. (2018). Hearing impairment and emotion identification from auditory and visual stimuli. International Journal of Listening, 32(3), 150-162.
16. Most, T., & Aviner, C. (2009). Auditory, visual, and auditory–visual perception of emotions by individuals with cochlear implants, hearing aids, and normal hearing. Journal of deaf studies and deaf education, 14(4), 449-464.
17. Paquette S, Ahmed GD, Goffi-Gomez MV, Hoshino AC, Peretz I, Lehmann A. Musical and vocal emotion perception for cochlear implants users. Hearing Research. 2018 Dec 1;370:272-82.
18. Chatterjee, M., Zion, D. J., Deroche, M. L., Burianek, B. A., Limb, C. J., Goren, A. P., ... & Christensen, J. A. (2015). Voice emotion recognition by cochlear-implanted children and their normally-hearing peers. Hearing research, 322, 151-162.
19. Jiam NT, Caldwell M, Deroche ML, Chatterjee M, Limb CJ. Voice emotion perception and production in cochlear implant users. Hearing Research. 2017 Sep 1;352:30-9.
20. Peelle JE. Listening effort: How the cognitive consequences of acoustic challenge are reflected in brain and behavior. Ear and hearing. 2018 Feb 23;39(2):204.