Functional Near-Infrared Spectroscopy
A Clinical Measure of Listening Effort?
When most people think of hearing loss, they think of a grandmother failing to notice the phone ring, or a grandfather asking you to “Say that again?” While there is nothing strictly wrong about this view of hearing loss, it is certainly incomplete. Age changes the auditory system in a variety of ways, but only some of these changes result in reduced sound intensityof the sort that would cause one to miss a phone ring. Instead, most of these changes result in reduced sound quality, degrading auditory signal such that, when it reaches the brain, its clarity is significantly compromised.1
With reduced sound quality and relatively intact sound intensity, listeners with age-related hearing loss can not only hear a phone ring but in most cases, they find speech to be highly intelligible. However, the difficulty arises from how effortful it is for them to comprehend speech. In many cases, the speech is not immediately clear, and it only becomes comprehensible through the recruitment of cognitive resources such as attention (to focus on the speech of interest) and working memory (to use context to fill in any unclear phrases).2In such cases, listeners are said to be exerting listening effort.
The case of listening effort demonstrates the complicated interplay of perception and cognition, as well as the limitations of considering hearing loss to be simply reduced sound intensity. Unfortunately, this misconception is also present in audiology clinics, where hearing loss is diagnosed based on low listening performance (e.g., elevated pure-tone thresholds) rather than high listening effort.3 Using this approach, it would not be possible to detect cases where a listener may be able to hear a sound or comprehend speech but needs to exert far more effort than the average person to do so. This effort can have cognitive or psychosocial consequences, such as memory impairment, fatigue, and social withdrawal.4
The above limitation is no fault of audiologists, who generally recognize the importance of listening effort and simply have no objective, reliable, and convenient method with which to measure it in a clinical setting. For instance, self-report measures of effort are not considered reliable, and while brain-based measures such as functional magnetic resonance imaging (fMRI) and electroencephalography show promise, they are inconvenient and usually incompatible with hearing aids.5 It is this problem—the lack of a suitable clinical measure of listening effort—that we at Ryerson University’s Science of Music, Auditory Research, and Technology (SMART) Lab have sought to address.
To measure listening effort, our lab has used a brain imaging method called functional near-infrared spectroscopy (fNIRS). Our fNIRS device (see Figure 1), which attaches to the forehead, includes light sources and light detectors. Light sources shine near-infrared light into the prefrontal cortex (PFC); some of this light is absorbed by blood with and without oxygen, while some of it is reflected back to the light detectors on the fNIRS device. Based on how much light is measured by the light detectors, the amount of oxygen in the PFC can be calculated.6 The amount of oxygen in the PFC indicates how active it is, which in turn indicates the extent to which cognitive resources, such as attention and working memory, are being recruited. In other words, the more oxygen in the PFC, the more effort is being exerted.
To assess whether fNIRS can measure listening effort, our lab has so far conducted three studies. In the first study,7 we aimed to assess whether fNIRS could measure effort generally, rather than effort specifically induced by sound quality. To do this, 16 younger adults with normal hearing completed a working memory task called the n-back. This tested participants’ memory for letters presented aurally through headphones. They completed four versions of this task (each about two minutes long), which became progressively more difficult: the 0-back (the easiest), 1-back, 2-back, and 3-back (the hardest). We found that, as the n-back became more difficult, the amount of oxygen in the PFC increased; in other words, participants exerted more effort (see Figure 2). This agrees with previous fMRI studies8 and shows that fNIRS can be used to measure effort, including during auditory tasks.
In a second study,9 we wanted to demonstrate the ability of fNIRS to measure listening effort in particular; that is, changes in effort corresponding to changes in listening difficulty, rather than changes in memory difficulty. To do this, 16 older adults with hearing loss who wore hearing aids completed the n-back, the same task as the first study. Once again, they completed four versions of this task, each increasing in difficulty (0-back, 1-back, 2-back, 3-back). However, this time, we added an additional factor: they completed each version of the task both without their hearing aids (at a sound level that was barely audible) and with their hearing aids. Previous studies have used fNIRS to measure listening effort,10 but this was the first study to measure it in older adults who use hearing aids.
As in the first study with younger adults, we expected that as task difficulty increased, the effort would as well. We also expected that effort would be reduced when participants wore their hearing aids, as hearing aids should make it much less cognitively demanding to comprehend the letters and therefore complete the task. As expected, we found that effort increased with task difficulty; however, the effort was not affected by whether hearing aids were used, at least not overall (see Figure 3). Nonetheless, hearing aids did reduce the effort for some listeners. In particular, the older a listener and the more severe their hearing loss, the more wearing hearing aids reduced effort. This result shows that fNIRS is not only able to measure effort in hearing aid users but that it is also sensitive to individual differences.
In a third study,11 we wanted to use fNIRS to measure listening effort in a more realistic task involving speech perception. To do this, 28 younger adults with normal hearing completed the Revised Speech Perception in Noise test,12 where they listened to sentences and repeated back the final words. Some of these final words were highly predictable (e.g., “It’s getting dark so light the lamp”), while others were not (e.g., “The girl should consider the flame"). Also, some sentences were presented among quiet background noise, while others were presented among loud background noise. This background noise compromised the clarity of the speech, similar to the effect of hearing loss.
We predicted that, when background noise was louder, the effort would increase, because participants would need to recruit more cognitive resources to extract and comprehend the final words. We also predicted that, when sentences were predictable, the effort would decrease, as the number of possible final words would be reduced and less processing would be needed to arrive at the correct interpretation. We found that, as expected, the effort was greater when background noise was louder (see Figure 4), as has been found in previous fMRI studies.13 However, the effort was not affected by whether the final words were predictable, despite predictability greatly improving intelligibility. This could be because using predictability is itself an effortful process, with cognitive resources required to make sense of the contextual information provided. A similar effect has been reported for visual speech cues: although seeing a person's mouth move while they speak improves speech intelligibility, it increases effort.14
Based on our lab’s early research, we are confident that fNIRS can serve as an objective, reliable, and convenient measure of listening effort across populations and tasks. As a result, we hope that fNIRS could one day be used to measure listening effort in a clinical setting. Such a clinical measure of listening effort could allow audiologists to gain a more complete picture of patients' hearing health, possibly identifying hearing difficulties that would go unnoticed by traditional, performance-based measures of hearing ability. Also, a clinical measure of listening effort could be used to assess the effectiveness of hearing aids. For instance, audiologists could adjust individual patients' hearing aid settings so that they not only maximize speech intelligibility but also minimize listening effort. This could also be of use to industrial researchers, with hearing aid companies able to develop and refine signal-processing algorithms based on the extent to which they reduce listening effort.
In sum, hearing loss is not just expressed as a failure to comprehend speech—it is also expressed as the increased effort required to comprehend speech. In the future, this cognitive side of hearing loss will need to be measured and considered to optimize hearing assessment and rehabilitation. fNIRS could be a part of this picture, although much more research—as well as the cooperation of researchers and audiologists—will be needed to bring fNIRS from the lab to the clinic.
Acknowledgements
This research has been supported by grants from the Natural Sciences and Engineering Research Council as well as Sonova Holding AG, both awarded to Dr. Frank Russo, director of the SMART Lab. I would also like to thank all the people who was involved in these projects: Dr. Frank Russo, Dr. Huiwen Goy, Dr. Kathy Pichora-Fuller, Rebecca Nurgitz, Fran Copelli, and Alberto Behar.
References
1. Plack CJ, Barker D, and Prendergast G. Perceptual consequences of “hidden” hearing loss. Trends Hear 2014;18, 1–11. doi:10.1177/2331216514550621
2. Alain C, Du Y, Bernstein LJ, et al. Listening under difficult conditions: An activation likelihood estimation meta-analysis. Human Brain Map 2018;39(7):2695–709. doi:10.1002/hbm.24031
3. Pichora-Fuller MK, Kramer SE, Eckert MA, et al. Hearing impairment and cognitive energy: The Framework for Understanding Effortful Listening (FUEL). Ear Hear 2016;37:5S–27S. doi:10.1097/aud.0000000000000312
4. Peelle JE. Listening effort: How the cognitive consequences of acoustic challenge are reflected in brain and behavior. Ear Hear 2018;39:204–14. doi:10.1097/aud.0000000000000494
5. Ohlenforst B, Zekveld AA, Jansma EP, et al. Effects of hearing impairment and hearing aid amplification on listening effort. Ear Hear 2017;38:267–81. doi:10.1097/aud.0000000000000396
6. Scholkmann F, Kleiser S, Metz AJ, et al. A review on continuous wave functional near-infrared spectroscopy and imaging instrumentation and methodology. NeuroImage 2014;85:6–27. doi:10.1016/j.neuroimage.2013.05.004
7. Rovetti J, Goy H, Nurgitz R, and Russo FA. (submitted). Comparing verbal working memory load in auditory and visual modalities using functional near-infrared spectroscopy.
8. Braver TS, Cohen JD, Nystrom LE, et al. A parametric study of prefrontal cortex involvement in human working memory. NeuroImage 1997;5(1):49–62. doi:10.1006/nimg.1996.0247
9. Rovetti J, Goy H, Pichora-Fuller MK, and Russo FA. Functional near-infrared spectroscopy as a measure of listening effort in older adults who use hearing aids. Trends Hear 2019;23:1–22, doi:10.1177/2331216519886722
10. Wijayasiri P, Hartley DEH, and Wiggins IM. Brain activity underlying the recovery of meaning from degraded speech: A functional near-infrared spectroscopy (fNIRS) study. Hear Res 2017;351:55–67. doi:10.1016/j.heares.2017.05.010
11. Rovetti J, Goy H, and Russo FA. (in preparation). Increased speech quality, but not predictability, decreases listening effort as measured using functional near-infrared spectroscopy.
12. Bilger RC, Nuetzel JM, Rabinowitz WM, and Rzeczkowski C. Standardization of a test of speech perception in noise. J Speech Lang Hear Res 1984;27:32–48. doi:10.1044/jshr.2701.32
13. Wild CJ, Yusuf A, Wilson DE, et al. Effortful listening: The processing of degraded speech depends critically on attention. J Neurosci 2012;32:14010–14021. doi:10.1523/jneurosci.1528-12.2012
14. Brown VA and Strand JF. About face: Seeing the talker improves spoken word recognition but increases listening effort. J Cognition 2019;2(1):1–20. doi: 10.31234/osf.io/m7b8q