Résultats de recherche pour « Audiovisual Speech-In-Noise »

Especially in challenging listening conditions, listeners can benefit from the audiovisual nature of speech by using visual information. Yet there exists great inter-individual variability, not only in understanding speech in noise, but also in the benefit obtained from additional visual cues. First empirical evidence suggests that the ability to integrate auditory and visual input, i.e. audiovisual integration, is altered in hearing impairment and is, at the same time, relevant for audiovisual speech intelligibility. The distinct role of mild hearing loss on audiovisual integration and the significance of these changes for speech intelligibility, however, need further scrutiny. Thus, here we investigated differences in audiovisual integration capacities between elderly, normal-hearing and hearing-impaired individuals using two tests of audiovisual integration (sound-induced flash illusion, McGurk task). To explore whether potential differences in audiovisual integration are meaningful for natural speech intelligibility, we then linked audiovisual integration capacities to speech-in-noise recognition using an audiovisual speech-reception threshold test, expecting this to reflect a more realistic listening scenario. Our results indicate that audiovisual integration abilities are already altered in mild hearing impairment, while the magnitude and direction of the effect depend on the specific test used. At the same time, audiovisual integration capacities seem relevant for predicting audiovisual speech intelligibility in noise, especially in those individuals with a hearing loss. We conclude that audiovisual integration abilities should therefore be considered for future predictions of speech recognition outcomes, which – in turn – should be assessed audiovisually, to account for the multisensory nature of speech and communication.

Successful communication requires that listeners not only identify speech, but do so while maintaining performance on other tasks, like remembering what a conversational partner said or paying attention while driving. This set of four experiments systematically evaluated how audiovisual speech—which reliably improves speech intelligibility—affects dual-task costs during speech perception (i.e., one facet of listening effort). Results indicated that audiovisual speech reduces dual-task costs in difficult listening conditions (those in which visual cues substantially benefit intelligibility), but may actually increase costs in easy conditions—a pattern of results that was internally replicated multiple times. This study also presents a novel dual-task paradigm specifically designed to facilitate conducting dual-task research remotely. Given the novelty of the task, this study includes psychometric experiments that establish positive and negative control, assess convergent validity, measure task sensitivity relative to a commonly-used dual-task paradigm, and generate performance curves across a range of listening conditions. Thus, in addition to evaluating the effects of audiovisual speech across a wide range of background noise levels, this study enables other researchers to address theoretical questions related to the cognitive mechanisms supporting speech processing beyond the specific issues addressed here and without being limited to in-person research.

Successful communication requires that listeners not only identify speech, but do so while maintaining performance on other tasks, like remembering what a conversational partner said or paying attention while driving. This set of four experiments systematically evaluated how audiovisual speech—which reliably improves speech intelligibility—affects dual-task costs during speech perception (i.e., one facet of listening effort). Results indicated that audiovisual speech reduces dual-task costs in difficult listening conditions (those in which visual cues substantially benefit intelligibility), but may actually increase costs in easy conditions—a pattern of results that was internally replicated multiple times. This study also presents a novel dual-task paradigm specifically designed to facilitate conducting dual-task research remotely. Given the novelty of the task, this study includes psychometric experiments that establish positive and negative control, assess convergent validity, measure task sensitivity relative to a commonly-used dual-task paradigm, and generate performance curves across a range of listening conditions. Thus, in addition to evaluating the effects of audiovisual speech across a wide range of background noise levels, this study enables other researchers to address theoretical questions related to the cognitive mechanisms supporting speech processing beyond the specific issues addressed here and without being limited to in-person research.

Face-to-face communication involves both auditory speech and visual bodily information (e.g. visible speech, gestures), and often occurs in noisy settings. In such settings, meaningful hand gestures have shown to facilitate speech comprehension. However, individual differences exist in how much listeners benefit from these gestures. One factor that could impact how a listener processes and integrates audiovisual information is their experience with environmental noise. For example, growing up or going to school in high-noise environments (such as in the proximity of frequent heavy traffic or airports) has been shown to influence language abilities, such as reading comprehension, and speech perception. In the current study, we investigated whether a listener’s experience with environmental noise affects how much they can benefit from gestures in adverse listening conditions. In an online experiment, 40 participants watched video clips of an actress articulating an action verb, accompanied by an iconic gesture or none, in clear or degraded speech. Participants identified the verb and rated the noise levels of their current and previous living environments based on their geographical locations. We hypothesized that high-noise environments would hinder task performance and audiovisual gain. The results indicated that the amount of environmental noise, in current and previous living environments, influenced how much participants benefited from gestures during degraded speech comprehension. Individuals from lower-noise environments benefitted more from visual semantic information than those from high-noise environments. Individuals from higher-noise environments performed better overall, and benefitted less from gestures, potentially because they developed adaptive mechanisms to cope with auditory degradation.

Abstract Seeing a speaker’s face benefits speech comprehension, especially in challenging listening conditions. This perceptual benefit is thought to stem from the neural integration of visual and auditory speech at multiple stages of processing, whereby movement of a speaker’s face provides temporal cues to auditory cortex, and articulatory information from the speaker’s mouth can aid recognizing specific linguistic units (e.g., phonemes, syllables). However it remains unclear how the integration of these cues varies as a function of listening conditions. Here we sought to provide insight on these questions by examining EEG responses to natural audiovisual, audio, and visual speech in quiet and in noise. Specifically, we represented our speech stimuli in terms of their spectrograms and their phonetic features, and then quantified the strength of the encoding of those features in the EEG using canonical correlation analysis. The encoding of both spectrotemporal and phonetic features was shown to be more robust in audiovisual speech responses then what would have been expected from the summation of the audio and visual speech responses, consistent with the literature on multisensory integration. Furthermore, the strength of this multisensory enhancement was more pronounced at the level of phonetic processing for speech in noise relative to speech in quiet, indicating that listeners rely more on articulatory details from visual speech in challenging listening conditions. These findings support the notion that the integration of audio and visual speech is a flexible, multistage process that adapts to optimize comprehension based on the current listening conditions. Significance Statement During conversation, visual cues impact our perception of speech. Integration of auditory and visual speech is thought to occur at multiple stages of speech processing and vary flexibly depending on the listening conditions. Here we examine audiovisual integration at two stages of speech processing using the speech spectrogram and a phonetic representation, and test how audiovisual integration adapts to degraded listening conditions. We find significant integration at both of these stages regardless of listening conditions, and when the speech is noisy, we find enhanced integration at the phonetic stage of processing. These findings provide support for the multistage integration framework and demonstrate its flexibility in terms of a greater reliance on visual articulatory information in challenging listening conditions.

Static and dynamic visual speech cues contribute to audiovisual (AV) speech recognition in noise. Static cues (e.g., “lipreading”) provide complementary information that enables perceivers to ascertain ambiguous acoustic-phonetic content. The role of dynamic cues is less clear, but one suggestion is that temporal covariation between facial motion trajectories and the speech envelope enables perceivers to recover a more robust representation of the time-varying acoustic signal. Modeling studies show this is computationally feasible, though it has not been confirmed experimentally. We conducted two experiments to determine whether AV speech recognition depends on the magnitude of cross-sensory temporal coherence (AVC). In Experiment 1, sentence-keyword recognition in steady-state noise (SSN) was assessed across a range of signal-to-noise ratios (SNRs) for auditory and AV speech. The auditory signal was unprocessed or filtered to remove 3-7 Hz temporal modulations. Filtering severely reduced AVC (magnitude-squared coherence of lip trajectories with cochlear-narrowband speech envelopes), but did not reduce the magnitude of the AV advantage (AV > A; ~ 4 dB). This did not depend on the presence of static cues, manipulated via facial blurring. Experiment 2 assessed AV speech recognition in SSN at a fixed SNR (-10.5 dB) for subsets of Exp. 1 stimuli with naturally high or low AVC. A small effect (~ 5% correct; high-AVC > low-AVC) was observed. A computational model of AV speech intelligibility based on AVC yielded good overall predictions of performance, but over-predicted the differential effects of AVC. These results suggest the role and/or computational characterization of AVC must be re-conceptualized.