Skip to Main content Skip to Navigation

Localisation et rehaussement de sources de parole au format Ambisonique : analyse de scènes sonores pour faciliter la commande vocale

Lauréline Perotin 1, 2
1 MULTISPEECH - Speech Modeling for Facilitating Oral-Based Communication
Inria Nancy - Grand Est, LORIA - NLPKD - Department of Natural Language Processing & Knowledge Discovery
Abstract : This work was conducted in the fast-growing context of hands-free voice command. In domestic environments, smart devices are usually laid in a fixed position, while the human speaker gives orders from anywhere, not necessarily next to the device, or nor even facing it. This adds difficulties compared to the problem of near-field voice command (typically for mobile phones) : strong reverberation, early reflections on furniture around the device, and surrounding noises can degrade the signal. Moreover, other speakers may interfere, which make the understanding of the target speaker quite difficult. In order to facilitate speech recognition in such adverse conditions, several preprocessing methods are introduced here. We use a spatialized audio format suitable for audio scene analysis : the Ambisonic format. We first propose a sound source localization method that relies on a convolutional and recurrent neural network. We define an input feature vector inspired by the acoustic intensity vector which improves the localization performance, in particular in real conditions involving several speakers and a microphone array laid on a table. We exploit the visualization technique called layerwise relevance propagation (LRP) to highlight the time-frequency zones that are correlate positively with the network output. This analysis is of paramount importance to establish the validity of a neural network. In addition, it shows that the neural network essentially relies on time-frequency zones where direct sound dominates reverberation and background noise. We then present a method to enhance the voice of the main speaker and ease its recognition. We adopt a mask-based beamforming framework based on a time-frequency mask estimated by a neural network. To deal with the situation of multiple speakers with similar loudness, we first use a wideband beamformer to enhance the target speaker thanks to the associated localization information. We show that this additional information is not enough for the network when two speakers are close to each other. However, if we also give an enhanced version of the interfering speaker as input to the network, it returns much better masks. The filters generated from those masks greatly improve speech recognition performance. We evaluate this algorithm in various environments, including real ones, with a black-box automatic speech recognition system. Finally, we combine the proposed localization and enhancement systems and evaluate the robustness of the latter to localization errors in real environments.
Document type :
Complete list of metadata

Cited literature [193 references]  Display  Hide  Download
Contributor : Thèses Ul <>
Submitted on : Wednesday, December 4, 2019 - 12:13:51 PM
Last modification on : Tuesday, February 2, 2021 - 2:26:02 PM
Long-term archiving on: : Thursday, March 5, 2020 - 3:31:44 PM


Files produced by the author(s)


  • HAL Id : tel-02393258, version 1


Lauréline Perotin. Localisation et rehaussement de sources de parole au format Ambisonique : analyse de scènes sonores pour faciliter la commande vocale. Traitement du signal et de l'image [eess.SP]. Université de Lorraine, 2019. Français. ⟨NNT : 2019LORR0124⟩. ⟨tel-02393258⟩



Record views


Files downloads