Skip to Main content Skip to Navigation
New interface

Modélisation de la coarticulation multimodale : vers l'animation d'une tête parlante intelligible

Théo Biasutto-Lervat 1 
1 MULTISPEECH - Speech Modeling for Facilitating Oral-Based Communication
Inria Nancy - Grand Est, LORIA - NLPKD - Department of Natural Language Processing & Knowledge Discovery
Abstract : This thesis deals with neural network based coarticulation modeling, and aims to synchronize facial animation of a 3D talking head with speech. Predicting articulatory movements is not a trivial task, as it is well known that production of a phoneme is greatly affected by its phonetic context, a phoneme called coarticulation. We propose in this work a coarticulation model, i.e. a model able to predict spatial trajectories of articulators from speech. We rely on a sequential model, the recurrent neural networks, and more specifically the Gated Recurrent Units, which are able to consider the articulation dynamic as a central component of its modeling. Unfortunately, the typical amount of data in articulatory and audiovisual databases seems to be quite low for a deep learning approach. To overcome this difficulty, we propose to integrate articulatory knowledge into the networks during its initialization. The RNNs robustness allow uw to apply our coarticulation model to predict both face and tongue movements, in french and german for the face, and in english and german for the tongue. Evaluation has been conducted through objective measures of the trajectories, and through experiments to ensure a complete reach of critical articulatory targets. We also conducted a subjective evaluation to attest the perceptual quality of the predicted articulation once applied to our facial animation system. Finally, we analyzed the model after training to explore phonetic knowledges learned.
Complete list of metadata
Contributor : Thèses UL Connect in order to contact the contributor
Submitted on : Wednesday, April 21, 2021 - 9:32:40 AM
Last modification on : Saturday, June 25, 2022 - 7:45:48 PM
Long-term archiving on: : Thursday, July 22, 2021 - 6:18:47 PM


Files produced by the author(s)


  • HAL Id : tel-03203815, version 1


Théo Biasutto-Lervat. Modélisation de la coarticulation multimodale : vers l'animation d'une tête parlante intelligible. Intelligence artificielle [cs.AI]. Université de Lorraine, 2021. Français. ⟨NNT : 2021LORR0019⟩. ⟨tel-03203815⟩



Record views


Files downloads