Skip to Main content Skip to Navigation
Theses

Modélisation de la coarticulation multimodale : vers l'animation d'une tête parlante intelligible

Théo Biasutto-Lervat 1
1 MULTISPEECH - Speech Modeling for Facilitating Oral-Based Communication
Inria Nancy - Grand Est, LORIA - NLPKD - Department of Natural Language Processing & Knowledge Discovery
Abstract : This thesis deals with neural network based coarticulation modeling, and aims to synchronize facial animation of a 3D talking head with speech. Predicting articulatory movements is not a trivial task, as it is well known that production of a phoneme is greatly affected by its phonetic context, a phoneme called coarticulation. We propose in this work a coarticulation model, i.e. a model able to predict spatial trajectories of articulators from speech. We rely on a sequential model, the recurrent neural networks, and more specifically the Gated Recurrent Units, which are able to consider the articulation dynamic as a central component of its modeling. Unfortunately, the typical amount of data in articulatory and audiovisual databases seems to be quite low for a deep learning approach. To overcome this difficulty, we propose to integrate articulatory knowledge into the networks during its initialization. The RNNs robustness allow uw to apply our coarticulation model to predict both face and tongue movements, in french and german for the face, and in english and german for the tongue. Evaluation has been conducted through objective measures of the trajectories, and through experiments to ensure a complete reach of critical articulatory targets. We also conducted a subjective evaluation to attest the perceptual quality of the predicted articulation once applied to our facial animation system. Finally, we analyzed the model after training to explore phonetic knowledges learned.
Complete list of metadata

https://hal.univ-lorraine.fr/tel-03203815
Contributor : Thèses Ul <>
Submitted on : Wednesday, April 21, 2021 - 9:32:40 AM
Last modification on : Thursday, April 22, 2021 - 3:29:09 AM
Long-term archiving on: : Thursday, July 22, 2021 - 6:18:47 PM

File

DDOC_T_2021_0019_BIASUTTO_LERV...
Files produced by the author(s)

Identifiers

  • HAL Id : tel-03203815, version 1

Citation

Théo Biasutto-Lervat. Modélisation de la coarticulation multimodale : vers l'animation d'une tête parlante intelligible. Intelligence artificielle [cs.AI]. Université de Lorraine, 2021. Français. ⟨NNT : 2021LORR0019⟩. ⟨tel-03203815⟩

Share

Metrics

Record views

90

Files downloads

79