HAL will be down for maintenance from Friday, June 10 at 4pm through Monday, June 13 at 9am. More information
Skip to Main content Skip to Navigation

Impact du Bruit d'Annotation sur l'Évaluation de Classifieurs

Ilias Benjelloun 1
1 SYNALP - Natural Language Processing : representations, inference and semantics
LORIA - NLPKD - Department of Natural Language Processing & Knowledge Discovery
Abstract : Classical performance evaluation metrics used in the domains of Classification and Machine Learning are based on the assumption that the reference data used for validation and comparison are not error-free. Recent work has shown that this hypothesis is almost never guaranteed. In order to evaluate and compare different classification methods objectively and with verifiable levels of confidence, we have started to explore empirical statistical techniques which are promising and which make it possible to avoid of the constraint of perfect data. These techniques now need to be formally and theoretically validated on the one hand and extended to classifiers that are more generic. Our experiments validated the approach on binary classifiers; they should be generalized to classifiers involving any number of classes. This project aims to revisit the whole performance evaluation process by studying and developing statistical tools expressing a 'confidence' in classification measures resulting from evaluation or benchmarking campaigns. The question to be answered is the following: given the response of n algorithms to a set of reference data, what confidence can be attached to the resulting ranking, given an estimated error rate of less than ε ? Or, conversely, from what error rate on the reference data can we consider, with a confidence rate of τ that the obtained classification cannot be guaranteed ? These questions can be expressed with several probabilistic formulations. For example, considering the data as realizations of a random variable (whose law may belong to a given parametric model), we can study the responses of the n algorithms as a function of this random variable. This will make it possible to compute the probability of having a given ranking of these answers and to test if the ranking is reliable. Assuming that the error distribution belongs to a parametric model, Bayesian statistical tools can be used to study the posterior distribution of the parameters in the light of the responses of the algorithms.
Document type :
Complete list of metadata

Contributor : Thèses Ul Connect in order to contact the contributor
Submitted on : Thursday, February 10, 2022 - 10:30:47 AM
Last modification on : Saturday, February 12, 2022 - 3:37:53 AM
Long-term archiving on: : Wednesday, May 11, 2022 - 6:20:10 PM


Files produced by the author(s)


  • HAL Id : tel-03564063, version 1


Ilias Benjelloun. Impact du Bruit d'Annotation sur l'Évaluation de Classifieurs. Informatique [cs]. Université de Lorraine, 2021. Français. ⟨NNT : 2021LORR0267⟩. ⟨tel-03564063⟩



Record views


Files downloads