Enhancing Statistical Editing in Official Statistics: Validation and Detection of Faulty Data through Artificial Intelligence - Softcover

Salamanca Dávila, Carlos Andres

Softcover

ISBN 10: 3389198377 ISBN 13: 9783389198377

Verlag: GRIN Verlag, 2026

Inhaltsangabe

Master's Thesis from the year 2025 in the subject Mathematics - Applied Mathematics, grade: 1,0, Justus-Liebig-University Giessen, language: English, abstract: This thesis investigates whether machine learning (ML) - i.e., statistical learning - can provide a scalable alternative for detecting errors in official statistics. Following a mathematical framework, we formalize ML classifiers from four algorithmic families for their deployment in the detection task: Random Forests, Boosting Methods, Support Vector Machines, and Neural Networks - and train and evaluate them on data provided by the German Federal Statistical Office. To address the scarcity of labeled data, we implement a systematic noise injection strategy that exposes models to controlled error rates and rare violation types. Furthermore, we extend the investigation beyond performance evaluation to examine the robustness of the models when the underlying stochastic mechanisms behind faulty data change. The results demonstrate that especially tree-based ensemble methods can achieve ROC- and PR-AUC scores exceeding 0.98, nearly matching deterministic validation and scaling linearly with computational efficiency of O(n). However, an important finding emerges: while classifiers remain robust when error probabilities shift from uniform randomness to covariate-dependent patterns, performance degrades when errors depend on the variable's own values. In particular, neural networks experience a decline in PR-AUC from 0.9 to 0.7. This work addresses a gap in the literature by providing the first empirical evidence of the impact of stochastic error-generation mechanisms on ML validation performance. The findings suggest that although the deployment of artificial intelligence in data validation offers an efficient alternative to deterministic validation, there are considerations related to the stochastic mechanism behind the data that must be acknowledged, as they can affect model performance.

Details zur Ausgabe

• Verlag: GRIN Verlag
• Erscheinungsdatum: 2026
• Sprache: Englisch
• ISBN 10: 3389198377
• ISBN 13: 9783389198377
• Einband: Taschenbuch
• Auflage: 1
• Anzahl der Seiten: 80
• Kontakt zum Hersteller: GRIN Publishing GmbH; info@grin.com; ; Waltherstraße 23; München; 80337; Deutschland