What is it about?
Identifying inconsistencies in large and complex datasets is a significant challenge. Manually defining rules to detect every possible contradiction is often impractical or too costly. In this paper, we present a novel, probability-based metric that evaluates whether data records are semantically consistent. The key idea is that records with internal contradictions behave like anomalies and can thus be detected automatically. A major benefit of this approach is that it requires no manually defined rules and is well-suited for high-dimensional big data. The method was successfully validated using real-world production data in collaboration with a leading German car manufacturer.
Featured Image
Photo by Joshua Sortino on Unsplash
Why is it important?
We propose a novel, probability-based metric that provides interpretable metric values representing the probability that a record in a database is semantically consistent. Unlike existing approaches, our metric is not based on explicit rules, but on the equivalence between internal contradictions and anomalies within the data. Based on the isolation forest as a highly scalable anomaly detection method, it can handle high-dimensional big data. To account for the associated uncertainty, we combine the idea of the isolation forest with the Grubbs outlier test to assess the probability that a record is semantically consistent.
Read the Original
This page is a summary of: Anomaly-based Assessment of Semantic Consistency: Design and Evaluation of a Novel Probability-based Metric in Cooperation with a German Car Manufacturer, Journal of Data and Information Quality, June 2025, ACM (Association for Computing Machinery),
DOI: 10.1145/3732783.
You can read the full text:
Contributors
The following have contributed to this page







