Evaluating Syntactic Annotation of Ancient Languages

What is it about?

In this paper, we introduce an extended version of the Vedic Treebank (VTB, Hellwig et al. 2020) which comes along with revisited and extended annotation guidelines. In order to assess the quality of our annotations as well as the usability and limits of the guidelines, we performed an inter-annotator agreement test. The results show that agreement between annotators is hampered by various factors, most prominently by insufficient understanding of the content because of the cultural and temporal gap and incomplete knowledge of Vedic grammar. An in-depth discussion of disagreeing annotations demonstrates that the setup of the workflow, too, has a major influence on the inter-annotator agreement. We suggest some measures that can help increase the transparency and annotation consistency according to current knowledge when annotating Vedic Sanskrit, or ancient language varieties in general.

Featured Image

Photo by Mika Baumeister on Unsplash

Photo by Mika Baumeister on Unsplash