Testing Statistical Significance when Testing with Multiple Datasets in Natural Language Processing

What is it about?

It has become standard to evaluate Natural Language Processing (NLP) algorithms on multiple datasets in order to ensure a consistent performance across varied setups. When doing so, one has to change the statistical analysis of the results to consider all tested hypotheses (one for each dataset). In this paper we explain how to perform such an analysis with a special consideration to NLP applications.

Featured Image

Why is it important?

It is crucial to perform a correct and valid statistical analysis in an empirical research area such as NLP. This paper goal is to ensure the researchers perform the statistical analysis in a valid way.