The Local Correlation Function: an Interpretable Metric of the Degree of Clustering

What is it about?

In this work, we propose the local correlation function, a new metric for spatial analysis. Many research fields have spatial data that can be represented as a group of objects in space, such as stars in the sky, trees in the forest, or cells in tissue. Researchers working with such data often want to quantify the spatial arrangement of the objects. Are they clustered, randomly distributed, or dispersed? What is the degree of clustering or dispersion in the group of objects? Answering these questions may be a key to understanding the processes in nature that researchers study.  While classical methods of spatial statistics can detect clustering and dispersion, they fail to describe the degree of clustering in data. The local correlation function is designed to bridge this gap. Our function is uniformly bounded between -1 and 1, and it outputs -1 for maximal dispersion, 0 for spatial randomness, and 1 for maximal clustering. The notions of maximal dispersion and clustering have not been proposed before and are another important contribution of our work. The scale of our function and choice of the meaning of its minimum, midpoint, and maximum are inspired by the linear correlation coefficient, which is familiar to most scientists. This similarity makes the local correlation function intuitively clear to researchers with different levels of statistical expertise. We perform thorough theoretical and empirical analyses of the local correlation function. The theoretical analysis provides insight into the uncertainty of the function estimate for a random arrangement of points and the relationship of the function to other functions of spatial statistics. The empirical analysis shows that the ability of the local correlation function to detect clustering and dispersion is comparable to that of other spatial statistics. It also does not confuse the random arrangement of points with clustering or dispersion more often. Thus, there is no trade-off between the ease of interpretation of the local correlation function and its statistical performance. Lastly, we demonstrate how the local correlation function can be used in data analysis using two datasets from different fields. The first one comes from digital pathology and contains images of tumors in which multiple types of immune cells can be detected. The other consists of the positions of air pollution and waste generation facilities in the US states. We show how the local correlation function can compare the degree of clustering of different types of objects (cell types or industrial facilities) in a single sample and multiple samples. In this way, we highlight that our metric can be used for spatial analysis in different domains.

Featured Image

Photo by NASA Hubble Space Telescope on Unsplash

Photo by NASA Hubble Space Telescope on Unsplash

Why is it important?

Scientific discovery requires explainable and interpretable methods. The clarity  of the local correlation function and its known theoretical properties make it an attractive tool for researchers in many fields. For example, in cancer studies, researchers seek the factors influencing a patient's prognosis or response to treatment. Tumor images contain significant information about disease progression, and some of this information may be clinically relevant. With deep learning we can build a model that uses the tumor images to predict the patient's outcome with high accuracy. However, having a black box model with good predictive performance does not expand our knowledge about the specific cancer type as it is hard to determine the basis of the prediction. On the contrary, the local correlation function provides an interpretable summary of the spatial structure of the tumor: the degree of clustering of different cell types. This enables researchers to assess whether this spatial information has clinical value in different types of cancers. This advantage of the local correlation function is valuable in any field that requires not only accurate predictive models but also the ability to extract information about the studies processes.

Perspectives