Regression Expression Variation Analysis (REVA): A rank-based multi-dimensional measure of correlation

Sometimes a simple question arises: how does the distance between two samples in multivariate space compare to another scalar value associated with each sample. Here, inspired by the Kendall rank correlation coefficient, we propose theory for a non-parametric test to statistically test this association based on the neighbors principle implicit in any machine learning algorithm which says that samples with similar labels should be close to one another in feature space as well. Our test, REVA, is independent of the scale of the scalar data, and thus generalizable to any comparison of samples with both high-dimensional data and a scalar. We use U-statistic theory to derive the asymptotic distribution of the new correlation coefficient, developing additional large and finite sample properties along the way. To establish the admissibility of the REVA statistic, and explore the utility and limitations of our model, we compared it to the most widely used distance based correlation coefficient in a range of simulated conditions, demonstrating that REVA does not depend on an assumption of linearity, and is robust to high levels of noise, high dimensions, and the presence of outliers. We apply the resulting statistic to problems in cancer biology motivated by the model that cancer cells with more similar gene expression profiles to one another can be expected to have a more similar response to therapy.