Receiver operating characteristic (ROC) movies, universal ROC (UROC) curves, and coefficient of predictive ability (CPA)

Abstract

Abstract Throughout science and technology, receiver operating characteristic (ROC) curves and associated area under the curve ( $$\mathrm{AUC}$$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"><mml:mi>AUC</mml:mi></mml:math> ) measures constitute powerful tools for assessing the predictive abilities of features, markers and tests in binary classification problems. Despite its immense popularity, ROC analysis has been subject to a fundamental restriction, in that it applies to dichotomous (yes or no) outcomes only. Here we introduce ROC movies and universal ROC (UROC) curves that apply to just any linearly ordered outcome, along with an associated coefficient of predictive ability ( $${\mathrm{CPA}}$$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"><mml:mi>CPA</mml:mi></mml:math> ) measure. $${\mathrm{CPA}}$$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"><mml:mi>CPA</mml:mi></mml:math> equals the area under the UROC curve, and admits appealing interpretations in terms of probabilities and rank based covariances. For binary outcomes $${\mathrm{CPA}}$$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"><mml:mi>CPA</mml:mi></mml:math> equals $$\mathrm{AUC}$$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"><mml:mi>AUC</mml:mi></mml:math> , and for pairwise distinct outcomes $${\mathrm{CPA}}$$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"><mml:mi>CPA</mml:mi></mml:math> relates linearly to Spearman’s coefficient, in the same way that the C index relates linearly to Kendall’s coefficient. ROC movies, UROC curves, and $${\mathrm{CPA}}$$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"><mml:mi>CPA</mml:mi></mml:math> nest and generalize the tools of classical ROC analysis, and are bound to supersede them in a wealth of applications. Their usage is illustrated in data examples from biomedicine and meteorology, where rank based measures yield new insights in the WeatherBench comparison of the predictive performance of convolutional neural networks and physical-numerical models for weather prediction.

Ähnliche Arbeiten