A procedure is described for extending classic receiver operator characteristic (ROC) curve analysis to discriminate three or more ordinal outcomes on a shared continuous measurement scale. The procedure comprises a two-stage, semiparametric approach combining conventional cumulative logit regression with a cumulative extension of ROC curve analysis. Performance with a three-level (ternary) ordinal outcome was evaluated under simulation, which included comparison of several criteria for selecting cutpoints to discriminate outcome levels: the Youden Index, Matthews Correlation Coefficient, Total Accuracy, and Markedness. Cutpoints computed from maximum likelihood regression parameters were also evaluated. The procedure performs as expected under a variety of simulated conditions: unbalanced data, proportional odds and non-proportional odds, and areas under the ROC curve (AUCs) from 0.70 to 0.95. Total Accuracy selected cutpoints with the least absolute percent-bias among the criteria compared, while the bias of parametric cutpoints was less than for Total Accuracy cutpoints and often negligible. The procedure was also applied to publicly available data related to computer imaging and biomarker exposure science, yielding good to excellent AUCs, as well as cutpoints with sensitivities and specificities of commensurate quality. Cumulative ROC curve analysis is readily implemented with available software, and the author provides the URL for his implementation for ternary ordinal outcomes in the SAS statistical software application.

Key Words: bioassay, cumulative logit regression, clinical chemistry, diagnostic test, Fieller's Method, machine learning, multinomial outcomes, ordinal outcomes, receiver operating characteristic curve

Disclaimers: The findings and conclusions in this report are those of the author and do not necessarily represent the official position of the Centers for Disease Control and Prevention. The author declares that he has no actual or potential competing financial interests.