Receiver Operator Curve (ROC) and Area under the curve (AUC)
The ROC curves with AUCs of the five best performing tools for each
dataset are provided in Figure 2a-c. For ABCA4 NCSS variants
those tools were SpliceAI, DSSP, S-CAP, Spidex and MaxEntScan. The best
performing tools for the ABCA4 DI variants were SpliceAI,
SpliceRover, MaxEntScan, NNSplice and Alamut 3/4. For MYBPC3 NCSS
variants, SpliceSiteFinder-like, Alamut 3/4 MaxEntScan, NNSplice and
GeneSplicer achieved the highest AUC. ROC curves and AUCs including all
tools are provided in Supplementary Figure 1 and Supplementary Table 3.
Figure 3 groups the different tools for each dataset into categories
based on their AUC value. The AUC of MMSplice was always lower than
0.59, which makes it one of the worst performing tools. In addition,
CADD showed a low performance with an AUC of 0.64 or lower. SpliceRover,
GeneSplicer, SPIDEX and S-CAP performed on average with an AUC between
0.60 and 0.79. Other tools with consistent, average performance and an
AUC between 0.7 and 0.81 were the Alamut 3/4 consensus, MaxEntScan and
NNSPLICE. SpliceSiteFinder-like, MTSplice and DSSP showed variable
performance with a high AUC (0.70-0.79) for one dataset and a low AUC
(0.5-0.59) for another dataset. SpliceAI was the best performing tool
for ABCA4 variants with an AUC higher than 0.8, but achieved an
AUC of only 0.72 for MYBPC3 variants.