Receiver Operator Curve (ROC) and Area under the curve (AUC)
The ROC curves with AUCs of the five best performing tools for each dataset are provided in Figure 2a-c. For ABCA4 NCSS variants those tools were SpliceAI, DSSP, S-CAP, Spidex and MaxEntScan. The best performing tools for the ABCA4 DI variants were SpliceAI, SpliceRover, MaxEntScan, NNSplice and Alamut 3/4. For MYBPC3 NCSS variants, SpliceSiteFinder-like, Alamut 3/4 MaxEntScan, NNSplice and GeneSplicer achieved the highest AUC. ROC curves and AUCs including all tools are provided in Supplementary Figure 1 and Supplementary Table 3.
Figure 3 groups the different tools for each dataset into categories based on their AUC value. The AUC of MMSplice was always lower than 0.59, which makes it one of the worst performing tools. In addition, CADD showed a low performance with an AUC of 0.64 or lower. SpliceRover, GeneSplicer, SPIDEX and S-CAP performed on average with an AUC between 0.60 and 0.79. Other tools with consistent, average performance and an AUC between 0.7 and 0.81 were the Alamut 3/4 consensus, MaxEntScan and NNSPLICE. SpliceSiteFinder-like, MTSplice and DSSP showed variable performance with a high AUC (0.70-0.79) for one dataset and a low AUC (0.5-0.59) for another dataset. SpliceAI was the best performing tool for ABCA4 variants with an AUC higher than 0.8, but achieved an AUC of only 0.72 for MYBPC3 variants.