Selecting optimal models using sequential approaches
Out of the 11 threshold dependent omission rates Maxent produces in its output we used “10th percentile training presence test omission” (hereafter ‘percentile OR’) and “balance training omission, predicted area and threshold values test omission” (hereafter ‘balance OR’) for the sequential model selection approaches. We chose percentile OR (Radosavljevic and Anderson, 2014; Galante et al. 2018) over the “minimum training presence test omission” (Shcheglovitova and Anderson, 2013; Radosavljevic and Anderson, 2014) since the latter is more sensitive to extreme localities and over predicts when calibration localities are many (Radosavljevic and Anderson, 2014). We used balance OR to assess utility of a new thresholding rule and its OR in selecting optimal model. Through different sequential combinations of the two ORs, AUCTEST and AUCDIFF we formulated four sequential approaches. They were: (i) sequential combination of percentile OR followed by AUCTEST (hereafter ORTEST_PER), (ii) sequential combination of balance OR followed by AUCTEST (hereafter ORTEST_BAL), (iii) sequential combination of percentile OR followed by AUCDIFF and then by AUCTEST (hereafter AUCDIFF_PER), and (iv) sequential combination of balance OR followed by AUCDIFF and then by AUCTEST (hereafter AUCDIFF_BAL) approaches.
We used composite models instead of the jackknife iterations (Galante et al., 2018) for each RM-FC combination to select the optimal model. However, Maxent averages all the jackknife iterations to produce the composite model irrespective of whether some individual jackknife models have good model discrimination (AUCTEST>.5), marginal discrimination (AUCTEST<.5) or no discrimination at all (AUCTEST=.5) (Figure S1). When the composite models are comprised of jackknife models with no discrimination they would have lower average ORs since Maxent assigns zero OR to the models with no discriminatory power, and thereby favours these as optimal models. Therefore, we first sorted composite models into four hierarchical groups beginning with (i) the composite models with all jackknife iterations with AUCTEST>.5, (ii) followed by ones with some jackknife iteration models with AUCTEST<.5, (iii) then with some jackknife iteration models with AUCTEST=.5 and (iv) ended with composite models with all their jackknife iteration models having AUCTEST=.5.
Following the above hierarchical groups, we then ranked the ORs, AUCDIFF and AUCTEST of the composite models. We accorded the highest rank to the models with the lowest OR since ORs higher than the theoretically expected value indicate overfitting (Radosavljevic and Anderson 2014). Similarly, we accorded the highest rank to the models with the lowest AUCDIFFsince less overfitting models are expected to have lower AUCDIFF (Warren and Seifert 2011, Radosavljevic and Anderson 2014). Here, we also considered negative AUCDIFF as equal to zero, the lowest AUCDIFF for model selection (Muscarella et al. 2014), though we used raw values for general analysis. For the AUCTEST we accorded the highest rank to the models with the highest AUCTEST since higher AUCTESTmeans better model performance or discriminatory ability (Radosavljevic and Anderson, 2014).
Once thus ranked, we followed the steps outlined in Figure 1. We chose the composite model or subset of composite models with the highest OR rank (corresponding to Step 1 of Figure 1). Since we used OR as the first criteria to select the optimal models if only a single composite model had the highest OR rank (i.e., the lowest ORs among the models) we considered it the optimal model for both ORTEST and AUCDIFF approaches (Figure 1). If Step 1 resulted in a subset of composite models we chose either a composite model or subset of models with the highest AUCTEST rank for the two ORTEST approaches (corresponding to Step 2b, Figure 1) (Shcheglovitova and Anderson 2013, Galante et al. 2018). Whereas, for AUCDIFF approaches we chose the model or models with the best ranked AUCDIFF (corresponding to Step 2a, Figure 1) followed by Step 2b (Radosavljevic and Anderson, 2014) depending on the outcome of Step 2a (Figure 1). After Step 2b, depending on the outcome, we followed Steps 3 to 5 for both the ORTEST and AUCDIFF approaches (Figure 1). In Step 3 we chose the models with the lowest average number of parameters since models with lower numbers of parameters are considered less complex and better models (Galante et al. 2018). We derived the average number of parameters for each candidate composite optimal model by dividing the sum of the number of parameters with non-zero lambda coefficients for each individual model extracted from the LAMBDA text file (Galante et al. 2018) by the number of iterations used for building SDM since Maxent does not provide directly the average number of parameters in the result for the composite models unlike it does for the threshold values and ORs. Further, when multiple optimal models had equal average number of parameters, we then chose models with the lower average lambda coefficients obtained by dividing the sum of the absolute value of lambda coefficients by the total number of parameters. However, for some species multiple optimal models with same RM values but different FCs had equal average numbers of parameters as well as the average absolute lambda coefficients. In such cases we used the composite models with simpler FC as the final optimal model since lower FCs are considered better for species with smaller occurrence in Maxent (Shcheglovitova and Anderson 2013).