Whilst the R-squared and adjusted R-squared measures observed in Table \ref{table:results_benchmarking} would suggest that the low-dimensional model provides a better fit, the associated F-ratio score and corresponding p-value suggests a lower significance than those values observed for the high-dimensional model. Conversely, the constant basis model does not appear to provide as good a fit to the expected scalar responses from the R-squared and adjusted R-squared values, but this is not surprising considering the more limited nature of models built on constant terms. Finally, the monomial basis system performs fractionally better on both the R-squared and adjusted R-squared measures whilst also achieving a comparable level of significance to the high-dimensional model. Consequently, from this benchmarking analysis it would appear that the high-dimensional and monomial basis system models are the most suitable candidates, but it is possible that the overall performance of the high-dimensional model could be further improved by sensitivity studies into the optimum number of B-splines to use in the regression fit.
To further validate the statistical significance of the four models considered here permutation testing is applied to count the proportion of generated F values that are larger than the F-statistic for each model (see section 9.5 of \cite{Ramsay_2009}). This involves repeatedly shuffling the expected mode classification labels versus the technology profiles being read (maintaining their original order) to see if it is still possible to fit the regression model to these reordered responses. In so doing, this test also creates a null distribution versus the qth quantile and observed F-statistic generated from the models themselves. The results of this analysis are shown in Fig. 19.