2.4. Permutation test for correlation coefficients
The classic approach to multiple hypothesis testing, for example, the
Bonferroni correction, is very conservative in some cases, increasing
the risk of Type 2 errors. In order to adequately test the six
correlations for which we formulated hypotheses, we applied a
simultaneous permutation test as described in the following. Let us
denote X1 (N250 targets),
X2 (N250 low-distinct non-targets),
X3 (N250 high-distinct non-targets),
X4 (recognition accuracy), X5(recognition speed). We were interested in six correlation coefficients\(\rho_{i}\) i = 1,…,6 (between each of the three
variables X 1, X 2,X 3 and each of the two variables:X 4, X 5). The problem can
be addressed by verifying the hypothesis:
\(H_{0}:\ \operatorname{}{|\rho_{i}}|=\ 0\)
\(H_{1}:\ \operatorname{|}\rho_{i}|>\ 0\)
Hence, we took the maximum from the (absolute) values of the six
mentioned correlation coefficients as a test statistic:
\(T=\operatorname{}{|r}_{i}|\) i = 1,…, 6
where ri is a sample Pearson correlation coefficient. To
perform the permutation test we calculated the value T0of statistics for our tested sample from the experiment. Assuming that
the null hypothesis is true, we performed N = 10,000 random
permutations (shuffles) of the data in each of the five variables,
calculated the value of the test statistic for each permutation and
created the empirical distribution of Tj ,j =1, 2,…,N .
If the calculated ASL (Achieved Significance Level) (Good, 1994):
\begin{equation}
\text{ASL}\approx\frac{\text{card}\{T_{i}\geq T_{0})}{N}\nonumber \\
\end{equation}
is less than the significance level p = 0.05, the null hypothesis
is rejected. Please note that apart from being less conservative than
Bonferroni correction, the permutation test of statistical significance
has the advantage of not assuming normal distribution of the data.