loading page

A new definition of statistical significance
  • Thomas F Heston
Thomas F Heston
Elson S Floyd College of Medicine, Washington State University, Spokane, Washington USA
Author Profile

Abstract

The classical definition of statistical significance is p <= 0.05, meaning a 1/20 chance the test statistic found is due to normal variation of the null hypothesis. This definition of statistical significance does not represent the likelihood that the alternative hypothesis is true. Hypothesis testing can be evaluated using a 2x2 table (shown below). Box "a" = true positives: p <= 0.05 and the alternative hypothesis is true. This is the study's power. A rule of thumb is that study power should be at least 80% (80% of the time the statistical test is positive when the alternative hypothesis is true). Therefore a = 0.80. Box "b" = false-positives: p <= 0.05 but the alternative hypothesis is false. By definition, when p = 0.05 the test statistic has a 5% probability of occurring by chance when the null hypothesis is true. Therefore, b = 0.05. Box "c" = false-negatives: p >= 0.05 but the alternative hypothesis is true. This occurs 20% of the time when the study's power is 80%. Therefore, c = 0.20. Box "d" = true-negatives: p >= 0.05 and the null hypothesis is true. This occurs 95% of the time when p <= 0.05. Therefore, d = 0.95. From this table we derive: Sensitivity = power = a/(a+c) = 80%. Specificity = (1-p) = d/(b+d) = 95%. Positive predictive value = power/(power + p-value) = a/(a+b) = 94%. Negative predictive value = d/(c+d) = 83%. The classical definition of statistical significance is (1-specificity) and does not take power into consideration. The proposed new definition of statistical significance is when the positive predictive value of a test statistic is 95% or greater. To arrive at this, the cut-off p-value representing statistical significance needs to be corrected for study power so that 0.05 > (p-value)/(p-value + power). To achieve a 95% predictive confidence,it can be derived that statistical significance is a p-value <= power / 19.