A signal detection analysis of “dramatic effects” based
on recognition heuristics
Recognition is a natural mechanism for making inferences and solving
problems – if the object or phenomenon is not recognized, further
ascertainment and reasoning processes cannot proceed.34 One strategy that relies on using recognition to
make inferences is called the recognition
heuristic.35 Recognition heuristic has been
demonstrated to provide accurate answers in a wide range of
circumstances, particularly when information is limited and uncertain.
It is considered to have a special status in our cognitive capabilities
because, as explained, if the object is not recognized then it becomes
impossible to draw any inferences.35 Therefore, if the
regulators do not recognize effect size as a criterion for making
decisions on whether to request further RCTs, then effect size would not
play a role in their decision-making. However, as discussed, we have
demonstrated ecological validity between effect size and decisions
whether to require further testing in RCTs: larger effect sizes were
associated with a greater likelihood of approval based on nonrandomized
data. 15,16 Thus, the magnitude of effect size serves
as a recognition heuristic related to the decision to approve drugs
based on non-randomized studies. Nonetheless, the specific decision will
depend on beliefs (stemming from familiarity) that the effect size
exceeding a certain threshold (T ) (e.g., RR>2, 5,
10) is sufficient to obviate testing in further RCTs.
Use of recognition heuristics, like any other decision rule, may result
in correct and incorrect inferences. 35 In turn,
analyzing the proportion of correct inferences based on recognition
heuristics lends itself to inquiry within a framework of
SDT.12,36 SDT resides on the notion that the two
possible events (signal , e.g. treatment effect is “true”),
and noise , e.g. treatment effect is not “true”) have
overlapping distributions on a given observation scale. Each of these
distributions is further divided into two possible outcomes, which are
determined by setting a decision criterion. The criterion divides the
signal distribution into true positives (hits, or sensitivity) and false
negatives (misses). The noise distribution is composed of true negatives
(correct rejections, or specificity) and false positives,
respectively.37 To assess the accuracy of recognition
heuristic of a continuous variable such as effect size, we assume that
judges have a criterion set at one point along the possible values of
their prior beliefs, which in our case corresponds to treatment effect
size, \(TE=\operatorname{}{(OR}\)). If the TE exceeds the given
threshold (T ) consistent with the judges prior beliefs, the rule
to activate recognition heuristic can formally be stated as:36
\begin{equation}
If\ TE=\ \operatorname{}{OR\geq T}\ ,\ then\ "Approve\ without\ further\ testing\ in\ RCTs"\ \nonumber \\
\end{equation}\begin{equation}
If\ TE=\ \operatorname{}{OR<T},\ then\ "\ Approve\ with\ request\ for\ further\ testing\ in\ RCTs"\nonumber \\
\end{equation}Using the previously defined frameworks for integrating heuristic
decision-making with SDT12,36, for each possible
cutoff value of \(TE=\operatorname{}{(OR})\), we can calculate
standard SDT statistics36,37 including sensitivity,
specificity, overall accuracy, positive predictive value (PPV) (i.e.recognition validity ) and d’ (discriminability). In turn,
we define the optimal recognition heuristic as the maximum TE
threshold for the largest d’ value. [Note that d’(discriminability) represents the standardized distance between the
signal i.e., no further RCTs needed and noise (further RCTs required)
distributions and is defined as:
\begin{equation}
d^{\prime}=\ zHit\ \ zFA,\nonumber \\
\end{equation}where z Hit and z FA are the z -scores of the true
positive and the false alarm rate, respectively].