Uncertainty estimation
Three types of uncertainty were identified during the development of the risk ranking framework. First, the uncertainty associated with the criteria weights assigned by the stakeholder-experts was characterized by a Beta-PERT distribution (Vose, 2008). For each pathogen, a total weighted risk score was obtained by adding each individual risk criterion score multiplied by values from the expert’s weight distribution for each criterion using the following equation (adapted from ECDC 2017):
\(Weighted\ risk\ score=\ \sum_{i=1}^{7}{Wi*S_{\text{ij}}}\) (2)
\begin{equation} W_{i}\sim\text{BetaPERT}(a,b,c)\nonumber \\ \end{equation}
where Sij is the score for pathogen j on criterion i as in Equation 1, and Wi is the probability distribution of the expert-designated weights for each criterion i . The Beta-PERT distribution was characterized by a minimum (a), most likely (b) and maximum value (c). Latin hypercube sampling (LHS) was performed in @Risk (Palisade, Inc.) to iterate over Equation 2 and sample stratified random numbers from each probability distribution of the expert-designated weights defined in the model (Vose, 2008). Significant correlations between input values were included in the model (Supporting Information). The LHS was repeated for 10,000 iterations to generate the final distribution of total weighted risk scores with mean and standard deviation values that accurately accounted for all possible weighted risk scores for a given set of parameters defined. Pairwise t-tests with a Bonferroni correction and nonparametric Kolmogorov-Smirnov tests (Arnold & Emerson, 2011) were applied to test for significant differences in mean total risk scores and overall total weighted risk distributions between pathogens, respectively.
The second type of uncertainty was related to the amount of published evidence supporting the risk score assigned to each criterion. A normalized scale (0-2, Table 3) was developed to estimate the evidence uncertainty associated with the total weighted and unweighted risk scores for each pathogen. If we were unable to find published information about a particular criterion for a particular pathogen, the risk score was extrapolated from similar pathogens and was assigned a high uncertainty score (2) for that criterion. Total evidence uncertainty score for each pathogen was estimated using Equation 3:\(Total\ evidence\ uncertainty=\sum_{i=1}^{7}U_{\text{ij}}\) (3)
where Uij is the normalized uncertainty score for pathogen j on criterion i. Total evidence uncertainty scores for each pathogen are reported in Table 4.
A third type of uncertainty was related to the ‘confidence level’ of the stakeholder-experts in assigning the weight values. Experts indicated their confidence in the assigned weights by a score between 1 (low) and 10 (high, integer number). The confidence scores were intended to illustrate the range and variety of confidence from various experts and not used in the final calculations of the risk ranking.