Rationale, Aims and Objectives: One of the sectors challenged by the COVID-19 pandemic is medical research. COVID-19 originates from a novel coronavirus (SARS-CoV-2) and the scientific community is faced with the daunting task of creating a novel model for this pandemic or, in other words, creating novel science. This paper aims to explore the intricate relationship between the different challenges that have hindered biomedical research and the generation of scientific knowledge during the COVID-19 pandemic. Methods: During the early stages of the pandemic, research conducted on hydroxychloroquine (HCQ) was chaotic and sparked several heated debates with respect to the scientific methods used and the quality of knowledge generated. Research on HCQ is used as a case study in this paper. The authors explored biomedical databases, peer-reviewed journals, pre-print servers and media articles to identify relevant literature on HCQ and COVID-19, and examined philosophical perspectives on medical research in the context of this pandemic and previous global health challenges. Results: This paper demonstrates that a lack of prioritization among research questions and therapeutics was responsible for the duplication of clinical trials and the dispersion of precious resources. Study designs, aimed at minimizing biases and increasing objectivity, were, instead, the subject of fruitless oppositions. These two issues combined resulted in the generation of fleeting and inconsistent evidence that complicated the development of public health guidelines. The reporting of scientific findings highlighted the difficulty of finding a balance between accuracy and speed. Conclusions: The COVID-19 pandemic presented challenges in terms of (1) finding and prioritizing relevant research questions, (2) choosing study designs that are appropriate for a time of emergency, (3) evaluating evidence for the purpose of making evidence-based decisions and (4) sharing scientific findings with the rest of the scientific community. This paper demonstrates that these challenges have often compounded each other.
Rationale, aims and objectives: People living with dementia admitted to hospitals are more likely to experience poorer outcomes than people without dementia. Caregivers play an important role in managing medications across transitions of care. This qualitative study explores the experiences and perspectives of caregivers about the medication management guidance provided at hospital discharge. Methods: A qualitative approach using semi-structured, telephone interviews was conducted with 31 caregivers of people with dementia across Australia. Purposive sampling was used to ensure maximum variation of diverse experiences and perspectives. Results: Caregivers’ experiences of medication guidance for people with dementia at discharge were described in three themes including: a) inadequate information about medication management at discharge; b) limited caregiver engagement in medication management decisions; and c) difficulties ensuring medication supply post discharge. Most participants indicated they would like to be included in discussions at discharge. However, participation was influenced by caregivers being overwhelmed by discharge processes; proactively seeking information on medication-related harm; and belief in advocacy as part of their caregiver role. Caregivers reported they would like to receive a tailored medication list for people with dementia which included information on medications that may impact on the patient’s cognition, and for hospital staff to communicate with both the community pharmacist and primary care physician, to improve co-ordination post transition. Discussion: In our study of caregivers of people with dementia, we identified key recommendations that could be tested to facilitate regular participation of people living with dementia and their caregiver around medication guidance at discharge.
Artificial intelligence and big data are more and more used in medicine, either in prevention, diagnosis or treatment, and are clearly modifying the way medicine is thought and practiced. Some authors argue that the use of artificial intelligence techniques to analyze big data would even constitute a scientific revolution, in medicine as much as in other scientific disciplines. Moreover, artificial intelligence techniques, coupled with mobile health technologies, could furnish a personalized medicine, adapted to the individuality of each patient. In this paper we argue that this conception is largely a myth: what health professionals and patients need is not more data, but data that are critically appraised, especially to avoid bias. The validity of data and the validity of inferences drawn from the data by algorithms are indeed a major epistemic issue, though rarely addressed as such by health professionals or philosophers of medicine. Considering the history of epidemiology, specifically the formation of the concept of bias, we propose three research priorities concerning the use of artificial intelligence and big data in medicine.
Motion capture and analysis techniques are emerging in the surgical education and surgical education research literature as viable ways to augment the assessment of technical skills. In particular, these methods provide an opportunity to reveal objective information about the efficiency of surgical procedures, above and beyond the accuracy of procedural outcomes. One assessment that is very prevalent in the literature are counts of the number of movements a surgeon makes in completing a technical performance. In this commentary, the number of movements metric is explored from kinesiology and engineering perspectives; two disciplines that have contributed heavily to the development of rigorous motion analysis methods. Furthermore, the assumption that skill efficiency improves linearly as a learner progresses along the continuum of expertise is challenged. While movement efficiency does certainly improve, this assumption does not necessarily capture the way that learners flexibly prioritize particular aspects of performance in the intermediate stages of skill learning. By way of this commentary, important a priori decisions that should proceed effective motion capture and analysis are highlighted, a call for the standardization of procedures is made, and an opportunity to better understand the way that computerized movement analysis techniques may contribute (or be detrimental) to competency constructs in surgical education and assessment is realized.
Rationale The end of life (EOL) experience in the intensive care unit (ICU) can be psychologically distressing. The 3 Wishes Project (3WP) personalizes the EOL experience by carrying out wishes for dying patients and their families. While the 3WP has been integrated in academic, tertiary care ICUs, implementing this project in a community ICU has yet to be described. Objectives To examine facilitators of, and barriers to, implementing the 3WP in a community ICU from the clinician and key-stakeholder perspective. Methods This mixed-method study evaluated the implementation of the 3WP in a 20-bed community ICU in Southern Ontario, Canada. Patients were considered for the 3WP if they had a high likelihood of imminent death or planned withdrawal of life-sustaining therapy. Quantitative data include patient demographic data and wishes implemented. Following the qualitative descriptive approach, semi-structured interviews were conducted with purposively sampled clinicians and key-stakeholders. Data from transcribed interviews were analyzed in triplicate through qualitative content analysis. Results During the 10-month period, 66 of 67 wishes were completed, with a median of 4.5 wishes per patient-family dyad. Interviews with 12 participants indicated that the 3WP personalized and enriched the EOL experience for patients, families and clinicians. Interviewees indicated higher intensity education strategies were needed to enable spread as the project grew. Clinicians described many physical resources for the project but required more non-clinical project support for orientation, continuing education and data collection. Instead, these roles were completed by clinicians with saturated work capacity which may have inhibited the spread of the project. Conclusions In this community hospital, ICU clinicians and key stakeholders reported the 3WP improved EOL care for patients, families, and clinicians. Project implementation in a community ICU requires investigators take into account project characteristics and adapt the intervention to the community hospital context.
The frantic search for a cure or prophylactic treatment of COVID-19 has unfortunately led to a dropping of the guard of many medical specialists resulting in widespread adoption of unproven treatment modalities. The recent article regarding the inconsistent physician attitudes towards hydroxychloroquine paints a depressing picture of the actual practices during the so-called era of evidence-based medicine. On this backdrop, we comment on how Romania (where this survey took place) has imposed some of the most severe lockdown measures in Europe, including the forced hospitalization of all confirmed with SARS-CoV-2 infection. Additionally, a therapeutic guideline was written into law, endorsing concomitant use of several drugs with unproven antiviral efficiency. This unprecedented situation has resulted in the sometimes indiscriminate prescription of off-label drugs, with a non-negligible risk of adverse reactions, especially in fragile patients with coexisting conditions. In light of the experience accrued in a COVID-19 dedicated unit, the authors discuss the importance of avoiding polypharmacy and administering all antiviral drugs within the confines of rigorously conducted clinical trials.
Despite the great promises that artificial intelligence (AI) holds for health care, the uptake of such technologies into medical practice is slow. In this paper, we focus on the epistemological issues arising from the development and implementation of a class of AI for clinical practice, namely clinical decision support systems (CDSS). We will first provide an overview of the epistemic tasks of medical professionals, and then analyse which of these tasks can be supported by CDSS, while also explaining why some of them should remain the territory of human experts. Clinical decision-making involves a reasoning process in which clinicians combine different types of information into a coherent and adequate ‘picture of the patient’ that enables them to draw explainable and justifiable conclusions for which they bear epistemological responsibility. Therefore, we suggest that it is more appropriate to think of a CDSS as clinical reasoning support systems (CRSS). Developing CRSS that support clinicians’ reasoning process therefore requires that: 1) CRSSs are developed on the basis of relevant and well-processed data; and 2) the system facilitates an interaction with the clinician. Therefore, medical experts must collaborate closely with AI experts developing the CRSS. In addition, responsible use of an CRSS requires that the data generated by the CRSS is empirically justified through an empirical link with the individual patient. In practice, this means that the system indicates what factors contributed to arriving at an advice, allowing the user (clinician) to evaluate whether these factors are medically plausible and applicable to the patient. Finally, we defend that proper implementation of CRSS allows combining human and artificial intelligence into hybrid intelligence, were both perform clearly delineated and complementary empirical tasks. Whereas CDRSs can assist with statistical reasoning and finding patterns in complex data, it is the clinicians’ task to interpret, integrate and contextualise.
Rationale, aims, and objectives: The complexity of healthcare systems makes errors unavoidable. To strengthen the dialogue around how physicians experience and share medical errors, the objective of this study was to understand how experienced generalist physicians make meaning of and grow from their medical errors. Methods: This study used a narrative inquiry approach to conduct and analyze in-depth interviews from 26 physicians from the generalist specialties of emergency, internal, and family medicine. We gathered stories via individual interview, analyzed them for key components, and rewrote a ‘meta-story’ in a chronological sequence. We conceptualized the findings into a metaphor to draw similarities, learn from, and apply new principles from other fields of practice. Results: Through analysis we interpreted the story of an elite athlete (physician) who is required to make numerous decisions in a short period of time within the construct of a chaotic sports field (clinical environment) among spectators (the patient’s family) whilst abiding by existing rules and regulations. Through sharing stories of success and failure, the team coach (clinical mentor) helps optimize the players’ professional and psychological development. Similarly, through sharing and learning from stories, team members (colleagues) and junior team members (trainees) also contribute to the growth of the protagonist’s character and the development of the overall team (clinic/hospital) and sport (healthcare system). Conclusion: We draw parallels between the clinical setting and a generalist physician’s experiences of a medical error with the environment and practices within professional sports. Using this comparison, we discuss the potential for meaningful coaching in medical education.
Rationale. Evidence-based practice (EBP) can improve health care in underprivileged countries. Bolivia’s EBP movement is nascent and the factors contributing to better implementation in nursing are unknown. Aim. To explore Bolivian nurses’ readiness to engage in EBP while highlighting the facilitators and barriers for pursuing EBP. Method. International collaborators used a sequential explanatory mixed methods study. First, general trends were disclosed via a survey of 170 nurses in La Paz, Bolivia, holding at least a baccalaureate regarding their perceived beliefs about EBP. The survey identified facilitators and barriers for implementing EBP in acute and ambulatory settings. Second, qualitative data was gathered via a focus group of nine nurses with the purpose of enhancing the survey findings. Results. The survey results showed that nurses believe that engaging in EBP can improve their clinical practice. However, the nurses’ research behaviors were found to be infrequent. Lack of support from the nurses’ clinics and hospitals and from non-nursing professionals were identified as barriers for engaging in EBP. The qualitative results revealed underlying limitations to nurses’ clinical practice, including “feeling undervalued.” Conclusions. There is a dearth of EBP knowledge among Bolivian nurses stemming from a lack of preparation in EBP environments, including EBP training opportunities. This situation affects nurses’ professional dimensions of relational work, power, and collaboration. Collaborative research among educators, professional nursing societies, and local and international organizations could provide initiatives for implementing EBP, based on local health profiles. Key words: international collaboration, evidence-based practice, nurse-multidisciplinary relationship, barriers to EBP.
ABSTRACT Rationale, aims and objectives The primary aim of the study was to understand the mindset of doctors and pharmacists, as they embark upon prescribing in a polypharmacy and multi-morbidity context during routine practice at a hospital acute admissions unit. The study also aimed to evaluate to what extent attitudes, embedded within real-life decision-making scenarios, relate to existing theory and models of prescribing decisions. Methods Anonymised case studies were identified from the medical notes of patients aged 65 and over with conditions likely to be associated with multi-morbidity, medication issues and polypharmacy: namely: fall, urinary tract infection, confusion or lower respiratory tract infection. A total of 39 doctors based on the acute medical admissions unit and 9 pharmacists were recruited to one of three focus groups. Patient case-studies provided the context for discussion from which verbatim transcripts were thematically analysed using an interpretative, qualitative approach. Sub-themes were matched to Murshid and Mohaidin’s proposed model of physician prescribing decisions. Results Seven principal themes were identified that were associated with prescribing decisions on the acute medical unit, namely, ‘patient characteristics’, ‘drug characteristics’, ‘pharmacist factors’, ‘trustworthiness’, ’reliability of medication history, ‘competing pressures and priorities’ and ‘responsibilities of prescribers’. Conclusion Prescribing decisions on the acute medical admissions unit were influenced by a variety of factors, some of which have already been acknowledged within existing theories and models. The findings provisionally offer new insights, which, subject to confirmation by further research, bring to light three attitudinal characteristics that may impact negatively upon the quality of prescribing decisions. These include, first, how perceived poor reliability of medication history may result in information gaps that compromise prescribing decisions; second, how competing pressures and priorities restrict doctors’ aptitude to conduct a review of medication and finally, how doctors may rationalise the assignment of medication review to the GP.
Linden and Yarnold1 recently proposed classification tree analysis (CTA), a machine‐learning procedure, as an alternative to conventional methods for analyzing mediation effects in treatment-outcome research. They note that CTA may have a number of advantages in this regard. It requires no assumptions about the distribution of variables or the functional form of the best-fitting model, for example, thus affording greater potential flexibility in identifying complex forms of association among variables with varying scales of measurement (e.g., binary and continuous). The authors further argue that CTA, unlike conventional approaches to testing mediation, “will not generate a model if a treatment-mediator-outcome relationship does not exist” (p. 359) and, conversely, that CTA “will systematically identify a treatment-by-mediator interaction if it exists, as well as any other interaction between variables.” (p. 359). Using data from the Job Search Intervention Study (JOBS II), they find that structural equation modeling, a conventional approach to testing mediation, failed to indicate support for job-search self-efficacy as a mediator of the effects of the intervention on whether the study participant was reemployed at follow-up. In contrast, the authors conclude that CTA applied to the same set of variables revealed that job-search self-efficacy was a mediator of intervention effects on employment among those in the treatment group.There are, however, two problematic aspects of the authors’ approach. First, as they point out, causal inference of a mediational pathway depends on the assumption that the associations involved exist net of potential confounding variables. In a randomized control trial such as JOBS II, confounding of the association between the mediator and outcome is of particular concern.2 The authors seek to address this concern by conducting a CTA that includes a set of 9 potential confounders (e.g., income, age, initial level of depressive symptoms) as candidate predictors. This approach does not ensure statistical control for these variables, however, for at least two reasons. First, there is no guarantee that the variables involved will actually be included in the resulting classification tree; failing to meet the criterion for inclusion does not rule out the possibility that a given potential confounder or combination of such confounders nonetheless share an association with the mediator and outcome to an extent that renders their association non-significant. This concern proves pertinent to the authors’ analysis as only 3 of the potential confounders earn entry into the resulting classification tree for reemployment status. A second concern with the approach taken by the authors is that whatever covariates are included in the classification tree may be included in positions that are inadequate for the purpose of controlling for confounding of the mediator-outcome association. Gender, for example, in their analysis is included in the control group branch of the tree, whereas job-search self-efficacy, the potential mediator, is included only in the treatment group arm. As a further example, education is included on the treatment group arm, but only after job-search self-efficacy’s role in the model already has already been established through its inclusion on a higher tier of the model. An alternative approach for addressing confounding in the context of CTA would be to residualize the candidate mediator on possible confounders and then fit a classification tree with this adjusted variable. If the residualized mediator continues to discriminate across levels of the outcome, conditional on treatment status, it could be concluded that the mediate-outcome portion of the potential mediational pathway of interest is evident independent of the measured confounders. Applying this approach, I find that the job-search self-efficacy continues to emerge as a discriminating variable for reemployment in the treatment arm branch of the classification tree that I fit with the same JOBS-II data. However, it cannot be assumed that this more rigorous approach to taking into account confounders will always yield the same result as one in which they are merely included as candidate predictors within a CTA.A more fundamental concern with the CTA approach to identifying mediation employed by the authors is that the mediator in the predictor-mediator and mediator-outcome segments of the mediational pathway are not assured of having a consistent definition. This is a result of the optimal cut-point on the mediator being determined separately for the two segments, through an optimal discriminant analysis for the treatment-mediator segment and CTA for the mediator-outcome segment. Thus, in the case of the mediational pathway for reemployment, treatment status predicts a dichotomous measure of job-search self-efficacy determined by a cut-score of 3.92 (i.e., high job-search self-efficacy corresponds to values above 3.92 and low to values equal to and below 3.92), whereas the dichotomous measure of job-search self-efficacy predicting reemployment is determined by a different cut-score of 4.92. Yet, it is essential by definition that the mediator in a mediational pathway that is influenced by the initial variable in the pathway (in this case, treatment) be the same variable that then influences the outcome (in this case, reemployment). In this case, nearly one half of the sample (47.3%; n = 426) has a different high-low classification on the job-search self-efficacy, suggesting considering divergence between the versions of this variable determined by the respective cut-scores. One approach to addressing this concern would be to define a new mediator that reflects the overlapping portion of the differing definitions. In the present example, this could be a dichotomous measure of job-search self-efficacy defined by a cut-score of greater 3.92, which would ensure that all those with scores classified as relatively high on the measure continue to be classified as such; alternatively, a cut-score of 4.92 could be used if priority is given to ensuring that all those classified as low remain in this category. Still another option would be to use the mid-point between the two cut-scores of 4.42. One could then evaluate the mediational pathway of interest using one or more of these options. Applying this approach using PROC CAUSALMED in SAS3, allowing for the suggested treatment-mediator interaction and including all covariates, I find evidence of an indirect mediated pathway when using the lower bound cut-off score of 3.92 for job-search self-efficacy (Odds Ratio for natural indirect effect of .959, 95% CI limits of .893, .998), but not when using the other two cut-off scores, although the differences in estimates are admittedly small in this case (complete results are available upon request).To summarize, Linden and Yarnold (2018) make a significant contribution by introducing CTA as a promising strategy for identifying mediated effects in intervention research. Further refinements to their approach, however, are recommended to more fully incorporate fundamental assumptions that accompany all tests for mediation as well as to evaluate and confirm potential mediational pathways using conventional procedures.References1Linden A, Yarnold PR. Identifying causal mechanisms in health care interventions using classification tree analysis. J Eval Clin Pract. 2018;24:353–361. https://doi.org/10.1111/jep.128482Imai K, Keele L, Tingley D. A general approach to causal mediation analysis. Psychol Methods . 2010;15:309‐334.3Yung, Y, Lamm, M, Zhang, W. Causal Mediation Analysis with the CAUSALMED Procedure . Paper SAS1991-2018. Cary, NC: SAS Institute Inc.; 2018.
Rationale, aims and objectives In the US, the reluctance of the federal government to impose a national stay-at-home policy in wake of COVID19 pandemic has left the decision of how to achieve social distancing to individual state governors. We hypothesized that in the absence of formal guidelines, the decision to close a state reflects the classic Weber-Fechner law of psychophysics- the amount by which a stimulus (such as number of cases or deaths) must increase in order to be noticed as a fraction of the intensity of that stimulus. Methods On April 12, 2020 we downloaded data from the New York Times database from all 50 states and the District of Columbia; by that time all but 7 states had issued the stay-at-home orders. We fitted the Weber-Fechner logarithmic function by regressing the log2 of cases and deaths respectively against the daily counts. We also conducted Cox regression analysis to determine if the probability of issuing the stay-at-home order increases proportionally as the number of cases or deaths increases. Results We found that the decision to issue the state-at-home order reflects the Weber-Fechner law. Both the number of infections (p=<0.0001; R2=0.79) and deaths (p<0.0001; R2=0.63) were significantly associated with the decision to issue the stay-at-home orders. The results indicate that for each doubling of infections or deaths, an additional 4 to 6 states will issue stay-at-home orders. Cox regression showed that when the number of deaths reached 256 and the number of infected people were over 16,000 the probability of issuing “stay-at-home” order was close to 100%. We found no difference in decision-making according to the political affiliation; the results remain unchanged on July 16,2020. Conclusions when there are not clearly articulated rules to follow, decision-makers resort to simple heuristics, in this case one consistent with the Weber-Fechner law.
Background: The health care delivery model in the United States does not work; it perpetuates unequal access to care, favors treatment over prevention, and contributes to persistent health disparities and lack of insurance. The historical lack of support in the United States for primary health care, universal health coverage, population health, addressing the social determinants of health, and community empowerment, creates opportunities for community health scientists to develop innovative solutions for addressing community health needs. Methods: We developed a model community health science approach combining community-oriented primary care (COPC), community-based participatory research (CBPR), asset-based community development, and service learning principles. The approach defines health as a social outcome, resulting from a combination of clinical science, collective responsibility, and informed social action. Results: From 2000-2020, we established partnerships with community organizations to reduce the risk of chronic disease in vulnerable minority communities. Our programs have provided structured community health science training for hundreds of physicians and other health care workers in training. Conclusion: As the U.S. begins to seek solutions to chronic health disparities and health inequities, community health science provides useful lessons in how to engage communities to address the deficits of the current system. Perhaps the greatest error that U.S. health care systems could make in trying to better address population health and the social determinants of health, would be ignoring the important community initiatives already underway in most local communities. Building partnerships based on local resources and ongoing social determinants of health initiatives is the key for medicine to meaningfully engage communities for reducing health disparities. This has been the greatest lesson we have learned during the past two decades, has provided the foundation for our community health science approach, and accounts for whatever success we have achieved.
Rationale Assessing the performance of diagnostic tests requires evaluation of the amount of diagnostic uncertainty the test reduces (i.e. 0% - useless test, 100% - perfect test). Statistical measures currently dominating the evidence-based medicine (EBM) field and particularly meta-analysis (e.g. sensitivity and specificity), cannot explicitly measure this uncertainty reduction. Mutual information (MI), an information theory statistic, is a more appropriate metric for evaluating diagnostic tests as it explicitly quantifies uncertainty and, therefore, facilitates natural interpretation of a test’s value. In this paper, we propose the use of MI as a single measure to express diagnostic test performance and demonstrate how it can be used in meta-analysis of diagnostic test studies. Methods We use two cases from the literature to demonstrate the applicability of MI meta-analysis in assessing diagnostic performance. These cases are: 1) Meta-analysis of studies evaluating ultrasonography (US) to detect endometrial cancer and 2) meta-analysis of studies evaluating magnetic resonance angiography to detect arterial stenosis. Results Results produced by the MI meta-analyses are comparable to the results of meta-analyses based on traditionally used statistical measures. However, the results of MI are easier to understand as it relates directly to the extent of uncertainty a diagnostic test can reduce. For example, a US test diagnosing endometrial cancer is 40% specific and 94% sensitive. The combination of these values is difficult to interpret and may lead to inappropriate assessment (e.g. one could favour the test due to its high sensitivity, ignoring its low specificity). In terms of MI however, the test reduces diagnostic uncertainty by 10%, which is marginal and thus the test is clearly not very useful. Conclusions We have demonstrated the suitability of MI in assessing the performance of diagnostic tests, which can facilitate easier interpretation of the true utility of diagnostic tests.
Purpose This paper aims to elucidate the factors that play into physicians’ experience of receiving practice data and to subsequently develop a model that describes how individuals may interact with the data they receive. Methods In a prior study, we conducted a needs analysis of 105 physicians in the Hamilton-Niagara area in order to understand which data metrics were most valuable to physicians. Using these results, we designed an interview guide to study physicians’ perspectives on audit and feedback. By intentional sampling, we recruited 15 physicians amongst gender groups, types of practice (academic vs community), and duration of practice. The interviews were conducted by a single author and transcribed without identifiers. We then began with an open coding analysis for all of the transcripts, and thereafter conducted axial coding to group the data into larger themes. Results Several environmental and personal attributes were identified as either enabling or counterproductive attributes for participant improvement. The final proposed model identifies different zones of engagement on the basis of both the individual practitioner’s growth mindset and the quality of the existing data system. In the highest engagement zone, the mindset of the collective leadership is one of growth. Systemic supports are in place which potentiates learning that may come from an individual motivated to use their own data. Conclusion Our model shows how data feedback systems and individual growth-oriented mindsets interact to augment or hinder clinical practice improvement. This model provides important guidance to academic and administrative structures looking to develop appropriate performance feedback systems with clinicians.
The COVID-19 has posed a wide range of urgent questions: about the disease, testing, immunity, treatments, and outcomes. Extreme situations, such as pandemics, call for exceptional measures. However, this threatens the production and application of evidence. This paper directs evidence production towards four types of uncertainty in order to address the challenges of the pandemic: Risk, Fundamental uncertainty, Ignorance, and Ambiguity. Eliminating ambiguity, being alert to the unknown, and gathering data to estimate risks are crucial to preserve evidence and save lives. Hence, in order to avoid fake facts and to provide sustainable solutions we need to pay attention to the various kinds of uncertainty. Producing high quality evidence is the solution, not the problem.
Rationale, aims and objectives New therapies are increasingly approved by regulatory agencies such as the Food and Drug Administration (FDA) and the European Medicines Agency (EMA) based on testing in non-randomized clinical trials. These treatments have typically displayed “dramatic effects” (i.e., effects that are considered large enough to obviate the combined effects of bias and random error). The agencies, however, have not identified how large these effects should be to avoid the need for further testing in randomized controlled trials (RCTs). We investigated the effect size that would circumvent the need for further RCTs testing by the regulatory agencies. We hypothesized that the approval of therapeutic interventions by regulators is based on heuristic decision-making whose accuracy can be best characterized by the application of signal detection theory (SDT). Methods We merged the EMA and FDA database of approvals based on non-RCT comparisons. We excluded duplicate entries between the two databases. We included a total of 134 approvals of drugs and devices based on non-RCTs. We integrated Weber-Fechner law of psychophysics and recognition heuristics within SDT to provide descriptive explanations of the decisions made by the FDA and EMA to approve new treatments based on non-randomized studies without requiring further testing in RCTs. Results Our findings suggest that when the difference between novel treatments and the historical control is at least one logarithm (base 10) of magnitude, the veracity of testing in non-RCTs seems to be established. Conclusion Drug developers and practitioners alike can use the change in one logarithm of effect size as a benchmark to decide if further testing in RCTs should be pursued, or as a guide to interpreting the results reported in non-randomized studies. However, further research would be useful to better characterize the threshold of effect size above which testing in RCTs is not needed.