What this study adds:
- The Adverse Inpatient Medication Event (AIME) model has potential
clinical utility and could assist with identifying high-risk
inpatients for early and targeted health professional review.
- The AIME model includes novel laboratory variables including
supra-therapeutic INR.
- The AIME model can be incorporated into hospital digital systems to
identify and monitor high-risk patients in real-time, to reduce
medication harm and improve patient outcomes
Introduction
Medications offer significant health benefits; however, their use can
also result in patient harm [1-2].
Medication harm has negative
clinical and economic outcomes, contributing to hospitalisations,
increased length of stay (LOS), morbidity and mortality [3]. Annual
costs are estimated at USD $42 billion internationally, and $1.2
billion in Australia [4-5].
The incidence of inpatient
medication harm ranges from four to 14%, of which up to 50% of events
are thought to be preventable [6-7]. Medication harm can cause
physical, cognitive and emotional impairment to the patient [8]. The
Australian healthcare system has introduced penalties for
hospital-acquired complications (HACs) due to medications [9].
Two effective strategies to reduce medication harm include
pharmacist-led medication reconciliation and a clinical evaluation of
medications [10]. These services are resource intensive [10] and
fiscal constraints, mean that pharmacists cannot provide extensive
services to every inpatient. A suggested solution is to identify
high-risk patients and prioritise them for early and targeted medication
management [11-12]. However, there is currently no known
standardised, evidence-based methods implemented in clinical practice.
An emerging approach are risk prediction models, which use statistical
algorithms to quantify the probability that a patient will experience
medication harm [13]. Predictive analytics plays a central role in
medicine, by using influential factors to predict outcomes and
facilitate timely, patient-specific interventions. Examples include the
Framingham risk score for cardiovascular risk prediction [14], the
CHA2DS2‐VASc score for predicting risk
of stroke [15], and the HAS-BLED score for anticoagulant-related
bleeding risk, in patients with atrial fibrillation [16].
Several medication risk prediction models have been developed and
validated for use in the hospital setting, none have been implemented in
practice, and all have methodological and/or reporting limitations
[17]. Models exist for predicting risk of medication errors,
medication-related problems, or actual harm, with studies predominantly
including small cohorts of older adults, in European hospital settings
[17]. Given that the majority of medication errors do not have
clinically significant consequences, models that do not predict patient
harm are limited in their potential to guide clinical decisions
[18]. To date, no models that predict medication harm have been
developed or evaluated for use in an Australian inpatient setting. As
population characteristics have an important role in risk assessment,
this study aimed to develop, validate, and report a robust model,
predicting actual medication harm, from data obtained in an acute
medical and rehabilitation setting.
Aim
To develop and internally validate
a risk prediction model, to identify and prioritise hospitalised
patients, at risk of medication harm.
Methods
Study participants and
design
The Adverse Inpatient Medication Event model (AIME) study was a
retrospective cohort study, including adult patients, sequentially
admitted over six months, from the 1st of July 2017 to
31st of December 2017, to the general medical and/or
the Geriatric Assessment and Rehabilitation Unit, of a quaternary
teaching hospital in Australia. Patients admitted for less than 24 hours
were excluded. Ethical approval was obtained from the Human Research
Ethics Committee at Metro Health South (HREC/17/QPAH/353).
Outcome measure
The outcome of interest in this study was inpatient medication harm, as
defined by Edwards and Aronson [19], and later updated by Aronson
and Ferner [20]; “An appreciably harmful or unpleasant reaction,
resulting from an intervention related to the use of a medicinal
product; adverse effects usually predict hazard from future
administration and warrant prevention, or specific treatment, or
alteration of the dosage regimen, or withdrawal of the product.” This
included any negative inpatient outcome or injury resulting in clinical
signs, symptoms or physiological abnormalities, related to the use of a
medication during hospitalisation.
Medication harm was identified from the medical records of patients
coded as having had an adverse medication event (Y-codes), using the
Tenth Edition of The International Classification of Disease Codes,
Australian Modified (ICD-10 AM). Additionally, the hospital incident
reporting database (RISKMAN) was reviewed to identify cases where a
medication incident had resulted in patient harm.
All medical records of any patients flagged with a potential medication
event were comprehensively reviewed by an investigator (NF), a senior
clinical pharmacist, to evaluate the medication harm events and
establish causality, severity and preventability. Causality was
ascertained using the Hallas criteria [21]. Definite, probable or
possible events were included. Events classified as unlikely were
excluded. Severity of harm was determined using the definitions
described by Morimoto et al. [22]. Preventability was assessed as
per Hallas et al. [21], and using the Schumock and Thornton criteria
[23]. Where there was uncertainty in rating an event, this was
discussed with senior colleagues. A randomly selected 30 cases were also
assessed for causality, severity and preventability by two independent
experts; a senior hospital pharmacist and a clinical pharmacologist.
Where event ratings differed between investigators, these were discussed
to resolve discrepancies, and reach consensus.
Selection and definitions of variables for model
development
Variables for model development were identified using two methods.
First, a systematic review of the literature was undertaken to identify
established risk factors for medication harm, and significant variables
in existing risk models [17]. Second, hospital pharmacist focus
groups and a national survey of Australian clinical pharmacists helped
identify key criteria routinely used to prioritise patients at high-risk
of medication harm [24]. To ensure clinical relevance and minimise
noise, we selected clinically useful variables that could be quantified
from digital sources, during a patient’s hospitalisation. Some
continuous variables, such as International Normalised Ratio (INR), were
categorised according to clinically meaningful risk thresholds,
identified from hospital guidelines and pharmacist prioritisation
criteria [24].
The 68 selected variables were categorised into patient demographics,
social risk factors, hospital utilisation data, medications used and
pathology results. Variables were extracted from the hospital’s
electronic medical records (EMR) and presented in reports developed and
validated for the purpose of this study, in collaboration with the
hospital’s Informatics team. Medications were grouped guided by the
Australian Medicines Handbook 2017 Edition [25]. Medications at
admission were defined as the number of distinct medications,
administered within the first 24 hours of hospitalisation.
Comorbidities were defined as per ICD-10 AM codes, which were used to
group conditions. Laboratory tests were those measured within the first
24 hours of admission. These included full blood count, renal function,
serum electrolyte levels, serum blood glucose levels and coagulation
studies.
Renal function at admission was the
estimated Glomerular Filtration Rate (eGFR) calculated using the Chronic
Kidney Disease Epidemiology Collaborative (CKD-EPI) equation [26].
This was analysed as both a continuous variable and categorised at
clinically informative thresholds [24]. Low serum sodium at
admission was defined and categorised as sodium levels ≤ 125 mmol/L. INR
and aPTT were also categorised into two groups, with supratherapeutic
levels defined as greater than 3, and greater than 100 seconds,
respectively [24]. Thresholds were informed from a previous study
[24].
Sample size
Sample size was estimated using an event per variable (EPV) ratio of 10.
This was based on a medication harm rate of 7% using local data
[6], and the estimated inclusion of up to 14 variables from
univariable pre-selection, as guided by prior prognostic model
development studies [17].
Data handling and modelling
methods
Before modelling, the distributions of variables were examined using
graphs and descriptive statistics. Model development involved two
stages. Univariable analysis was used to identify significant
relationships with the outcome; chi-squared tests or regression analyses
determined which variables should be included in a multivariable
logistic regression analysis. Continuous variables were first analysed
as continuous and later categorised using thresholds previously
described [24]. Modelling was undertaken with both continuous and
dichotomised variables, and where categorisation improved model fit, the
dichotomised variable was used. Log transformation was applied where it
optimised model fit.
Polynomials and interaction
terms
Polynomials were examined for continuous variables with potential
curvilinearity. Clinically plausible interaction terms were also
examined and included potential interactions between gender with heart
disease, and diabetes with glucose levels.
Missing values
The number of missing values was identified for each variable and
patterns of missingness were examined. There were ten patients with
missing data for inpatient medications and so complete case analysis was
used. Missing laboratory test results (except INR and aPTT) were imputed
using the mean value. Given the small number of missing values for the
majority of variables, single imputation was considered sufficient to
obtain reasonable predictions [27].
There were a larger proportion of missing values for INR and aPTT, and
missingness was not deemed to be random. As INR and aPTT are not
routinely measured for all patients, only patients who clinically
required a test had values reported. Therefore, INR and aPTT were
categorised into two groups and the most clinically plausible values
were used to impute variables. Patients without an INR test during
hospitalisation were categorised into the lower INR group (≤ 3).
Patients without an aPTT test were categorised into the lower aPTT group
(≤ 100 seconds). A Clinical Pharmacologist and Biostatistician
independent of this study were consulted regarding the imputation of
these variables.
Selection of variables in multivariable
analysis
Binomial logistic regression analysis was undertaken to identify the
optimal combination of the most influential predictors in the AIME
model. The alpha level to determine appropriate variable inclusion was
set at p ≤ 0.10 [28]. Using the above alpha level, significant
variables from univariable analysis were included in the multivariable
analysis.
The final model was selected using backward elimination as it is
preferred to forward selection in predictive modelling [29].
Analysis was undertaken using R statistical software®version 5.3.1 [30], using both a manual method and also the
automated stepAIC function from the R package ‘MASS’ [31]. The final
model was determined by minimising the Akaike Information Criterion
(AIC), and retaining variables with p-values < 0.10, whilst
ensuring a clinically plausible risk model.
Model Performance
Model performance was evaluated using three measures; discrimination,
calibration, and variance, as recommended in the
Transparent Reporting of a
multivariable prediction model for Individual Prognosis Or Diagnosis
(TRIPOD) Statement [13, 32]. Model discrimination was measured using
area under the receiver operative characteristic (AuROC) curve, where an
AuROC of 1 is considered a perfect model and 0.5 unsatisfactory
[27].
Model calibration was assessed using plots and formal statistical
testing, using the Hosmer-Lemeshow goodness of fit (GOF) test. The
Nagelkerke R² was calculated as a measure of variability explained by
the model [27] and the Youden’s index was used to identify the
optimal threshold.
Model validation
Internal validation of the final model was undertaken using 10-fold
cross validation with 200 replications. The performance measures for the
optimism corrected, internally validated model, were reported.
Clinical usefulness of the AIME
model
The clinical usefulness of the AIME model was evaluated using decision
curve analysis (DCA). DCA adds to performance measures to identify the
potential net benefits and harms of using a model in practice, using
values of true positives and false positives, at different
thresholds. Decision curves were constructed for the AIME model, and a
simpler ‘Polypharmacy’ model. The Polypharmacy model, based on a common
approach of ‘number of medications’ for patient prioritisation [33],
was informed by the average number of medications administered to study
participants within the first 24 hours of hospitalisation. The
standardised net benefit (equation 1.1) of applying the models were
calculated at different threshold probabilities to create the decision
curves. The models demonstrated clinical usefulness where they had
positive net benefit values [34]. In this study, the term ‘clinical
usefulness’ was used to show where a model had greater net benefit than
a “treat-all” (pharmacist intervention for all patients), or
“treat-none” (no patient receives pharmacist intervention) approach.
Using guidance from the literature we identified a probability threshold
of greater than 5% risk as a suitable threshold for intervention
[33]. This was based on consensus by an expert panel, with the
assumption here being that it would be unlikely for a pharmacist to
prioritise a patient for urgent review who had a probability of risk
below this threshold. This was then compared with our findings from the
decision curve.
Equation 1.: Net Benefit Calculation for Decision Curve [34]
\begin{equation}
Net\ Benefit=\frac{\text{TruePositives}}{n}-\frac{\text{FalsePositives}}{n}\ (\frac{\text{Pt}}{1-Pt})\nonumber \\
\end{equation}Key: Pt is threshold probability
Results
Study participants
A total of 1982 patients were included in the study. The median (IQR)
patient age was 74 (62-86) years, and 883 (45%) of patients were males.
Key baseline characteristics of the participants are reported in Tables
1 and 2.
Insert Table 1
Insert Table 2
Study outcome (medication
harm)
A total of 136 (7%) inpatients experienced one or more medication harm
events. Some patients experienced multiple inpatient events, resulting
in a total of 155 events. The causality assessment classified 20% of
events as definite, 45% as probable, and 35% as possible. Events
classified as unlikely to be medication related were excluded. Severity
assessment showed that 12% of events were significant (defined as an
adverse reaction that does not require a change in therapy, but may need
supportive treatment), 70% were serious (requiring dose reduction or
therapy cessation, some requiring additional therapeutic measures or
specific treatment, and/or a minor increase in LOS). Eighteen percent of
events were severe and potentially life-threatening (leading to severe
or permanent harm, and causing a substantial increase in LOS [defined
as greater than two days]). Examples of severe events included Heparin
induced thrombocytopenia and thrombosis (HITT) resulting in pulmonary
embolism, insulin related hypoglycaemia leading to seizure, anaphylaxis
due to antibiotics, and gastrointestinal bleeding due to heparin. There
were no fatal events.
Preventability assessment classified 28% of cases as definitely
preventable, 31% as possibly preventable, and 41% as not preventable.
Events classified as preventable predominantly comprised of incorrect
medication choice, incorrect dose, inappropriate combinations of agents
and/or inadequate patient review, deprescribing and surveillance, as
well as ‘lack of knowledge’ errors. Greater focus on rationalisation of
medications, for example older adults on multiple centrally acting or
cardiovascular agents, education on best prescribing practices, and
specialist involvement were identified by reviewers as strategies that
may have mitigated events (for example, inappropriate conversion of
opioids).
A total of 82% of events were classified as type A reactions (common,
predictable and often dose-related), and 18% as type B reactions
(uncommon, unpredictable and often immune mediated reactions)[19].
Adverse medication events, summarised by medication class and patient
reactions are reported in Table 3. The top medication class implicated
in medication harm were cardiovascular agents (in particular,
beta-receptor blocking agents, digoxin, and diuretics).
Insert Table 3
Univariable preselection of
variables
The twenty-one variables which had a statistically significant
relationship with medication harm, at an alpha level of 10% (p ≤ 0.1),
are shown in Table 4. Renal function was not statistically significant
(p = 0.67), but was included, given its clinical relevance and presence
in prior risk models. The inclusion of 22 variables with 136 harm events
gave an EPV of approximately 6.
Insert Table 4
Multivariable Logistic Regression
Analysis
Length of stay (LOS) was log transformed and INR, serum sodium and
number of medications were dichotomised. The final model consisted of 10
variables (Table 5). The variables in the final prediction model (and
reference levels for coding) are shown in Table 5:
Insert Table 5
Model Performance
The AuROC curve for the model prior to cross validation was 0.72, 95%
CI 0.67 – 0.76. The cross-validated model performance reduced to 0.70,
95% CI 0.65 – 0.74 (Figure 1).
Insert Figure 1
The AIME model was well calibrated, with a Hosmer-Lemeshow statistic of
p = 0.53. The regression equation to calculate the probability of
medication harm for hospitalised patients is shown as Equation 1.2.
Youden’s index was used to identify the optimal probability threshold
for obtaining a balance between sensitivity and specificity. At a
threshold probability of 0.05 for identifying high-risk patients, the
model had a sensitivity of 77% and specificity of 58%.
\(p=\ \frac{1}{1+\ e^{-\ (-4.56+\beta_{1}X_{1}+\beta_{2}X_{2}+\ \ldots\ +{\beta_{10}X}_{10})}}\)Equation 1.: AIME model
where,