Article Figures & Data
Figures
- Figure 1.
PRISMA flow diagram. Abbreviation: PRISMA, Preferred Reporting Items for Systematic Reviews and Meta-Analyses.
Tables
Study, Country Model Name (If Applicable) Study Population, Setting, Time Period Outcome(s) Sample Size Outcome Rate Modelling Methods Artificial Intelligence Methods Used† Validation Approach Validated Modelc-statistic*(95% CI) Model Outcome: Emergency Department Visits Frost DW, et al., 2017
CanadaAged ≥50, ≥1-year General Practice, 2011 to 2012 Frequent ED Use (≥3 ED visits)
(12-month)Derivation: 21,680
Validation: 895Training Cohort = 5.7% Logistic Regression Yes Internal:
Split sampleValidation
c = 0.71 (no CIs)Howell P, & Elkin PL., 2019
United StatesNot Reported, 2017 to 2018 Emergency Department Visit
(12-month)2991 Not Reported Random Forest Yes External validation only Validation
c = 0.83 (no CIs)Pearce et al., 2019
AustraliaPOLAR Diversion All patients, General Practice, 2010 to 2015 Emergency Department Visit
(12-month)37,665 ≥1 ED Visit = 23% Support Vector Machine Yes Internal: Split sample
10-Fold Cross-validation0 to 30 days:
Sensitivity = 68%; PPV = 73.7%
31 to 365 days:
Sensitivity = 10%; PPV = 36.8%Hu Z, et al., 2015
United StatesAll patients,
2012 to 2013Emergency Department Visit
(6-month,
all-cause)Retrospective cohort: 829,641
Prospective cohort: 875,979Retrospective = 11.48%,
Prospective = 11.37%Survival Forest Decision Trees Yes External validation: Prospective Retrospective
c = 0.74 (no CIs)
Prospective
c = 0.73 (no CIs)Hao et al., 2014
United StatesAll patients,
2012 to 2013Emergency Department Revisit
(30-days)(ED encounters)
Derivation: 293,461
Prospective validation: 193,886Retrospective = 19.4%
Prospective = 20.5%Decision Tree Yes Split sample
External validation:
ProspectiveRetrospective
c = 0.71 (no CIs)
Prospective
c = 0.70 (no CIs)Bhavsar et al., 2018
United States(1) Electronic Health Record (HER)
(2) EHR + Neighborhood Socioeconomic Status (SES)Aged ≥18, ≥1 health care encounter in previous year, Durham County resident,
2009 to 2015Emergency Department Visit Derivation: 90,097
Validation: 122,812Not Reported Random Survival Forest Yes Temporal Split (1) c = 0.75
(no CIs)
(2) c = 0.75
(no CIs)Crane et al., 2010
United StatesAged ≥60, inpatient, primary care community dwelling, assisted living patients,
2005 to 2006Emergency Room Visit (2-year) 12,650 Not Reported Logistic Regression No Bootstrapping
(450 samples)AUC = 0.64
(no CIs)Model Outcome: Hospitalizations Rahimian et al., 2018
United KingdomPredictor Sets
(1) QAdmissions (QA)
(2) QAdmissions+
(QA+)
(3) Temporal (T)Aged 18 to 100,
≥1-year at General Practice Clinic,
1985 to 2015Emergency hospital admission
(12-, 24-, 36-, 48-, 60-months)Total = 4637,297
Derivation= 3749,932
Validation= 887,365Derivation Avg =7.8%
Validation Avg =10.4%(1) Cox proportional hazards (CPH)
(2) Gradient boosting classifier (GBC)
(3) Random forest (RF)Yes Split Sample (80/20),
5-fold Cross validation
External validation(1) GBC
AUC = 0.80
(no CIs)
(2) GBC
AUC = 0.81
(no CIs)
(3) GBC
AUC = 0.83
(no CIs)Gao et al., 2014
United StatesPredictor Sets:
(1) Hospital characteristics + patient demographic, socioeconomic variables
(2) Model 2 + prior year utilization, cost
(3) Model 3: + 394 HCCsVeterans Affairs (VA) Patients Treated for Ambulatory Care Sensitive Conditions (ACSCs),
2011 to 2012ACSC Hospital Admission
(90-days, 1-year)2987,052 90-day admission =0.73%
1-year admission =2.39%Logistic regression (hierarchical) No Split sample (50/50),
2-fold cross-validation90-day:
(2) c = 0.72
(0.72-0.73)
(3) c = 0.83
(0.82-0.83)
(4) c = 0.86
(0.85-0.86)
1-year:
(3) c = 0.83
(0.83-0.84)Perkins et at., 2013
United StatesAged 18 to 88, General practice patients with chronic kidney disease stage ≥3, hospitalized with heart failure diagnosis,
2004 to 2010Hospital readmission
(30-day)607 19.10% Logistic regression (multivariate) No Bootstrap resampling with 1000 samples c = 0.74 (no CIs) Morawski et al., 2020
United StatesPredictor Sets:
(1) Electronic Health Record (EHR)
(2) EHR + Claims DataAged ≥18, Medicare/Medicaid insured patients, 2013 to 2015 Hospital admission
(6-month)185,388 5% Logistic regression No Split sample (80/20) (2) AUC = 0.84
(0.83-0.85)
(3) AUC = 0.84
(0.84-0.85)Shadmi et al., 2015
IsraelPreadmission Readmission Detection
Model (PREADM)Aged ≥18, general medicine patients admitted to hospital, 2009 to 2010 Hospital readmission
(30-day,
all-cause)Total Admissions = 33,639
Derivation = 22,406
Validation =
11, 23316.80% Logistic regression (multivariate) Yes Split sample (33/66)
External validation: temporalExternal validation c = 0.69 (no CIs) Brisimi et al., 2018
United StatesAll ages,
patients with ≥1 heart-related diagnosis or procedure,
or diabetes mellitus diagnosis,
2001 to 2010Hospitalization (heart-disease related, 12-month)
Hospitalization (diabetes related,
12-month)Heart disease = 45,579
Diabetes = 33,122Heart-disease dataset = 6.7%
Diabetes dataset = 17.0%(1) SVM
(2) Random Forest (RF)
(3) Sparse Logistic Regression (LR)
(4) K-Likelihood Ratio Test (k-LRT)
(5) Joint Clustering and Classification (JCC)Yes Split sample (60/40)
RF: bootstrapping with replacement, cross-validation
ACC: out of sample classificationHeart Data:
Random Forest
AUC = 81.62%
(no CIs)
Diabetes Data:
Random Forest
AUC = 84.53%
(no CIs)Wallace et al., 2016
Ireland(1) Original Pra
(2) Modified PraAged ≥ 70, general medical services, 2010 to 2012 Emergency hospital admission (12-month, all-cause) 862 18% Logistic regression No External validation: Prospective (1) c = 0.65
(0.61-0.7)
(2) c = 0.67
(0.62-0.72)Chenore et al., 2013
United KingdomAll patients, general practice in Devon UK, 2008 to 2010 Emergency hospital admission (12-month) 722,383 Emergency admissions = 5.6% Binary logistic regression No Split sample (80/20) c = 0.78
(0.78-0.78)Donnan et al., 2008
United KingdomPredicting Emergency Admissions Over the Next Year (PEONY) Aged ≥40, general practice, residing in Tayside, Scotland, 1996 to 2004 Emergency hospital admission (12-month) Total = 186,523
Derivation = 90,522
Validation = 96,001Derivation
=7.5%Logistic regression No Split sample (50/50) c = 0.79 (no CIs) Hippisley-Cox et al., 2013
United Kingdom(1) QAdmissions Score
(2) QAdmissions + Hospital Episode Statistics Linked DataAged 18 to 100, ≥1-year general practice,
2010 to 2011Emergency hospital admission
(1- and 2-year)Derivation = 2849,381
Internal Validation = 1340,622
External Validation = 2475,360Derivation = 9.3%
Internal validation = 9.9%
External validation = 9.5%Cox proportional hazards (CPH) No Internal validation (75% of practices)
External validation(1) Women c = 0.76 (0.76-0.77),
Men c = 0.767 (0.76-0.77)
(2) Women c = 0.77 (0.77-0.77),
Men c = 0.77
(0.77-0.77)Watson et al., 2011
United StatesPatients hospitalized and discharged with heart failure diagnosis,
2007 to 2008Hospital readmission (30-day,
all-cause)790 12.8% Logistic regression (multivariate) No None c = 0.67 (no CIs) Zeltzer, 2019
Israel(1) Claims and EMR full
(2) Claims, EMR and Admission fullPatients admitted to hospital overnight, 2016 to 2017 Readmission to hospital (30-day) 144,966 index hospital admissions
(118,510 patients)14.7% Extreme gradient boosting Yes Split sample (83/17)
10-fold cross validation, bootstrapping(1) AUC = 0.70
(0.69-0.71)
(2) AUC = 0.71 (0.70-0.72)Bhavsar et al., 2018
United States(1) Electronic Health Record (HER)
(2) EHR + Neighborhood Socioeconomic Status (SES)Aged ≥18, ≥1 health care encounter in previous year, Durham County resident,
2009 to 2015Inpatient encounters;
Hospitalizations
due to accidents, asthma, influenza, myocardial infarction, strokeDerivation= 90,097
Validation= 122,812Not Reported Random Survival Forest Yes Temporal split Hospitalizations:
Myocardial infarction
(1) c = 0.892
(2) c = 0.892
Stroke:
(1) c = 0.854
(2) c = 0.855
Asthma:
(1) c = 0.752
(2) c = 0.756
Accident:
(1) c = 0.747
(2) c = 0.755
Influenza:
(1) c = 0.562
(2) c = 0.565
Inpatient visit:
(1) c = 0.740
(2) c = 0.742Crane et al., 2010
United StatesAged ≥60, inpatient, primary care community dwelling, assisted living patients,
2005 to 2006Hospitalizations (total number)
(2-year)12,650 Not Reported Logistic Regression No Bootstrapping
(450 samples)AUC = 0.705
(no CIs)Wang et al., 2013
United StatesAged ≥18, Veterans Health Administration (VHA) enrolled patient, 2009 to 2010 (1) First hospitalization (90-day, all-cause)
(2) First hospitalization (1-year, all-cause)4598,408 90-day hospitalization = 2.7%
1-year hospitalization = 8.2%Logistic regression (multinomial) No Split sample (60/40) (1) c = 0.83
(0.83-0.83)
(2) c = 0.81
(0.81-0.81)Model Outcome: Mortality Simon et al., 2018
United StatesAged ≥13, mental health diagnosis at time of outpatient visit at primary care clinic,
2009 to 2015(1) Suicide Mortality
(90-day after mental health specialty visit)
(2) Suicide Mortality
(90-day after primary care visit)19,961,059 visits
(2960,929 patients)Suicide mortality
= 0.04%Logistic regression No Split sample (65/35) (1) c = 0.86 (0.85, 0.88),
(2) c = 0.83 (0.81, 0.85)Glanz et al., 2018
United StatesAged ≥18, ≥3 opioid prescription dates within 90 days, 2006 to 2014 Fatal pharmaceutical opioid and heroin overdoses Derivation
= 42,828
Validation
= 10,708Derivation = 0.03%
Validation = 0.50%Cox proportional hazards regression No Internal validation: Harrell bootstrap resampling
External validation: Geographic validationc = 0.75
(0.70-0.80)DelPozo-Banos et al., 2018
United KingdomAged ≥10, general practice ≥80% EMR data 5 years prior, residents of Wales, 2001 to 2015 Suicide mortality 54, 684 4.76% Artificial neural networks Yes 10 × 10 fold cross-validation Mean Error Rate 26.78% (S.D. 1.46)
Sensitivity = 64.6
Specificity = 81.9Hippisley-Cox, 2017
United KingdomAged 15 to 99, general practice patients with colorectal cancer, 1998 to 2014 (1) Death (all-cause)
(2) Death (colorectal cancer)Derivation=
44,145
Internal validation=
15,214
External validation=
437,821(1) Death (all-cause)
Derivation = 60.9%
Validation = 30.6%
(2) Death (colorectal cancer)
Derivation = 61.8%
Validation = 31.3%Cox-hazard models No Sample split, random (75/25)
External validation(1)
Women
c = 0.778
(0.77-0.78),
Men c = 0.76
(0.76-0.76)
(2)
Women c = 0.80
(0.78-0.81)
Men c = 0.80
(0.79-0.81)Barak-Corren et al., 2017
United StatesAged 10 to 90, inpatient and outpatient care,
1998 to 2012Suicide Mortality 1724,885 Not reported Bayesian classifier models No Simulated prospective approach Female AUC = 0.77 (0.77-0.78)
Male
AUC = 0.76 (0.75-0.77)Bloom et al., 2019
United KingdomPatients with a diagnosis of chronic, obstructive pulmonary disease (COPD),
2010 to 2015Mortality (COPD, 12-month) Derivation = 54,990
Validation = 4931Derivation = 21%
Validation = 29%Cox regression models No Split sample (50/50),
External validationc = 0.67
(0.65-0.70)Jung et al., 2019
United StatesAged 65 to 89, general practice, 2011 to 2014 Mortality
(all-cause,
1-year)349,667 2.1% (1) Logistic regression (LR)
(2) Gradient boosted trees (GBT)Yes Split sample (70/30)
LR: 10-fold cross validationLR AUC =
80.7% (No CIs)
GBT AUC = 84.8% (no CIs)Mathias et al., 2013
United StatesPatients aged ≥50, 2003 to 2008 Mortality (5-year, all-cause) 7463 11% Rotation forest Yes 10-fold cross-validation c = 0.86
(0.85-0.87)Tierney et al., 1997
United StatesAged ≥14, general practice patients with reactive airway disease, 1992 to 1995 Mortality (3-year, all-cause) 1536
Derivation = 752
Validation = 78412% Logistic regression (multivariable) No Split sample (50/50) c = 0.76
(no CIs)O’Mahony et al., 2014
United Kingdom, Spain, Greece, ItalyAged ≥16, European centers, patients with hypertrophic cardiomyopathy Mortality (sudden cardiac death, 5-year) Derivation = 3675
External validation =
15935% Cox regression models (multivariable) No Bootstrapping 200 samples
External validation:
1 health centerc = 0.67
(0.64-0.70)Nijhawan et al., 2012
United StatesHIV-positive patients admitted to hospital,
2006 to 2008Mortality (30-day from index admission discharge) 1509 index hospital admissions
(2476 patients)3% Logistic regression (multivariate) No Split sample (50/50), cross-validation c = 0.79
(0.74-0.84)Zeltzer, 2019
Israel(1) Claims and EMR full
(2) Claims, EMR and Admission fullPatients admitted to hospital overnight, 2016 to 2017 (2) Inpatient mortality
(3) Mortality (12-month, all-cause)144,966 index hospital admissions
(118,510 patients)Inpatient mortality = 2.6%
Mortality (12-month) = 12.5%Extreme gradient boosting Yes Split sample (83/17),
10-fold cross validation, bootstrappingInpatient mortality:
(1) AUC = 0.91 (no CIs)
(2) AUC = 0.95 (0.94-0.96)
1-year all-cause mortality:
(1) AUC = 0.91 (0.92-0.93)
(2) AUC = 0.92 (no CIs)Wang et al., 2013
United StatesAged ≥18, Veterans Health Administration (VHA) enrolled patient, 2009 to 2010 Death (without hospitalization, 90-day)
Death (without hospitalization, 1-year)4598,408 Death (90-day)
= 0.7%
Death (1-year)
= 2.8%Logistic regression (multinomial) No Split sample (60/40) Death within 90-days c = 0.87 (0.86-0.87)
Death within 1 year c = 0.85
(0.85-0.85)†Artificial intelligence methods were defined as the use of a computer to simulate intelligent behavior and more specifically, virtual applications in which algorithms improve learning through experience, with minimal human intervention.83
Abbreviations: CI, confidence interval; AUC, area under the receiver operating characteristic curve.
- Table 2.
Top 20 Predictor Variables Included and Considered in Models Predicting Emergency Department Visits, Hospitalizations, and Mortality
Study Outcomes Emergency Department Visits (Model n = 8) Hospital Admissions (Model n = 27) Mortality (Model n = 16) Number of models which included or excluded the respective variables in final models Predictor Variables Included in Final Model Excluded after Evaluation Included in Final Model Excluded after Evaluation Included in Final Model Excluded after Evaluation Sociodemographic Age 7 (87.5%) 0 (0.0%) 23 (85.2%) 3 (11.1%) 15 (93.8%) 1 (6.25%) Sex 5 (62.5%) 2 (25.0%) 22 (81.5%) 4 (14.8%) 13 (81.3%) 2 (12.5%) Race/Ethnicity 3 (37.5%) 1 (12.5%) 15 (55.6%) 3 (11.1%) 6 (37.5%) 5 (31.3%) Socioeconomic Status* 2 (25.0%) 1 (12.5%) 11 (40.7%) 5 (18.5%) 2 (12.5%) 6 (37.5%) Marital Status 1 (12.5%) 0 (0.0%) 9 (33.3%) 1 (3.7%) 3 (18.8%) 2 (12.5%) Insurance Payer 3 (37.5%) 0 (0.0%) 11 (40.7%) 1 (3.7%) 4 (25.0%) 2 (12.5%) Access to Care* 0 (0.0%) 0 (0.0%) 9 (33.3%) 1 (3.7%) 0 (0.0%) 2 (12.5%) Health Profile Smoking Status 1 (12.5%) 0 (0.0%) 7 (25.9%) 2 (7.4%) 3 (18.8%) 1 (6.3%) BMI, Weight 2 (25.0%) 0 (0.0%) 8 (29.6%) 2 (7.4%) 5 (31.3%) 1 (6.3%) Medical History Medical Diagnoses 8 (100%) 0 (0.0%) 21 (77.8%) 0 (0.0%) 14 (87.5%) 1 (6.3%) Mental Illness 1 (12.5%) 0 (0.0%) 11 (40.7%) 2 (7.4%) 8 (50.0%) 2 (12.5%) Substance Use 0 (0.0%) 0 (0.0%) 4 (14.8%) 3 (11.1%) 5 (31.3%) 4 (25.0%) Medication Use 4 (50.0%) 0 (0.0%) 13 (48.1%) 1 (3.7%) 12 (75.0%) 2 (12.5%) Clinical Findings Laboratory Tests 4 (50.0%) 0 (0.0%) 11 (40.7%) 0 (0.0%) 5 (31.3%) 0 (0.0%) Laboratory Results 3 (37.5%) 0 (0.0%) 10 (37.0%) 0 (0.0%) 9 (56.3%) 0 (0.0%) Vital Signs 1 (12.5%) 0 (0.0%) 8 (29.6%) 1 (3.7%) 4 (25.0%) 2 (12.5%) Procedure History* 2 (25.0%) 1 (12.5%) 6 (22.2%) 1 (3.7%) 7 (43.8%) 1 (6.3%) Health Care Utilization Prior Emergency Department Visits 4 (50.0%) 0 (0.0%) 18 (66.7%) 1 (3.7%) 7 (43.8%) 3 (18.8%) Prior Inpatient Admissions 4 (50.0%) 0 (0.0%) 20 (74.1%) 1 (3.7%) 11 (68.8%) 1 (6.3%) Emergency Admissions 0 (0.0%) 0 (0.0%) 12 (44.4%) 0 (0.0%) 4 (25.0%) 0 (0.0%) Non-Urgent Admissions 0 (0.0%) 0 (0.0%) 4 (14.8%) 2 (7.4%) 4 (25.0%) 1 (6.3%) No. of Inpatient Bed Days 4 (50.0%) 1 (12.5%) 8 (29.6%) 1 (3.7%) 2 (12.5%) 2 (12.5%) Primary Care Visits 2 (25.0%) 0 (0.0%) 11 (40.7%) 4 (14.8%) 3 (18.8%) 6 (37.5%) Outpatient Visits 4 (50.0%) 0 (0.0%) 6 (22.2%) 0 (0.0%) 2 (12.5%) 0 (0.0%) ↵*Variable inclusion examples: Socioeconomic Status: neighborhood income, individual income, deprivation index, zip code proxy measure; Access to Care: Health Region, proximity to health center, access to family doctor; Procedure History: surgical procedures, cardiovascular procedures.
Abbreviation: BMI, Body mass index.
- Table 3.
Methodological Quality Assessment of Included Prediction Models following the Prediction Model Risk of Bias Assessment Tool (PROBAST) Guidelines
Study Participants Predictors Outcome Analysis Overall Frost DW, et al., 2017 − − − ? − Howell P, & Elkin PL., 2019 − ? ? − ? Pearce et al., 2019 − + − + + Hu Z, et al., 2015 − − − − − Hao et al., 2014 + − − + + Bhavsar et al., 2018 − − ? + + Crane et al., 2010 − − + + + Rahimian et al., 2018 − − − − − Gao et al., 2014 − ? + ? + Perkins et at., 2013 − − + + + Morawski et al., 2020 − + − + + Shadmi et al., 2015 − ? + + + Brisimi et al., 2018 − + + ? + Wallace et al., 2016 + − + + + Chenore et al., 2013 − − − + + Donnan et al., 2008 − − − + + Watson et al., 2011 − − − + − Hippisley-Cox et al., 2013 − − − − − Nijhawan et al., 2012 − + − + + Zeltzer et al., 2019 − ? − + + Wang et al., 2013 − − − + + Simon et al., 2018 − + − + + Glanz et al., 2018 − − − ? + DelPozo-Banos et al., 2018 − − − + + Hippisley-Cox, 2017 − − − − − Barak-Corren et al., 2017 − − − + + Bloom et al., 2019 − − − − − Jung et al., 2019 − − − + + Mathias et al., 2013 + − − ? + Tierney et al., 1997 − ? − + + O’Mahony et al., 2014 − + − + + Note: ROB, risk of bias.
− indicates low ROB; + indicates high ROB; and ? indicates unclear ROB.
