Table 1.

Characteristics of Studies and Model Development

Study, CountryModel Name (If Applicable)Study Population, Setting, Time PeriodOutcome(s)Sample SizeOutcome RateModelling MethodsArtificial Intelligence Methods UsedValidation ApproachValidated Modelc-statistic*(95% CI)
Model Outcome: Emergency Department Visits
 Frost DW, et al.,  2017
 Canada
Aged ≥50, ≥1-year General Practice, 2011 to 2012Frequent ED Use (≥3 ED visits)
(12-month)
Derivation: 21,680
Validation: 895
Training Cohort = 5.7%Logistic RegressionYesInternal:
Split sample
Validation
c =0.71 (no CIs)
 Howell P, & Elkin  PL., 2019
 United States
Not Reported, 2017 to 2018Emergency Department Visit
(12-month)
2991Not ReportedRandom ForestYesExternal validation onlyValidation
c =0.83 (no CIs)
 Pearce et al., 2019
 Australia
POLAR DiversionAll patients, General Practice, 2010 to 2015Emergency Department Visit
(12-month)
37,665≥1 ED Visit = 23%Support Vector MachineYesInternal: Split sample
10-Fold Cross-validation
0 to 30 days:
Sensitivity = 68%; PPV = 73.7%
31 to 365 days:
Sensitivity = 10%; PPV = 36.8%
 Hu Z, et al., 2015
 United States
All patients,
2012 to 2013
Emergency Department Visit
(6-month,
all-cause)
Retrospective cohort: 829,641
Prospective cohort: 875,979
Retrospective = 11.48%,
Prospective = 11.37%
Survival Forest Decision TreesYesExternal validation: ProspectiveRetrospective
c =0.74 (no CIs)
Prospective
c =0.73 (no CIs)
 Hao et al., 2014
 United States
All patients,
2012 to 2013
Emergency Department Revisit
(30-days)
(ED encounters)
Derivation: 293,461
Prospective validation: 193,886
Retrospective = 19.4%
Prospective = 20.5%
Decision TreeYesSplit sample
External validation:
Prospective
Retrospective
c =0.71 (no CIs)
Prospective
c = 0.70 (no CIs)
 Bhavsar et al., 2018
 United States
(1) Electronic Health Record (HER)
(2) EHR + Neighborhood Socioeconomic Status (SES)
Aged ≥18, ≥1 health care encounter in previous year, Durham County resident,
2009 to 2015
Emergency Department VisitDerivation: 90,097
Validation: 122,812
Not ReportedRandom Survival ForestYesTemporal Split(1) c =0.75
(no CIs)
(2) c =0.75
(no CIs)
 Crane et al., 2010
 United States
Aged ≥60, inpatient, primary care community dwelling, assisted living patients,
2005 to 2006
Emergency Room Visit (2-year)12,650Not ReportedLogistic RegressionNoBootstrapping
(450 samples)
AUC = 0.64
(no CIs)
Model Outcome: Hospitalizations
 Rahimian et al.,  2018
 United Kingdom
Predictor Sets
(1) QAdmissions (QA)
(2) QAdmissions+
(QA+)
(3) Temporal (T)
Aged 18 to 100,
≥1-year at General Practice Clinic,
1985 to 2015
Emergency hospital admission
(12-, 24-, 36-, 48-, 60-months)
Total = 4637,297
Derivation= 3749,932
Validation= 887,365
Derivation Avg =7.8%
Validation Avg =10.4%
(1) Cox proportional hazards (CPH)
(2) Gradient boosting classifier (GBC)
(3) Random forest (RF)
YesSplit Sample (80/20),
5-fold Cross validation
External validation
(1) GBC
AUC = 0.80
(no CIs)
(2) GBC
AUC = 0.81
(no CIs)
(3) GBC
AUC = 0.83
(no CIs)
 Gao et al., 2014
 United States
Predictor Sets:
(1) Hospital characteristics + patient demographic, socioeconomic variables
(2) Model 2 + prior year utilization, cost
(3) Model 3: + 394 HCCs
Veterans Affairs (VA) Patients Treated for Ambulatory Care Sensitive Conditions (ACSCs),
2011 to 2012
ACSC Hospital Admission
(90-days, 1-year)
2987,05290-day admission =0.73%
1-year admission =2.39%
Logistic regression (hierarchical)NoSplit sample (50/50),
2-fold cross-validation
90-day:
(2) c =0.72
(0.72-0.73)
(3) c =0.83
(0.82-0.83)
(4) c =0.86
(0.85-0.86)
1-year:
(3) c =0.83
(0.83-0.84)
 Perkins et at., 2013
 United States
Aged 18 to 88, General practice patients with chronic kidney disease stage ≥3, hospitalized with heart failure diagnosis,
2004 to 2010
Hospital readmission
(30-day)
60719.10%Logistic regression (multivariate)NoBootstrap resampling with 1000 samplesc = 0.74 (no CIs)
 Morawski et al.,  2020
 United States
Predictor Sets:
(1) Electronic Health Record (EHR)
(2) EHR + Claims Data
Aged ≥18, Medicare/Medicaid insured patients, 2013 to 2015Hospital admission
(6-month)
185,3885%Logistic regressionNoSplit sample (80/20)(2) AUC = 0.84
(0.83-0.85)
(3) AUC = 0.84
(0.84-0.85)
 Shadmi et al., 2015
 Israel
Preadmission Readmission Detection
Model (PREADM)
Aged ≥18, general medicine patients admitted to hospital, 2009 to 2010Hospital readmission
(30-day,
all-cause)
Total Admissions = 33,639
Derivation = 22,406
Validation =
11, 233
16.80%Logistic regression (multivariate)YesSplit sample (33/66)
External validation: temporal
External validation c =0.69 (no CIs)
 Brisimi et al., 2018
 United States
All ages,
patients with ≥1 heart-related diagnosis or procedure,
or diabetes mellitus diagnosis,
2001 to 2010
Hospitalization (heart-disease related, 12-month)
Hospitalization (diabetes related,
12-month)
Heart disease = 45,579
Diabetes = 33,122
Heart-disease dataset = 6.7%
Diabetes dataset = 17.0%
(1) SVM
(2) Random Forest (RF)
(3) Sparse Logistic Regression (LR)
(4) K-Likelihood Ratio Test (k-LRT)
(5) Joint Clustering and Classification (JCC)
YesSplit sample (60/40)
RF: bootstrapping with replacement, cross-validation
ACC: out of sample classification
Heart Data:
Random Forest
AUC = 81.62%
(no CIs)
Diabetes Data:
Random Forest
AUC = 84.53%
(no CIs)
 Wallace et al., 2016
 Ireland
(1) Original Pra
(2) Modified Pra
Aged ≥ 70, general medical services, 2010 to 2012Emergency hospital admission (12-month, all-cause)86218%Logistic regressionNoExternal validation: Prospective(1) c =0.65
(0.61-0.7)
(2) c =0.67
(0.62-0.72)
 Chenore et al.,  2013
 United Kingdom
All patients, general practice in Devon UK, 2008 to 2010Emergency hospital admission (12-month)722,383Emergency admissions = 5.6%Binary logistic regressionNoSplit sample (80/20)c =0.78
(0.78-0.78)
 Donnan et al., 2008
 United Kingdom
Predicting Emergency Admissions Over the Next Year (PEONY)Aged ≥40, general practice, residing in Tayside, Scotland, 1996 to 2004Emergency hospital admission (12-month)Total = 186,523
Derivation = 90,522
Validation = 96,001
Derivation
=7.5%
Logistic regressionNoSplit sample (50/50)c =0.79 (no CIs)
 Hippisley-Cox  et al., 2013
 United Kingdom
(1) QAdmissions Score
(2) QAdmissions + Hospital Episode Statistics Linked Data
Aged 18 to 100, ≥1-year general practice,
2010 to 2011
Emergency hospital admission
(1- and 2-year)
Derivation = 2849,381
Internal Validation = 1340,622
External Validation = 2475,360
Derivation = 9.3%
Internal validation = 9.9%
External validation = 9.5%
Cox proportional hazards (CPH)NoInternal validation (75% of practices)
External validation
(1) Women c = 0.76 (0.76-0.77),
Men c = 0.767 (0.76-0.77)
(2) Women c = 0.77 (0.77-0.77),
Men c = 0.77
(0.77-0.77)
 Watson et al., 2011
 United States
Patients hospitalized and discharged with heart failure diagnosis,
2007 to 2008
Hospital readmission (30-day,
all-cause)
79012.8%Logistic regression (multivariate)NoNonec =0.67 (no CIs)
 Zeltzer, 2019
 Israel
(1) Claims and EMR full
(2) Claims, EMR and Admission full
Patients admitted to hospital overnight, 2016 to 2017Readmission to hospital (30-day)144,966 index hospital admissions
(118,510 patients)
14.7%Extreme gradient boostingYesSplit sample (83/17)
10-fold cross validation, bootstrapping
(1) AUC = 0.70
(0.69-0.71)
(2) AUC = 0.71 (0.70-0.72)
 Bhavsar et al., 2018
 United States
(1) Electronic Health Record (HER)
(2) EHR + Neighborhood Socioeconomic Status (SES)
Aged ≥18, ≥1 health care encounter in previous year, Durham County resident,
2009 to 2015
Inpatient encounters;
Hospitalizations
due to accidents, asthma, influenza, myocardial infarction, stroke
Derivation= 90,097
Validation= 122,812
Not ReportedRandom Survival ForestYesTemporal splitHospitalizations:
Myocardial infarction
(1) c = 0.892
(2) c = 0.892
Stroke:
(1) c = 0.854
(2) c = 0.855
Asthma:
(1) c = 0.752
(2) c = 0.756
Accident:
(1) c = 0.747
(2) c = 0.755
Influenza:
(1) c = 0.562
(2) c = 0.565
Inpatient visit:
(1) c = 0.740
(2) c = 0.742
 Crane et al., 2010
 United States
Aged ≥60, inpatient, primary care community dwelling, assisted living patients,
2005 to 2006
Hospitalizations (total number)
(2-year)
12,650Not ReportedLogistic RegressionNoBootstrapping
(450 samples)
AUC = 0.705
(no CIs)
 Wang et al., 2013
 United States
Aged ≥18, Veterans Health Administration (VHA) enrolled patient, 2009 to 2010(1) First hospitalization (90-day, all-cause)
(2) First hospitalization (1-year, all-cause)
4598,40890-day hospitalization = 2.7%
1-year hospitalization = 8.2%
Logistic regression (multinomial)NoSplit sample (60/40)(1) c = 0.83
(0.83-0.83)
(2) c = 0.81
(0.81-0.81)
Model Outcome: Mortality
 Simon et al., 2018
 United States
Aged ≥13, mental health diagnosis at time of outpatient visit at primary care clinic,
2009 to 2015
(1) Suicide Mortality
(90-day after mental health specialty visit)
(2) Suicide Mortality
(90-day after primary care visit)
19,961,059 visits
(2960,929 patients)
Suicide mortality
= 0.04%
Logistic regressionNoSplit sample (65/35)(1) c = 0.86 (0.85, 0.88),
(2) c = 0.83 (0.81, 0.85)
 Glanz et al., 2018
 United States
Aged ≥18, ≥3 opioid prescription dates within 90 days, 2006 to 2014Fatal pharmaceutical opioid and heroin overdosesDerivation
= 42,828
Validation
= 10,708
Derivation = 0.03%
Validation = 0.50%
Cox proportional hazards regressionNoInternal validation: Harrell bootstrap resampling
External validation: Geographic validation
c = 0.75
(0.70-0.80)
 DelPozo-Banos  et al., 2018
 United Kingdom
Aged ≥10, general practice ≥80% EMR data 5 years prior, residents of Wales, 2001 to 2015Suicide mortality54, 6844.76%Artificial neural networksYes10 × 10 fold cross-validationMean Error Rate 26.78% (S.D. 1.46)
Sensitivity = 64.6
Specificity = 81.9
 Hippisley-Cox,  2017
 United Kingdom
Aged 15 to 99, general practice patients with colorectal cancer, 1998 to 2014(1) Death (all-cause)
(2) Death (colorectal cancer)
Derivation=
44,145
Internal validation=
15,214
External validation=
437,821
(1) Death (all-cause)
Derivation = 60.9%
Validation = 30.6%
(2) Death (colorectal cancer)
Derivation = 61.8%
Validation = 31.3%
Cox-hazard modelsNoSample split, random (75/25)
External validation
(1)
Women
c = 0.778
(0.77-0.78),
Men c = 0.76
(0.76-0.76)
(2)
Women c = 0.80
(0.78-0.81)
Men c = 0.80
(0.79-0.81)
 Barak-Corren  et al., 2017
 United States
Aged 10 to 90, inpatient and outpatient care,
1998 to 2012
Suicide Mortality1724,885Not reportedBayesian classifier modelsNoSimulated prospective approachFemale AUC = 0.77 (0.77-0.78)
Male
AUC = 0.76 (0.75-0.77)
 Bloom et al., 2019
 United Kingdom
Patients with a diagnosis of chronic, obstructive pulmonary disease (COPD),
2010 to 2015
Mortality (COPD, 12-month)Derivation = 54,990
Validation = 4931
Derivation = 21%
Validation = 29%
Cox regression modelsNoSplit sample (50/50),
External validation
c = 0.67
(0.65-0.70)
 Jung et al., 2019
 United States
Aged 65 to 89, general practice, 2011 to 2014Mortality
(all-cause,
1-year)
349,6672.1%(1) Logistic regression (LR)
(2) Gradient boosted trees (GBT)
YesSplit sample (70/30)
LR: 10-fold cross validation
LR AUC =
80.7% (No CIs)
GBT AUC = 84.8% (no CIs)
 Mathias et al., 2013
 United States
Patients aged ≥50, 2003 to 2008Mortality (5-year, all-cause)746311%Rotation forestYes10-fold cross-validationc  =0.86
(0.85-0.87)
 Tierney et al., 1997
 United States
Aged ≥14, general practice patients with reactive airway disease, 1992 to 1995Mortality (3-year, all-cause)1536
Derivation = 752
Validation = 784
12%Logistic regression (multivariable)NoSplit sample (50/50)c = 0.76
(no CIs)
 O’Mahony et al.,  2014
 United Kingdom,  Spain, Greece,  Italy
Aged ≥16, European centers, patients with hypertrophic cardiomyopathyMortality (sudden cardiac death, 5-year)Derivation = 3675
External validation =
1593
5%Cox regression models (multivariable)NoBootstrapping 200 samples
External validation:
1 health center
c = 0.67
(0.64-0.70)
 Nijhawan et al., 2012
 United States
HIV-positive patients admitted to hospital,
2006 to 2008
Mortality (30-day from index admission discharge)1509 index hospital admissions
(2476 patients)
3%Logistic regression (multivariate)NoSplit sample (50/50), cross-validationc =0.79
(0.74-0.84)
 Zeltzer, 2019
 Israel
(1) Claims and EMR full
(2) Claims, EMR and Admission full
Patients admitted to hospital overnight, 2016 to 2017(2) Inpatient mortality
(3) Mortality (12-month, all-cause)
144,966 index hospital admissions
(118,510 patients)
Inpatient mortality = 2.6%
Mortality (12-month) = 12.5%
Extreme gradient boostingYesSplit sample (83/17),
10-fold cross validation, bootstrapping
Inpatient mortality:
(1) AUC = 0.91 (no CIs)
(2) AUC = 0.95 (0.94-0.96)
1-year all-cause mortality:
(1) AUC = 0.91 (0.92-0.93)
(2) AUC = 0.92 (no CIs)
 Wang et al., 2013
 United States
Aged ≥18, Veterans Health Administration (VHA) enrolled patient, 2009 to 2010Death (without hospitalization, 90-day)
Death (without hospitalization, 1-year)
4598,408Death (90-day)
= 0.7%
Death (1-year)
= 2.8%
Logistic regression (multinomial)NoSplit sample (60/40)Death within 90-days c = 0.87 (0.86-0.87)
Death within 1 year c = 0.85
(0.85-0.85)
  • †Artificial intelligence methods were defined as the use of a computer to simulate intelligent behavior and more specifically, virtual applications in which algorithms improve learning through experience, with minimal human intervention.83

  • Abbreviations: CI, confidence interval; AUC, area under the receiver operating characteristic curve.