Study, Country | Model Name (If Applicable) | Study Population, Setting, Time Period | Outcome(s) | Sample Size | Outcome Rate | Modelling Methods | Artificial Intelligence Methods Used† | Validation Approach | Validated Modelc-statistic*(95% CI) |
---|---|---|---|---|---|---|---|---|---|
Model Outcome: Emergency Department Visits | |||||||||
Frost DW, et al., 2017 Canada | Aged ≥50, ≥1-year General Practice, 2011 to 2012 | Frequent ED Use (≥3 ED visits) (12-month) | Derivation: 21,680 Validation: 895 | Training Cohort = 5.7% | Logistic Regression | Yes | Internal: Split sample | Validation c = 0.71 (no CIs) | |
Howell P, & Elkin PL., 2019 United States | Not Reported, 2017 to 2018 | Emergency Department Visit (12-month) | 2991 | Not Reported | Random Forest | Yes | External validation only | Validation c = 0.83 (no CIs) | |
Pearce et al., 2019 Australia | POLAR Diversion | All patients, General Practice, 2010 to 2015 | Emergency Department Visit (12-month) | 37,665 | ≥1 ED Visit = 23% | Support Vector Machine | Yes | Internal: Split sample 10-Fold Cross-validation | 0 to 30 days: Sensitivity = 68%; PPV = 73.7% 31 to 365 days: Sensitivity = 10%; PPV = 36.8% |
Hu Z, et al., 2015 United States | All patients, 2012 to 2013 | Emergency Department Visit (6-month, all-cause) | Retrospective cohort: 829,641 Prospective cohort: 875,979 | Retrospective = 11.48%, Prospective = 11.37% | Survival Forest Decision Trees | Yes | External validation: Prospective | Retrospective c = 0.74 (no CIs) Prospective c = 0.73 (no CIs) | |
Hao et al., 2014 United States | All patients, 2012 to 2013 | Emergency Department Revisit (30-days) | (ED encounters) Derivation: 293,461 Prospective validation: 193,886 | Retrospective = 19.4% Prospective = 20.5% | Decision Tree | Yes | Split sample External validation: Prospective | Retrospective c = 0.71 (no CIs) Prospective c = 0.70 (no CIs) | |
Bhavsar et al., 2018 United States | (1) Electronic Health Record (HER) (2) EHR + Neighborhood Socioeconomic Status (SES) | Aged ≥18, ≥1 health care encounter in previous year, Durham County resident, 2009 to 2015 | Emergency Department Visit | Derivation: 90,097 Validation: 122,812 | Not Reported | Random Survival Forest | Yes | Temporal Split | (1) c = 0.75 (no CIs) (2) c = 0.75 (no CIs) |
Crane et al., 2010 United States | Aged ≥60, inpatient, primary care community dwelling, assisted living patients, 2005 to 2006 | Emergency Room Visit (2-year) | 12,650 | Not Reported | Logistic Regression | No | Bootstrapping (450 samples) | AUC = 0.64 (no CIs) | |
Model Outcome: Hospitalizations | |||||||||
Rahimian et al., 2018 United Kingdom | Predictor Sets (1) QAdmissions (QA) (2) QAdmissions+ (QA+) (3) Temporal (T) | Aged 18 to 100, ≥1-year at General Practice Clinic, 1985 to 2015 | Emergency hospital admission (12-, 24-, 36-, 48-, 60-months) | Total = 4637,297 Derivation= 3749,932 Validation= 887,365 | Derivation Avg =7.8% Validation Avg =10.4% | (1) Cox proportional hazards (CPH) (2) Gradient boosting classifier (GBC) (3) Random forest (RF) | Yes | Split Sample (80/20), 5-fold Cross validation External validation | (1) GBC AUC = 0.80 (no CIs) (2) GBC AUC = 0.81 (no CIs) (3) GBC AUC = 0.83 (no CIs) |
Gao et al., 2014 United States | Predictor Sets: (1) Hospital characteristics + patient demographic, socioeconomic variables (2) Model 2 + prior year utilization, cost (3) Model 3: + 394 HCCs | Veterans Affairs (VA) Patients Treated for Ambulatory Care Sensitive Conditions (ACSCs), 2011 to 2012 | ACSC Hospital Admission (90-days, 1-year) | 2987,052 | 90-day admission =0.73% 1-year admission =2.39% | Logistic regression (hierarchical) | No | Split sample (50/50), 2-fold cross-validation | 90-day: (2) c = 0.72 (0.72-0.73) (3) c = 0.83 (0.82-0.83) (4) c = 0.86 (0.85-0.86) 1-year: (3) c = 0.83 (0.83-0.84) |
Perkins et at., 2013 United States | Aged 18 to 88, General practice patients with chronic kidney disease stage ≥3, hospitalized with heart failure diagnosis, 2004 to 2010 | Hospital readmission (30-day) | 607 | 19.10% | Logistic regression (multivariate) | No | Bootstrap resampling with 1000 samples | c = 0.74 (no CIs) | |
Morawski et al., 2020 United States | Predictor Sets: (1) Electronic Health Record (EHR) (2) EHR + Claims Data | Aged ≥18, Medicare/Medicaid insured patients, 2013 to 2015 | Hospital admission (6-month) | 185,388 | 5% | Logistic regression | No | Split sample (80/20) | (2) AUC = 0.84 (0.83-0.85) (3) AUC = 0.84 (0.84-0.85) |
Shadmi et al., 2015 Israel | Preadmission Readmission Detection Model (PREADM) | Aged ≥18, general medicine patients admitted to hospital, 2009 to 2010 | Hospital readmission (30-day, all-cause) | Total Admissions = 33,639 Derivation = 22,406 Validation = 11, 233 | 16.80% | Logistic regression (multivariate) | Yes | Split sample (33/66) External validation: temporal | External validation c = 0.69 (no CIs) |
Brisimi et al., 2018 United States | All ages, patients with ≥1 heart-related diagnosis or procedure, or diabetes mellitus diagnosis, 2001 to 2010 | Hospitalization (heart-disease related, 12-month) Hospitalization (diabetes related, 12-month) | Heart disease = 45,579 Diabetes = 33,122 | Heart-disease dataset = 6.7% Diabetes dataset = 17.0% | (1) SVM (2) Random Forest (RF) (3) Sparse Logistic Regression (LR) (4) K-Likelihood Ratio Test (k-LRT) (5) Joint Clustering and Classification (JCC) | Yes | Split sample (60/40) RF: bootstrapping with replacement, cross-validation ACC: out of sample classification | Heart Data: Random Forest AUC = 81.62% (no CIs) Diabetes Data: Random Forest AUC = 84.53% (no CIs) | |
Wallace et al., 2016 Ireland | (1) Original Pra (2) Modified Pra | Aged ≥ 70, general medical services, 2010 to 2012 | Emergency hospital admission (12-month, all-cause) | 862 | 18% | Logistic regression | No | External validation: Prospective | (1) c = 0.65 (0.61-0.7) (2) c = 0.67 (0.62-0.72) |
Chenore et al., 2013 United Kingdom | All patients, general practice in Devon UK, 2008 to 2010 | Emergency hospital admission (12-month) | 722,383 | Emergency admissions = 5.6% | Binary logistic regression | No | Split sample (80/20) | c = 0.78 (0.78-0.78) | |
Donnan et al., 2008 United Kingdom | Predicting Emergency Admissions Over the Next Year (PEONY) | Aged ≥40, general practice, residing in Tayside, Scotland, 1996 to 2004 | Emergency hospital admission (12-month) | Total = 186,523 Derivation = 90,522 Validation = 96,001 | Derivation =7.5% | Logistic regression | No | Split sample (50/50) | c = 0.79 (no CIs) |
Hippisley-Cox et al., 2013 United Kingdom | (1) QAdmissions Score (2) QAdmissions + Hospital Episode Statistics Linked Data | Aged 18 to 100, ≥1-year general practice, 2010 to 2011 | Emergency hospital admission (1- and 2-year) | Derivation = 2849,381 Internal Validation = 1340,622 External Validation = 2475,360 | Derivation = 9.3% Internal validation = 9.9% External validation = 9.5% | Cox proportional hazards (CPH) | No | Internal validation (75% of practices) External validation | (1) Women c = 0.76 (0.76-0.77), Men c = 0.767 (0.76-0.77) (2) Women c = 0.77 (0.77-0.77), Men c = 0.77 (0.77-0.77) |
Watson et al., 2011 United States | Patients hospitalized and discharged with heart failure diagnosis, 2007 to 2008 | Hospital readmission (30-day, all-cause) | 790 | 12.8% | Logistic regression (multivariate) | No | None | c = 0.67 (no CIs) | |
Zeltzer, 2019 Israel | (1) Claims and EMR full (2) Claims, EMR and Admission full | Patients admitted to hospital overnight, 2016 to 2017 | Readmission to hospital (30-day) | 144,966 index hospital admissions (118,510 patients) | 14.7% | Extreme gradient boosting | Yes | Split sample (83/17) 10-fold cross validation, bootstrapping | (1) AUC = 0.70 (0.69-0.71) (2) AUC = 0.71 (0.70-0.72) |
Bhavsar et al., 2018 United States | (1) Electronic Health Record (HER) (2) EHR + Neighborhood Socioeconomic Status (SES) | Aged ≥18, ≥1 health care encounter in previous year, Durham County resident, 2009 to 2015 | Inpatient encounters; Hospitalizations due to accidents, asthma, influenza, myocardial infarction, stroke | Derivation= 90,097 Validation= 122,812 | Not Reported | Random Survival Forest | Yes | Temporal split | Hospitalizations: Myocardial infarction (1) c = 0.892 (2) c = 0.892 Stroke: (1) c = 0.854 (2) c = 0.855 Asthma: (1) c = 0.752 (2) c = 0.756 Accident: (1) c = 0.747 (2) c = 0.755 Influenza: (1) c = 0.562 (2) c = 0.565 Inpatient visit: (1) c = 0.740 (2) c = 0.742 |
Crane et al., 2010 United States | Aged ≥60, inpatient, primary care community dwelling, assisted living patients, 2005 to 2006 | Hospitalizations (total number) (2-year) | 12,650 | Not Reported | Logistic Regression | No | Bootstrapping (450 samples) | AUC = 0.705 (no CIs) | |
Wang et al., 2013 United States | Aged ≥18, Veterans Health Administration (VHA) enrolled patient, 2009 to 2010 | (1) First hospitalization (90-day, all-cause) (2) First hospitalization (1-year, all-cause) | 4598,408 | 90-day hospitalization = 2.7% 1-year hospitalization = 8.2% | Logistic regression (multinomial) | No | Split sample (60/40) | (1) c = 0.83 (0.83-0.83) (2) c = 0.81 (0.81-0.81) | |
Model Outcome: Mortality | |||||||||
Simon et al., 2018 United States | Aged ≥13, mental health diagnosis at time of outpatient visit at primary care clinic, 2009 to 2015 | (1) Suicide Mortality (90-day after mental health specialty visit) (2) Suicide Mortality (90-day after primary care visit) | 19,961,059 visits (2960,929 patients) | Suicide mortality = 0.04% | Logistic regression | No | Split sample (65/35) | (1) c = 0.86 (0.85, 0.88), (2) c = 0.83 (0.81, 0.85) | |
Glanz et al., 2018 United States | Aged ≥18, ≥3 opioid prescription dates within 90 days, 2006 to 2014 | Fatal pharmaceutical opioid and heroin overdoses | Derivation = 42,828 Validation = 10,708 | Derivation = 0.03% Validation = 0.50% | Cox proportional hazards regression | No | Internal validation: Harrell bootstrap resampling External validation: Geographic validation | c = 0.75 (0.70-0.80) | |
DelPozo-Banos et al., 2018 United Kingdom | Aged ≥10, general practice ≥80% EMR data 5 years prior, residents of Wales, 2001 to 2015 | Suicide mortality | 54, 684 | 4.76% | Artificial neural networks | Yes | 10 × 10 fold cross-validation | Mean Error Rate 26.78% (S.D. 1.46) Sensitivity = 64.6 Specificity = 81.9 | |
Hippisley-Cox, 2017 United Kingdom | Aged 15 to 99, general practice patients with colorectal cancer, 1998 to 2014 | (1) Death (all-cause) (2) Death (colorectal cancer) | Derivation= 44,145 Internal validation= 15,214 External validation= 437,821 | (1) Death (all-cause) Derivation = 60.9% Validation = 30.6% (2) Death (colorectal cancer) Derivation = 61.8% Validation = 31.3% | Cox-hazard models | No | Sample split, random (75/25) External validation | (1) Women c = 0.778 (0.77-0.78), Men c = 0.76 (0.76-0.76) (2) Women c = 0.80 (0.78-0.81) Men c = 0.80 (0.79-0.81) | |
Barak-Corren et al., 2017 United States | Aged 10 to 90, inpatient and outpatient care, 1998 to 2012 | Suicide Mortality | 1724,885 | Not reported | Bayesian classifier models | No | Simulated prospective approach | Female AUC = 0.77 (0.77-0.78) Male AUC = 0.76 (0.75-0.77) | |
Bloom et al., 2019 United Kingdom | Patients with a diagnosis of chronic, obstructive pulmonary disease (COPD), 2010 to 2015 | Mortality (COPD, 12-month) | Derivation = 54,990 Validation = 4931 | Derivation = 21% Validation = 29% | Cox regression models | No | Split sample (50/50), External validation | c = 0.67 (0.65-0.70) | |
Jung et al., 2019 United States | Aged 65 to 89, general practice, 2011 to 2014 | Mortality (all-cause, 1-year) | 349,667 | 2.1% | (1) Logistic regression (LR) (2) Gradient boosted trees (GBT) | Yes | Split sample (70/30) LR: 10-fold cross validation | LR AUC = 80.7% (No CIs) GBT AUC = 84.8% (no CIs) | |
Mathias et al., 2013 United States | Patients aged ≥50, 2003 to 2008 | Mortality (5-year, all-cause) | 7463 | 11% | Rotation forest | Yes | 10-fold cross-validation | c = 0.86 (0.85-0.87) | |
Tierney et al., 1997 United States | Aged ≥14, general practice patients with reactive airway disease, 1992 to 1995 | Mortality (3-year, all-cause) | 1536 Derivation = 752 Validation = 784 | 12% | Logistic regression (multivariable) | No | Split sample (50/50) | c = 0.76 (no CIs) | |
O’Mahony et al., 2014 United Kingdom, Spain, Greece, Italy | Aged ≥16, European centers, patients with hypertrophic cardiomyopathy | Mortality (sudden cardiac death, 5-year) | Derivation = 3675 External validation = 1593 | 5% | Cox regression models (multivariable) | No | Bootstrapping 200 samples External validation: 1 health center | c = 0.67 (0.64-0.70) | |
Nijhawan et al., 2012 United States | HIV-positive patients admitted to hospital, 2006 to 2008 | Mortality (30-day from index admission discharge) | 1509 index hospital admissions (2476 patients) | 3% | Logistic regression (multivariate) | No | Split sample (50/50), cross-validation | c = 0.79 (0.74-0.84) | |
Zeltzer, 2019 Israel | (1) Claims and EMR full (2) Claims, EMR and Admission full | Patients admitted to hospital overnight, 2016 to 2017 | (2) Inpatient mortality (3) Mortality (12-month, all-cause) | 144,966 index hospital admissions (118,510 patients) | Inpatient mortality = 2.6% Mortality (12-month) = 12.5% | Extreme gradient boosting | Yes | Split sample (83/17), 10-fold cross validation, bootstrapping | Inpatient mortality: (1) AUC = 0.91 (no CIs) (2) AUC = 0.95 (0.94-0.96) 1-year all-cause mortality: (1) AUC = 0.91 (0.92-0.93) (2) AUC = 0.92 (no CIs) |
Wang et al., 2013 United States | Aged ≥18, Veterans Health Administration (VHA) enrolled patient, 2009 to 2010 | Death (without hospitalization, 90-day) Death (without hospitalization, 1-year) | 4598,408 | Death (90-day) = 0.7% Death (1-year) = 2.8% | Logistic regression (multinomial) | No | Split sample (60/40) | Death within 90-days c = 0.87 (0.86-0.87) Death within 1 year c = 0.85 (0.85-0.85) |
†Artificial intelligence methods were defined as the use of a computer to simulate intelligent behavior and more specifically, virtual applications in which algorithms improve learning through experience, with minimal human intervention.83
Abbreviations: CI, confidence interval; AUC, area under the receiver operating characteristic curve.