Skip to main content

Main menu

  • HOME
  • ARTICLES
    • Current Issue
    • Ahead of Print
    • Archives
    • Abstracts In Press
    • Special Issue Archive
    • Subject Collections
  • INFO FOR
    • Authors
    • Reviewers
    • Call For Papers
    • Subscribers
    • Advertisers
  • SUBMIT
    • Manuscript
    • Peer Review
  • ABOUT
    • The JABFM
    • The Editing Fellowship
    • Editorial Board
    • Indexing
    • Editors' Blog
  • CLASSIFIEDS
  • Other Publications
    • abfm

User menu

Search

  • Advanced search
American Board of Family Medicine
  • Other Publications
    • abfm
American Board of Family Medicine

American Board of Family Medicine

Advanced Search

  • HOME
  • ARTICLES
    • Current Issue
    • Ahead of Print
    • Archives
    • Abstracts In Press
    • Special Issue Archive
    • Subject Collections
  • INFO FOR
    • Authors
    • Reviewers
    • Call For Papers
    • Subscribers
    • Advertisers
  • SUBMIT
    • Manuscript
    • Peer Review
  • ABOUT
    • The JABFM
    • The Editing Fellowship
    • Editorial Board
    • Indexing
    • Editors' Blog
  • CLASSIFIEDS
  • JABFM on Bluesky
  • JABFM On Facebook
  • JABFM On Twitter
  • JABFM On YouTube
Research ArticleResearch Letter

Prospective Validation of a Simple Risk Score to Predict Hospitalization during the Omicron Phase of COVID-19

Mark H. Ebell, Roya Hamadani and Autumn Kieber-Emmons
The Journal of the American Board of Family Medicine March 2024, 37 (2) 324-327; DOI: https://doi.org/10.3122/jabfm.2023.230208R1
Mark H. Ebell
From the Department of Epidemiology and Biostatistics, College of Public Health, University of Georgia, Athens, GA (MHE); University of South Florida Morsani College of Medicine and Lehigh Valley Health Network, Allentown, PA (RH, AKE).
MD, MS
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Roya Hamadani
From the Department of Epidemiology and Biostatistics, College of Public Health, University of Georgia, Athens, GA (MHE); University of South Florida Morsani College of Medicine and Lehigh Valley Health Network, Allentown, PA (RH, AKE).
MPH
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Autumn Kieber-Emmons
From the Department of Epidemiology and Biostatistics, College of Public Health, University of Georgia, Athens, GA (MHE); University of South Florida Morsani College of Medicine and Lehigh Valley Health Network, Allentown, PA (RH, AKE).
MD, MPH
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • Article
  • Figures & Data
  • References
  • Info & Metrics
  • PDF
Loading

Abstract

Introduction: We previously developed a simple risk score with 3 items (age, patient report of dyspnea, and any relevant comorbidity), and in this report validate it in a prospective sample of patients, stratified by vaccination status.

Methods: Data were abstracted from a structured electronic health record of primary care and urgent care 8 patients with COVID-19 in the Lehigh Valley Health Network from 11/21/2021 and 10/31/2022 9 (Omicron variant). Our previously derived risk score was calculated for each of 19,456 patients, 10 and the likelihood of hospitalization was determined. Area under the ROC curve was calculated.

Results: We were able to place 13,239 patients (68%) in a low-risk group with only a 0.16% risk of 13 hospitalization. The moderate risk group with 5622 patients had a 2.2% risk of hospitalization 14 and might benefit from close outpatient follow-up, whereas the high-risk group with only 574 15 patients (2.9% of all patients) had an 8.9% risk of hospitalization and may require further 16 evaluation. Area under the curve was 0.844.

Discussion: We prospectively validated a simple risk score for primary and urgent care patients with COVID1919 that can support outpatient triage decisions around COVID-19.

  • Risk Score
  • Clinical Prediction Rule
  • SARS-CoV-2
  • COVID-19
  • Prognosis
  • Hospitalization

Introduction

We recently developed and internally validated the Lehigh Outpatient COVID Hospitalization (LOCH) risk score using data for patients diagnosed in the primary or urgent care setting.1 It assigned 1.5 points for dyspnea; 1 point for any comorbidity (diabetes mellitus, asthma, COPD, hypertension, cardiovascular disease, chronic kidney or liver disease, or cancer); and 1 point for age 50 to 59, 1.5 points for age 60 to 69, and 2.5 points for age 70 and older. The original risk score classified patients into low (0.22%), moderate (1.3%) and high-risk (8.7%) groups for hospitalization. The area under the receiver operating characteristic curve (AUROCC) was 0.847. We now report on a large prospective validation stratified by vaccination status.

Methods

We used the electronic health record to identify all 19,456 primary and urgent care outpatients in the Lehigh Valley Health Network in Pennsylvania with a positive PCR test for COVID-19 between 11/21/21 and 10/31/22 (Omicron variant). Data for predictor variables used in the risk score were linked to vaccination and hospitalization data. We assumed that any hospitalization within 30 days after a positive test was associated with the COVID-19 infection. Patients were classified as vaccinated if they had a single Janssen/J&J vaccination or 2 or more Pfizer/Moderna vaccination. Analysis used Stata 17 (College Station Texas) for risk score validation and descriptive analysis, and R to estimate the AUROCC (R Foundation for Statistical Computing, Vienna, Austria).

Results

The mean age of the 19,456 patients in our validation population was 45.6 years, 57.7% were female, and 38.4% reported having at least 1 of the relevant comorbidities for the LOCH risk score. Overall, 194 of 19,456 available patients (1.00%) were hospitalized. For persons 70 years and older, 4.1% were hospitalized, compared with 1.9% of those age 60 to 69 years and 0.44% for those younger than 60 years. The rate of hospitalization was significantly lower in vaccinated patients (0.72% vs 1.26%, P < .001). The results of the prospective validation are summarized in Table 1 for all patients as well as stratified by vaccination status. The AUROCC was 0.844 (see Appendix Figure 1).

Appendix Figure 1.
  • Download figure
  • Open in new tab
Appendix Figure 1.

Receiver operating characteristic curve for the prospective validation population of 19,262 patients.

View this table:
  • View inline
  • View popup
Table 1.

Prospective Validation of the LOCH Risk Score, Including Stratification by Vaccination Status. For the Stratified Analysis, Proposed Low (Green), Moderate (Blue), and High Risk (Red) Groups Are Shown

Discussion

We propose that low and moderate risk vaccinated patients and low risk unvaccinated patients on the LOCH risk score (68.2% of all patients, 0.16% hospitalized) can be reassured and do not require special follow-up or other investigations. High risk vaccinated patients (3.42% hospitalized) and moderate risk unvaccinated patients (1.68% hospitalized) may benefit from closer clinical follow-up and additional testing such as chest imaging or c-reactive protein, or referral to the ED based on clinical judgment.2 Finally, high risk unvaccinated patients (8.89% hospitalized) should be referred for further evaluation.

Strengths of our study include prospective validation in outpatients, validation when the Omicron variant was dominant, and excellent accuracy, similar to that of the original study. The predictors are simple enough to memorize, and none requires physical examination, vital signs, or blood tests. This makes it potentially suitable for telehealth, although separate validation in that setting is needed. It is also simpler than some other proposed risk scores for outpatients, with better accuracy.3,4 For example, the OutCoV score has 5 variables, an AUROCC of 0.81, and a higher rate of hospitalization in their low risk group (1.8% vs 0.16% with LOCH).3 Finally, the LOCH risk score identifies 68.2% of patients who do not require further follow-up or testing.

A limitation is that although this was a temporally prospective validation, it is in the same health system as the original risk score. Another limitation is that we were unable to determine the precise reason for hospitalization given the data available to us, and we were unable to include patients testing at home. However, the overall rate of hospitalization in our study is similar to that reported during the Omicron wave.5,6 Previous infection was not available to us due as we used deidentified data. Previous infection may further reduce the likelihood of severe illness and could be incorporated into clinical decision making. Changes in virulence due to new variants SARS-CoV-2 (either greater or lesser virulence) would necessitate recalibration of the clinical recommendations; the lower virulence of Omicron and the availability of vaccination data were key motivators behind this report. Finally, our clinical recommendations for risk groups are based on the clinical expertise of the authors; we are not aware of guidelines or decision threshold studies for patients with COVID-19.

In summary, we have developed and prospectively validated a simple risk score to predict hospitalization in outpatients with COVID-19. We welcome others who could provide an external validation in a new setting. The risk score is available at: https://ebell-projects.shinyapps.io/LehighRiskScore/.

Appendix

Notes

  • This article was externally peer reviewed.

  • Funding: None.

  • Conflict of interest: The authors have no conflicts of interest to report.

  • To see this article online, please go to: http://jabfm.org/content/37/2/324.full.

  • Received for publication May 31, 2023.
  • Revision received October 12, 2023.
  • Accepted for publication October 26, 2023.

References

  1. 1.↵
    1. Ebell MH,
    2. Hamadani R,
    3. Kieber-Emmons A
    . Development and validation of simple risk scores to predict hospitalization in outpatients with COVID-19 Including the Omicron variant. J Am Board Fam Med 2022;35:1058–64. Published online September 16, 2022:jabfm.2022.AP.220056.
    OpenUrlAbstract/FREE Full Text
  2. 2.↵
    1. Bentivegna M,
    2. Ebell MH,
    3. Hulme C
    . Primary care relevant risk factors for adverse outcomes in patients with COVID-19 infection: a systematic review. J Am Board Fam Med 2021 Feb; 34(Suppl): S113-S126.
    OpenUrlAbstract/FREE Full Text
  3. 3.↵
    1. Jacqueries F,
    2. Baggio S,
    3. Gayet-Ageron A,
    4. et al
    . Development and validation of the OUTCoV score to predict the risk of hospitalisation among patients with SARS-CoV-2 infection in ambulatory settings: a prospective cohort study. BMJ Open 2021;11:e044242.
    OpenUrlAbstract/FREE Full Text
  4. 4.↵
    1. Ebell MH,
    2. Cai X,
    3. Lennon R,
    4. et al
    . Development and validation of the COVID-NoLab and COVID-SimpleLab risk scores for prognosis in 6 US health systems. J Am Board Fam Med 2021;34:S127–S135.
    OpenUrlAbstract/FREE Full Text
  5. 5.↵
    Omicron Disease Severity—What We Know So Far. Public Health Ontario. Available at: https://publichealthontario.ca/wwksf-omicron-disease-severity.pdf. Published March 2022 and retrieved 10-3–23.
  6. 6.↵
    1. Catala M,
    2. Coma E,
    3. Alonso S,
    4. et al
    . Transmissibility, hospitalization and intensive care admissions due to omicron compared to delta variants of SARS-DoV-2 in Catalonia: a cohort study and ecological analysis. Front Public Health 2022;10:961030. Available at: https://doi.org/10.3389/fpubh.2022.961030.
    OpenUrl
PreviousNext
Back to top

In this issue

The Journal of the American Board of Family     Medicine: 37 (2)
The Journal of the American Board of Family Medicine
Vol. 37, Issue 2
March-April 2024
  • Table of Contents
  • Table of Contents (PDF)
  • Cover (PDF)
  • Index by author
  • Back Matter (PDF)
  • Front Matter (PDF)
Print
Download PDF
Article Alerts
Sign In to Email Alerts with your Email Address
Email Article

Thank you for your interest in spreading the word on American Board of Family Medicine.

NOTE: We only request your email address so that the person you are recommending the page to knows that you wanted them to see it, and that it is not junk mail. We do not capture any email address.

Enter multiple addresses on separate lines or separate them with commas.
Prospective Validation of a Simple Risk Score to Predict Hospitalization during the Omicron Phase of COVID-19
(Your Name) has sent you a message from American Board of Family Medicine
(Your Name) thought you would like to see the American Board of Family Medicine web site.
CAPTCHA
This question is for testing whether or not you are a human visitor and to prevent automated spam submissions.
4 + 9 =
Solve this simple math problem and enter the result. E.g. for 1+3, enter 4.
Citation Tools
Prospective Validation of a Simple Risk Score to Predict Hospitalization during the Omicron Phase of COVID-19
Mark H. Ebell, Roya Hamadani, Autumn Kieber-Emmons
The Journal of the American Board of Family Medicine Mar 2024, 37 (2) 324-327; DOI: 10.3122/jabfm.2023.230208R1

Citation Manager Formats

  • BibTeX
  • Bookends
  • EasyBib
  • EndNote (tagged)
  • EndNote 8 (xml)
  • Medlars
  • Mendeley
  • Papers
  • RefWorks Tagged
  • Ref Manager
  • RIS
  • Zotero
Share
Prospective Validation of a Simple Risk Score to Predict Hospitalization during the Omicron Phase of COVID-19
Mark H. Ebell, Roya Hamadani, Autumn Kieber-Emmons
The Journal of the American Board of Family Medicine Mar 2024, 37 (2) 324-327; DOI: 10.3122/jabfm.2023.230208R1
Twitter logo Facebook logo Mendeley logo
  • Tweet Widget
  • Facebook Like
  • Google Plus One

Jump to section

  • Article
    • Abstract
    • Introduction
    • Methods
    • Results
    • Discussion
    • Appendix
    • Notes
    • References
  • Figures & Data
  • References
  • Info & Metrics
  • PDF

Related Articles

  • No related articles found.
  • PubMed
  • Google Scholar

Cited By...

  • Research to Improve Clinical Care in Family Medicine: Big Data, Telehealth, Artificial Intelligence, and More
  • Google Scholar

More in this TOC Section

  • Timing of Certification Stage Completion Associated with Subsequent Certification Exam Outcomes Among Board-Certified Family Physicians
  • Interpersonal Continuity of Care May Help Delay Progression to Type 2 Diabetes
  • Impact of Point of Care Hemoglobin A1c Testing on Time to Therapeutic Intervention
Show more Research Letter

Similar Articles

Keywords

  • Risk Score
  • Clinical Prediction Rule
  • SARS-CoV-2
  • COVID-19
  • Prognosis
  • Hospitalization

Navigate

  • Home
  • Current Issue
  • Past Issues

Authors & Reviewers

  • Info For Authors
  • Info For Reviewers
  • Submit A Manuscript/Review

Other Services

  • Get Email Alerts
  • Classifieds
  • Reprints and Permissions

Other Resources

  • Forms
  • Contact Us
  • ABFM News

© 2025 American Board of Family Medicine

Powered by HighWire