Abstract
Introduction: Outpatient physicians need guidance to support decisions regarding hospitalization of COVID-19 patients and how closely to follow outpatients. Thus, we sought to develop and validate simple risk scores to predict hospitalization for outpatients with COVID-19 that do not require laboratory testing or imaging.
Methods: We identified outpatients 12 years and older who had a positive polymerase chain reaction test for SARS-CoV-2. Logistic regression was used to derive a risk score in patients presenting before March, 2021, and it was validated in a cohort presenting from March to September 2021 and an Omicron cohort from December, 2021 to January, 2022.
Results: Overall, 4.0% of 5843 outpatients in the early derivation cohort (before 3/1/21), 4.2% of 3806 outpatients in the late validation cohort, and 1.2% in an Omicron cohort were hospitalized. The base risk score included age, dyspnea, and any comorbidity. Other scores added fever, respiratory rate and/or oxygen saturation. All had very good overall accuracy (AUC 0.85–0.87) and classified about half of patients into a low-risk group with < 1% hospitalization risk. Hospitalization rates in the Omicron cohort were 0.22%, 1.3% and 8.7% for the base score. Two externally derived risk scores identified more low risk patients, but with a higher overall risk of hospitalization than our novel risk scores.
Conclusions: A simple risk score suitable for outpatient and telehealth settings can classify over half of COVID-19 outpatients into a very low risk group with a 0.22% hospitalization risk in the Omicron cohort. The Lehigh Outpatient COVID Hospitalization (LOCH) risk score is available online as a free app: https://ebell-projects.shinyapps.io/LehighRiskScore/.
- Clinical Decision-Making
- Clinical Prediction Rule
- COVID-19
- Family Medicine
- Infectious Diseases
- Mobile Applications
- Outpatients
- Pennsylvania
- Retrospective Studies
- SARS-CoV-2 Variants
Introduction
Most patients with COVID-19 are initially evaluated in the outpatient setting, and a decision must be made whether to manage them as outpatients or to hospitalize them. For those managed initially in the outpatient setting, a decision must also be made whether they require close follow-up or whether they are at a low risk of deterioration and can be told to simply follow-up as needed. A risk score designed for outpatients could help support clinician decision making around hospitalization and follow-up.
However, while many COVID-19 risk scores have been proposed, almost all have been developed and validated to predict mortality in hospitalized patients.1 These include the 4C risk score,2 the ABCS risk score,3 and the COVID-GRAM risk score.4 Almost all these risk scores require laboratory tests and in some cases imaging, which is often unavailable in the outpatient or telehealth settings. An exception is the COVID-NoLab risk score for inpatient mortality, which requires only oxygen saturation, age, and the respiratory rate.5 In addition, the OutCoV score was derived and internally validated in Switzerland to predict the likelihood of hospitalization among outpatients and does not require laboratory testing.6 It consists of 5 easily ascertained variables: age, fever, dyspnea, hypertension, and chronic respiratory disease. However, it has not been externally validated.
We assembled a dataset of consecutive patients who had been diagnosed with COVID-19 in an outpatient primary care or urgent care clinic in a large health system in Pennsylvania. The primary objective was to develop and temporally validate simple risk scores that do not require laboratory tests or imaging to predict the likelihood of hospitalization or death in outpatients with COVID-19. A secondary objective was to prospectively validate 2 previously developed mortality risk scores that did not require laboratory testing, the OutCoV score developed in outpatients and the COVID-NoLab risk scores developed originally for inpatients.5,6
Methods
Population Studied
Lehigh Valley Hospital (LVH) is a not-for-profit academic community hospital, which is a legal entity of Lehigh Valley Health Network (LVHN). LVHN has grown to become the largest health care provider in the Pennsylvania Lehigh Valley region, with 9 not-for-profit hospital campuses, nearly 2000 inpatient beds, more than 275 physician practice locations, 25 health centers and 20 ExpressCARE locations, and includes the region’s only Children's Hospital. The LVH hub is located in the Lehigh Valley region of eastern Pennsylvania, approximately 90 miles from New York City, 60 miles from Philadelphia, and 20 miles from the New Jersey border, and encompasses Allentown, the third largest city in Pennsylvania. According to the 2020 Census, Allentown is home to 125,845 people. Of this population, 52% identify as Latino, 14.7% as Black/African American, and 2.9% as Asian.7
The electronic health record was used to identify all outpatients diagnosed with COVID-19 by polymerase chain reaction (PCR) test between March 13, 2020 and September 30, 2021. Outpatients with an outpatient PCR diagnosis of COVID-19 from December 20, 2021 to January 7, 2022 were additionally identified to prospectively validate the risk score in patients during the Omicron phase of the pandemic. Data were obtained from query of LVHN’s Epic electronic medical record and included all primary care and ExpressCARE (urgent care) outpatient visits during the time window that included a COVID-19 PCR order from that visit, and a subsequent positive COVID-19 PCR test result. Patients with missing data for age, respiratory rate, or oxygen saturation were excluded from the population used to derive and internally validate the clinical prediction rules, as were patients less than 12 years of age.
Analytic Plan
Univariate analysis was performed using a c2 test for dichotomous variables and Student’s t test for normally distributed continuous variables to identify individual patient characteristics associated with hospitalization or death. Potential cut-off points for continuous variables were selected by inspection of histograms and contingency tables. All comorbidities associated with hospitalization were combined into a single “any comorbidity” variable. Patients with hospitalization within the study period but before the date of the outpatient visit were excluded.
Patients presenting before March 1, 2021 were used to derive the risk score (early cohort), and patients presenting from March 1, 2021 onwards were used to validate it (late cohort). The risk score was also validated in a cohort presenting when Omicron was the predominant variant, with patients presenting between December 20, 2021 to January 7, 2022. That end date was chosen to assure at least 2 weeks of follow-up for all patients. Logistic regression was performed with hospitalization or death as the dependent variable, and all patient characteristics significantly associated with hospitalization or death in univariate analysis as the independent variables. We evaluated multiple models using different cutoffs for oxygen saturation, age and respiratory rate in the derivation group to identify the models with the highest area under the receiver operating characteristic curve.
The final models were converted to simple point scores by dividing each β-coefficient by the smallest β-coefficient and rounding the resulting number. Low, moderate and high-risk groups were identified by inspection of a table showing the likelihood of hospitalization associated with each risk score. The primary goal was to identify as large as possible a low-risk group with a less than 1% likelihood of hospitalization. The resulting risk scores were then validated in the late cohort of outpatients and in the Omicron cohort. Analysis was performed using Stata v. 17 (StataCorp, College Station, TX). The study was approved by the University of Georgia Human Subjects Office (PROJECT00004060) and LVHN staff received approval to use the University of Georgia determination.
Results
Data were available for 13418 outpatients diagnosed with COVID-19 between March 20, 2020 and September 30, 2021. After excluding patients under age 12 and patients with missing data, the final dataset used to derive and internally validated the risk scores included 9649 outpatients with COVID-19. The early cohort had 5843 patients with a 28-day hospitalization rate of 4.0% while the late cohort had 3806 patients with a 28-day hospitalization rate of 4.2%. A total of 641 patients in the entire cohort were hospitalized and of that number, 55 died following their outpatient COVID-19 diagnosis, all who were hospitalized. Of the hospitalized patients, 89 of 641 (13.9%) were hospitalized on the same day as their outpatient visit. The characteristics of patients who were hospitalized or died and of those who were not in the derivation and internal validation cohorts are summarized in Table 1. Increasing age, increasing respiratory rate, lower oxygen saturation, a complaint of dyspnea, and all comorbidities were associated with an increased likelihood of hospitalization.
Characteristics of Included Patients in the Derivation and Internal Validation Cohort
The Omicron cohort had 6138 patients with at least 2 weeks of follow-up, and a 1.9% overall rate of hospitalization. As in the earlier cohorts, the median age of not hospitalized patients in the Omicron cohort was 41 years (interquartile range [IQR] 29 to 54 years), while for hospitalized patients it was 63 years (IQR 51 to 73 years).
Four logistic regression models were developed based on the assumption that in telehealth settings temperature, oxygen saturation, respiratory rate, or all 3 may not be available. Model A used age, dyspnea, and the presence of a comorbidity; Model B added fever; Model C added fever and a respiratory rate > 20/min; and Model D added fever and oxygen saturation ≤ 95%. A model adding fever, elevated respiratory rate and low oxygen saturation had excessive collinearity for the respiratory rate predictor and is not shown. The models are shown in Table 2, along with corresponding points assigned to each variable based on the β-coefficient. The 4 models had very good discrimination based on the area under the receiver operating characteristic curve, with a range from 0.847 to 0.0.872. ROC curves for Models A and D are shown in Figure 1.
Multivariate Models, Showing Assignment of Points Based on the Beta-Coefficients
Receiver operating characteristic curves.
The performance of each risk score in the early (derivation), late (validation), and Omicron cohorts is summarized in Table 3. Each model classified at least half of patients in the low-risk group, with a 0.34% to 0.47% risk of hospitalization or death in the early derivation cohort and a 0.79% to 0.89% risk of hospitalization in the later validation cohort.
Classification Accuracy of 3 Novel Risk Scores and 2 Externally Derived Risk Scores in the Early Derivation, Late Validation, and Omicron Cohorts
The overall likelihood of hospitalization in the Omicron cohort was 1.2%, lower than in the earlier cohorts. In the Omicron cohort, the hospitalization rates for Risk Score A were 0.22% in the low-risk group, 1.3% in the moderate risk group, and 8.7% in the high-risk group. Again, over 60% of patients were classified in the low-risk group. The other risk scores performed similarly well in the Omicron cohort. As a sensitivity analysis we also calculated the risk of hospitalizations for each risk group after excluding 16 of 74 hospitalized patients who were admitted on the same day as the outpatient visit. The low-, moderate-, and high-risk groups were 0.22%, 0.80%, and 6.1%, respectively.
The classification accuracy of the OutCoV and COVID-NoLab risk scores in the late cohort is also shown in Table 3. These risk scores identified more patients in the low-risk groups, but also had higher rates of hospitalization in that group, 3.8% to 4.0% in the early cohort and 2.8% in the late cohort. In the Omicron cohort, both risk scores identified low-risk groups with hospitalization rates of 0.6% to 0.71% but classified only a small number of patients in the high-risk groups.
Discussion
We developed 4 simple risk scores for hospitalization in outpatients with COVID-19, and prospectively validated them in the same population. The base Risk Score A included age, dyspnea and the presence of a comorbidity, with Risk Score B adding fever, Risk Score C adding fever and the respiratory rate, and Risk Score D adding fever and the oxygen saturation. All 4 models all had very good accuracy based on the AUC (0.847 to 0.872).
In the validation group, the risk scores classified at least half of patients being in a low-risk group with a likelihood of hospitalization less than 1% in both the late validation cohort (largely Delta variant) and less than 0.5% in the Omicron cohort. These patients could be managed initially as outpatients with guidance to contact their primary care physician in the event of worsening symptom. The moderate risk groups for the 4 risk scores had a 5.2% to 6.2% likelihood of hospitalization in the Delta cohort and 1.3% to 1.7% in the Omicron cohort, which is similar to that for the population as a whole. These patients could also be managed as outpatients initially, but with closer follow-up. For example, they could be given an oxygen saturation monitor with daily check-in from a nurse or other clinician to assess their status. Finally, about 9% of patients fell into a high-risk group with a 19.2% to 24.3% likelihood of hospitalization in the Delta cohort and 8.7% to 14.0% in the Omicron cohort. It may be appropriate to refer these patients to the emergency department for more detailed evaluation including imaging, laboratory testing, and a period of observation.
This kind of evidence-based strategy would help avoid unnecessary visits to the emergency department by low and moderate risk patients, especially during surges in cases, and would identify moderate risk patients who could initially be managed as outpatients but who should have closer follow-up than the low-risk patients. This approach rationalizes care and helps allocate limited resources most efficiently. It is reassuring that our risk score had similar discrimination during waves dominated by 2 different variants (Delta and Omicron), which bodes well for its performance with future variants. Our ongoing ability to rapidly pull and analyze data will allow us to validate this rule in future surges and with different variants.
We also evaluated the accuracy of 2 previously developed clinical prediction rules. The COVID-NoLab score was originally designed for inpatient risk prediction, but very few patients in our sample had either oxygen saturation < 93% or respiratory rate > 30 breaths/minute. While it identified very few high-risk patients, it classified 62% of outpatients in the Omicron cohort into a low-risk group of whom 0.6% were hospitalized. The OutCoV risk score similarly identified few high-risk patients but did classify 75% of patients into a low-risk group with a 0.71% risk of hospitalization. Both scores therefore identify more low-risk outpatients than the novel risk scores, but at the cost of a higher risk of hospitalization in those low-risk groups (0.6% to 0.71% vs 0.22%). Depending on the tolerance for risk, this may be an acceptable trade-off.
Strengths and Limitations
Our study had several notable strengths. We had access to a large, real-world population of outpatients being managed in primary and urgent care settings. The risk scores developed in the earliest 60% of patients validated very well in the later cohort of patients, with similar classification accuracy and in a cohort of patients recruited while Omicron was the dominant variant. Most importantly, the risk scores all validated well in a large cohort of patients during a period when Omicron was the dominant variant. The risk scores are simple, and even the simplest had very good overall accuracy and was able to identify clinically useful low, moderate, and high-risk groups. While telehealth patients were not included in our sample, all our clinical variables (especially for Risk Score A) can be easily obtained in that setting.
The study also had limitations. Data were obtained retrospectively from the electronic health record, and we were unable to directly verify diagnoses. In addition, visits to the LVHN network for COVID-19 concerns were available both through telehealth (video or phone encounters) as well as outpatient and urgent care visits during the initial months of the pandemic and then also during surge windows. However, our sample did not include some individuals in the health system who chose to do telehealth visits for their symptoms and COVID-19 testing needs. Selection bias regarding patient characteristics of those individuals who did come to an in-person visit could have affected the risk scores. However, it is more likely that those who sought in-person outpatient or urgent care visits were sicker than the overall universe of COVID-19 patients, and hence the risk score population may be biased toward those with a higher risk of hospitalization. It will also be important to evaluate the risk score in vaccinated versus unvaccinated populations, possibly incorporating lack of vaccination as an additional risk factor. However, we did not have access to full vaccination records, only those administered within the LVHN. Finally, we only had 2 weeks of follow-up for Omicron patients but are continuing to gather data and will update the online risk calculator as it emerges.
Conclusion
We have developed and temporally validated simple risk scores for outpatients with COVID-19 that identify a large proportion of outpatients who have very low risk of hospitalization (0.22% in the low-risk group for the Omicron cohort). They also classify about 1 in 9 Omicron patients into a high-risk group who have a substantial likelihood of eventual hospitalization and warrant very close follow-up or referral to the emergency department for further evaluation. Risk score A, which we call the Lehigh Outpatient COVID Hospitalization (LOCH) risk score is available online as a simple interactive app.
Notes
This article was externally peer reviewed.
Funding: None.
Conflicts of interest: None.
To see this article online, please go to: http://jabfm.org/content/00/0/000.full.
- Received for publication February 11, 2022.
- Revision received June 10, 2022.
- Accepted for publication June 14, 2022.
In this issue
Jump to section
Related Articles
Cited By...
- No citing articles found.