Abstract
Background: The EvidenceNOW initiative provided smaller primary care practices with external support interventions to implement quality improvement strategies focused on cardiovascular disease prevention. This manuscript reports effectiveness of EvidenceNOW interventions in improving quality metrics.
Methods: Seven regional Cooperatives delivered external support interventions (practice facilitation, health information technology support to assist with audit and feedback, performance benchmarking, learning collaboratives, and establishing community linkages) to 1278 smaller primary care practices. Outcomes included proportion of eligible patients meeting Centers for Medicaid and Medicare Services-specified ABCS metrics, that is, Aspirin for those at risk of ischemic vascular disease; achieving target Blood pressure among hypertensives; prescribing statin for those with elevated Cholesterol, diabetes, or increased cardiovascular disease risk; and screening for Smoking and providing cessation counseling. An event study compared prepost changes in outcomes among intervention practices and a difference-in-differences design compared intervention practices to 688 external comparison practices.
Results: Mean baseline outcomes ranged from 61.5% (cholesterol) to 64.9% (aspirin). In the event study, outcomes improved significantly (aspirin: +3.39 percentage points, 95% CI, 0.61–6.17; blood pressure: +1.59, 95% CI, 0.12–3.06; cholesterol: +4.43, 95% CI, 0.33–8.53; smoking: +7.33, 95% CI, 4.70–9.96). Difference-in-differences estimates were similar in magnitude but statistically significant for smoking alone. Preintervention trends were significant for smoking, but parallel-trends tests were not significant.
Conclusions: EvidenceNOW Cooperatives improved cardiovascular prevention quality metrics among small and medium sized primary care practices across the US. While estimated improvements were small, they reflected average changes across a large and diverse sample of practices.
- Cardiovascular Diseases
- Delivery of Health Care
- Hyperlipidemia
- Hypertension
- Quality Improvement
- Primary Health Care
Introduction
More people die from cardiovascular disease in the United States (US) than any other disease.1 The 2019 American College of Cardiology and American Heart Association (ACC/AHA) Guideline on the Primary Prevention of Cardiovascular Disease emphasize improving delivery of the ABCS of heart health, which included prescribing Aspirin for those at risk for ischemic vascular disease, achieving target Blood pressures for those with hypertension, prescribing statins to those with diabetes, high Cholesterol, or at risk for atherosclerotic cardiovascular disease, and screening and cessation counseling for Smoking.2,3 These strategies have been shown to reduce cardiovascular disease morbidity and mortality.3,4 Despite substantial clinical evidence, however, national ABCS rates remain suboptimal.5
To address this, the Centers for Medicare and Medicaid Services (CMS) and Centers for Disease Control and Prevention (CDC) launched the Million Hearts initiative in 2011.6⇓–8 Its goal is to prevent 1 million heart attacks by 2022.6 Most Americans receive primary and secondary preventive care for cardiovascular disease in small to medium-sized primary care practices in their communities.9 Thus, improving ABCS rates among smaller practices is crucial to realizing the Million Hearts initiative.
Quality improvement strategies shown to increase ABCS adoption in health systems are available but are often not used widely as standard practice in primary care. For instance, the CDC’s Best Practices Guide for Cardiovascular Disease Prevention highlight that the use of team-based care, clinical decision support systems, and self-measured blood pressure monitoring is cost-effective in improving clinical outcomes.10 However, primary care practices experience significant barriers to routine use of these strategies resulting in suboptimal ABCS rates and approaches to improve care delivery such as the Comprehensive Primary Care Initiative or Patient-Centered Medial Home have not consistently shown improvements in quality of care.11⇓⇓–14
Recognizing this gap in ABCS delivery and limited quality improvement resources among smaller practices,15 the Agency for Health care Research and Quality (AHRQ) funded a national initiative called EvidenceNOW encouraging health systems and health professionals to focus on improving the ABCS and also practice capacity.16,17 EvidenceNOW funded 7 regional “cooperatives” (the term AHRQ used to reference grantees) spanning 12 US states to provide external support interventions at scale to primary care practices in their region with the goal of implementing quality improvement strategies targeting the ABCS. In this manuscript, we report the overall effectiveness of EvidenceNOW interventions on practice-level change in performance on cardiovascular disease (ABCS) clinical quality metrics.
Methods
EvidenceNOW Cooperatives enrolled over 1500 practices across 7 regions. Appendix Figure 1 shows the geographic distribution of EvidenceNOW Cooperatives. AHRQ also funded an external evaluation, Evaluating System Change and Learning to Take Evidence to Scale (ESCALATES), led by our team (Cohen, PI).
EvidenceNOW consort diagram.
Interventions
In its Request for Applications, AHRQ sought applications that proposed multi-component interventions providing quality improvement support to small and medium-sized primary care practices. It strongly encouraged practice facilitation as a central and unifying intervention. Thus, EvidenceNOW Cooperatives established infrastructure to deliver external support to practices recruited in their regions. They all implemented practice facilitation. In addition, each cooperative implemented a distinct package of interventions focused on improving the ABCS of heart health, including providing health information technology support to assist practices with audit and feedback, performance benchmarking, creating learning collaboratives, and establishing community linkages. While Cooperatives targeted interventions at improving ABCS outcomes, practices varied in the extent to which they implemented these interventions based on their motivations, priority, and resources. Each Cooperative evaluated implementation and effectiveness of its own intervention.18⇓⇓⇓⇓⇓–24 We (ESCALATES) evaluated the effectiveness of the overall initiative.17
Study Designs
Because Cooperatives lacked internal practice controls, we used 2 quasi-experimental designs to evaluate the overall effectiveness of interventions: (1) an event study including only intervention practices and (2) a difference-in-differences approach comparing intervention practices with an external comparison group. The study period for both approaches ranged from the last quarter of 2015 (baseline) to the second quarter of 2018.
The event study assessed changes in ABCS relative to intervention start (the event) among all EvidenceNOW practices. The baseline period was the quarter just before the start of intervention for each practice cohort (fourth quarter of 2015 for the first cohort to the fourth quarter of 2016 for the fifth cohort). Across Cooperatives, interventions started in 5 cohorts between the first quarter of 2016 and the first quarter of 2017 and ended on a rolling basis over the next 12 months.
The difference-in-differences design evaluated change in outcomes for the years 2015 (preintervention), 2016 (during intervention) and 2017 (postintervention), comparing EvidenceNOW practices to an external comparison group of primary care practices provided by the DARTNet institute.25 The baseline period for this analysis was the fourth quarter of 2015. Appendix Figure A2 depicts the outcome measurement time periods.
The difference-in-differences design is generally preferred over other types of pre-post designs (such as the event study) because it does not rely on the assumption that outcomes would not have changed in the absence of the intervention. However, the design requires outcome trends of comparison practices (ie, Dartnet) to accurately reflect those of intervention practices (ie, EvidenceNOW) in the absence of any intervention.26,27 Because of potential data quality issues for some measures in the Dartnet data, we compared findings across both designs to facilitate interpretation.
Data Sources and Practice Samples
We specified practice samples from 2 data sources: (1) EvidenceNOW (intervention practices), and (2) DARTNet (external comparison group).
Intervention Practices
Cooperatives initially assessed 7773 practices for eligibility and randomized 1795 for interventions (Figure 1). While eligibility criteria varied somewhat, most Cooperatives excluded larger practices (defined by AHRQ as ≥10 FTE clinicians) and those that did not have an electronic health record. However, 2 cooperatives that were experiencing challenges recruiting included a few practices with 11 to 15 FTE clinicians. Additional exclusion criteria were specific to Cooperatives’ regional context (eg, using a specific electronic health record (EHR) system) and recruitment strategy (eg, from specific practice networks) and described in detail elsewhere.18⇓⇓⇓⇓⇓–24 In addition, for our sensitivity analyses described below, we further excluded federally qualified health center (FQHC) practices from the EvidenceNOW sample for comparability with the external comparison group.
ABCS outcome metrics were collected from all practices. We excluded practices with unreliable outcome data, which included practices that (1) did not provide outcomes for at least 1 quarter before the intervention, (2) did not provide outcomes following the intervention, and (3) had low denominator values during one or more study quarters, defined as fewer than 10 patients in the denominator for the aspirin measure and fewer than 30 patients in the denominator for other measures. Of the 1795 practices randomized to interventions, 1278 (71%) were retained in the final sample. Specific to each outcome, the final sample had between 872 (cholesterol) and 1244 (blood pressure) EvidenceNOW practices.
External Comparison Practices
Because EvidenceNOW Cooperatives’ study designs did not include internal control practices, we assembled a group of practices external to EvidenceNOW to serve as a comparison group. These were derived from the DARTNet Institute, a real-time EHR data repository. The institute provides secure, deidentified datasets of clinical quality metrics from 12 distinct practice-based research networks that includes over 1000 primary care practices. We applied the same exclusion criteria to DartNet practices.25 The final sample had between 377 (blood pressure) and 688 (smoking) DartNet practices.
As a sensitivity analysis, we defined a subsample of DARTNet practices located in states represented within EvidenceNOW because they might more accurately reflect regional outcome trends. For this sensitivity analysis (but not the main analysis), we excluded federally qualified health center (FQHC) practices from the EvidenceNOW subsample, because DARTNet did not include FQHC practices located in EvidenceNOW states. The EvidenceNOW subsample had between 580 (cholesterol) to 776 (aspirin) practices; the DARTNet subsample had between 75 (blood pressure) to 117 (smoking) practices.
Measures
The ESCALATES team, in collaboration with Cooperatives, harmonized ABCS measure definitions and practice characteristics. Details about data collection are described in previous studies.28,29
Outcomes
We used the Centers for Medicaid and Medicare Services (CMS) specifications for the ABCS outcomes in use in 2015 (Table 1). Each outcome measured the proportion of eligible patients prescribed the recommended treatment in each practice. Cooperatives obtained practice-level performance on ABCS outcomes from practices’ electronic health records (EHRs) through quality reports, health information exchange reports, or, in some cases, chart audits, and shared with the ESCALATES team.
Specification of ABCS Clinical Quality Outcome Measures
Covariates
Practice characteristics were obtained through surveys administered to participating practices by Cooperatives; these included: practice size (solo, 2 to 5, 6 to 10, 11 to 15); practice ownership (clinician-owned, hospital/health system-owned, FQHC, and other); practice location (rural, large town, suburban, urban); whether a practice participated in demonstration programs; whether a practice was in a medically underserved area; health insurance distribution (percent of patients with commercial insurance, uninsured, on Medicaid, on Medicare, dually eligible for Medicaid and Medicare, or on other insurance) and race/ethnicity distribution (percent of white, black, Hispanic patients) of practice patient panel. DARTNet data included practice ownership, race and ethnicity and practice characteristics.
Statistical Analysis
We used multivariable linear regression models for both designs. The event study included binary indicators for quarters relative to the last quarter before start of intervention, ranging from -4 to 9, with 0 indicating the quarter intervention began. These indicators measured change in the outcome relative to the last preintervention quarter. We, then adjusted for practice characteristics and included binary indicators for Cooperatives, intervention cohorts and the quarter of first data submission in the regression. After model estimation, we averaged coefficients for the fifth to eighth postintervention quarter, representing changes in the second year of the intervention.
The difference-in-differences approach included indicator variables for postintervention calendar quarters (first quarter of 2016 to second quarter of 2018), a binary indicator for the intervention group (EvidenceNOW compared with DARTNet), and interaction terms between time indicators and the intervention group indicator. Interaction terms measured changes in outcomes of EvidenceNOW practices compared with DARTNet practices. We adjusted for practice characteristics and included binary indicators for intervention cohorts and the quarter of first data submission in the regression. We then averaged coefficients for the year 2017, which approximated the second intervention year. We used bootstrapping with 1000 repetitions to cluster standard errors at the Cooperative level (for the event-study) or at the state levels (for the EvidenceNOW-DARTNet comparison).
To test the assumption that outcomes would not have changed without intervention in the event study, we calculated average preintervention outcome change based on estimates for the fourth to second preintervention period. We note that most practices did not submit data starting 4 quarters before intervention begin due to a staggered recruitment process. For the differences-in-differences analysis, we assessed differential trends between EvidenceNOW and DARTNet practices for the first 2 quarters of 2016. We selected these two quarters because we lacked a true preintervention period for this analysis and because interventions were unlikely to have immediate effects on outcomes. When preintervention trends or parallel-trends test were significant, we estimated trend-adjusted models.30,31 Trend adjusted models assumed that trends before the intervention would have continued at the same rate during the intervention period absent treatment. All analyses were conducted using R version 3.5.1. The Institutional Review Board of Oregon Health & Science University approved this study.
Results
Most EvidenceNOW practices (83.4%) had ≤10 clinicians (Table 2). While EvidenceNOW’s focus was on smaller practices, about 10% of the practices recruited were slightly larger (11 to 15 clinicians). Sixty percent of practices were in urban core areas. DARTNet practices were more than twice as likely to be health system-owned compared with EvidenceNOW practices.
Characteristics of the EvidenceNOW and DARTNet Practice Sample
Very few DARTNet practices (3%) across all states were FQHCs (none of the DARTNet practices in EvidenceNOW states were FQHCs). EvidenceNOW and DARTNet practices had a similar percentage of white patients. EvidenceNOW practices had a higher proportion of black and Hispanic patients compared with DARTNet practices. Mean baseline ABCS performance among EvidenceNOW practices were 64.9% (aspirin), 63.6% (blood pressure), 61.5% (cholesterol), and 62.0% (smoking) with wide variation across practices. Among DARTNet practices, mean baseline ABCS levels were 29.6% (aspirin), 64.7% (blood pressure), 39.4% (cholesterol), and 12.6% (smoking). Except for blood pressure, outcome levels for DARTNet practices at baseline were much lower than EvidenceNOW practices.
In the event study, all ABCS outcomes improved. (Table 3). Average outcome change 2 years after the intervention (between baseline and the fifth to eighth quarter) was +3.39 percentage points (95% CI: 0.61 to 6.17, P < .05) for aspirin, +1.59 percentage points (95% CI: 0.12 to 3.06, P < .05) for blood pressure, +4.43 percentage points (95% CI: 0.33 to 8.53, P < .05) for cholesterol and +7.33 (95% CI: 4.70 to 9.96, P < .001) for smoking. All outcomes improved gradually during the first 4 postintervention quarters, and then remained at the same level for aspirin, blood pressure and cholesterol, while further increasing for smoking (Figure 2). Preintervention trend estimates for the event study were not statistically significant, except for a positive pretrend for the smoking outcome in the event study (pretrend estimate: 2.03, P < .01, Table 3). Adjusting for this pretrend resulted in a significant negative estimate for smoking (see Appendix Table B5).
Event study estimates.
Baseline ABCS Levels and Estimates of Effectiveness of EvidenceNOW
In the difference-in-differences design, parallel-trends test was not statistically significant for any of the outcomes. As shown in Table 3, EvidenceNOW practices showed higher performance on all measures as compared with DARTNet controls but with varying levels of statistical significance (Aspirin +3.75 percentage points, P = .0875; Blood pressure +2.76 percentage points, P = .0556; Cholesterol +3.87 percentage points, P = .2990; and Smoking +8.32 percentage points, P = .0027). Outcomes for DARTNet practices between 2016 and 2017 did not show any clear trends (Appendix Figure A2) and aspirin and cholesterol outcomes for the first and second quarter of 2018 were much larger than for other quarters. Estimates from sensitivity analysis using the subsample of DARTNet practices from EvidenceNOW regions were similar in magnitude to the full sample, with 2 coefficients (blood pressure and smoking) reaching significance (Table 3).
Discussion
EvidenceNOW was one of the largest primary care quality improvement initiatives focused on improving cardiovascular disease preventive care by providing external support interventions to smaller practices. Overall effectiveness of these interventions in improving ABCS outcomes is best evaluated from trials that incorporate a robust comparison group of practices, either through randomization or through external comparisons. EvidenceNOW projects were pragmatic trials with varying designs; thus, in the absence of control practices, quasi-experimental study designs can help in making causal inference.32 We employed 2 such designs to evaluate effectiveness of interventions and the results were largely complementary. We found that EvidenceNOW interventions were associated with small improvements in the ABCS quality metrics 2 years after the intervention (event study). When comparing with an external group of practices (difference in differences), EvidenceNOW practices demonstrated improvements for all quality metrics, but statistically significant improvements were observed only for the smoking screening/cessation counseling metric.
Overall, there was consistency of findings across the 2 designs except for the smoking outcome metric. The difference in differences design showed an almost 8 percentage point improvement in smoking among EvidenceNOW practices compared with Dartnet comparison practices. Importantly, the parallel trends test was nonsignificant, suggesting that our estimate of the smoking improvement might be appropriate and valid. Thus, observed estimates of smoking outcome improvement were likely associated with intervention effects. While the event study also showed a 7.3 percentage point improvement, we observed a positive pretrend, possibly suggesting a secular effect. We did not observe such a trend in the Dartnet comparison practices and other studies have also not demonstrated such improvements in smoking and other outcomes over time.33⇓–35 It is possible that practices motivated to improve smoking outcomes may have self-selected into EvidenceNOW or implemented changes in anticipation of receiving the intervention. If that were the case, observed improvements may not be completely attributable to EvidenceNOW interventions.
Supporting practices by providing facilitation, expert consultation, and health information technology in implementing quality improvements has been shown to improve a range of practice outcomes.36⇓⇓⇓–40 However, most of these findings come from small practice samples enrolled in controlled research studies.19,20,22 This study is one of the first to assess effectiveness in improving quality metrics at a large scale when diverse primary care practices were provided with external support through primary care extension infrastructure. We observed small improvements in all ABCS outcomes, with the smoking metric demonstrating statistical significance for both research designs. While observed changes were small, it is important to note that benefits in cardiovascular disease outcomes may be observed, for instance, even with minor reductions in blood pressure or just by appropriately prescribing statins to those at increased risk.1⇓–3
The Affordable Care Act mandated that Health and Human Services establish a primary care extension program to provide practices with external support. However, the program was never funded.41,42 Our study suggests that such an extension program could help practices improve ABCS clinical quality metrics. While improvements in the outcome metrics were small, they were observed across a large and diverse sample of practices suggesting that external support interventions provided to practices may have population-level benefits perhaps even for other chronic conditions such as diabetes and mental health. If such benefits are to be adopted more widely and sustained, more research is needed identifying strategies to mitigate the influence of practice-level barriers to implementing quality improvements, such as competing demands and a challenging reimbursement environment for primary care.
Using two statistical methods including the inclusion of an external comparison group of practices is a strength of this evaluation of pragmatic EvidenceNOW interventions. However, our study also had several limitations. First, practices had different capabilities in extracting data on ABCS quality metrics from EHRs, which might have affected data quality. We minimized this concern by working collaboratively with cooperatives to ensure data quality and by excluding practices with unreliable data. Second, effect estimates might be affected by practices’ first data submission, either relative to the time since the intervention or relative to the baseline quarter. We addressed this concern by including indicators for intervention cohorts and the first quarter of data submission in regressions. Third, practices submitted baseline data at different points before intervention start. Therefore, preintervention trends may not have been precisely estimated. Fourth, the DARTNet sample was relatively small limiting power of our difference-in-differences analysis. Unfortunately, we did not have access to other suitable external data sources and had to exclude some practices due to data quality concerns. Fourth, there may be potential for self-selection bias as practices that chose to enroll may be more likely to be motivated to make changes and engage in facilitation. The relatively high mean baseline outcome levels may, in part, be because of this reason. However, we also observed wide variation in baseline ABCS rates across practices demonstrating that EvidenceNOW Cooperatives also reached practices with lower performance on these metrics. Fifth, data on other confounders (eg, age, race/ethnicity) were not available and therefore not included in our analysis. Finally, DARTNet practices showed large changes in outcomes in 2018 that likely reflect data quality or coding issues. Although we were unable to identify the underlying reason for those changes, our analysis is based on change in outcomes using 2017 postintervention data and thus, these issues did not affect our findings.
It is important to note that external support strategies implemented in EvidenceNOW were based on the best clinical evidence available at the time of the initiative. Therefore, changes in guidelines (such as for aspirin use in 2021) would not be reflected in this work.
Notwithstanding these limitations, our study provides valuable lessons for future studies seeking to evaluate effectiveness of external support interventions in primary care. First, they should construct a valid comparison group of practices that follows the same data collection protocols as intervention practices. Second, they should include a sufficient number of preintervention data points to reliably attribute outcome change to interventions. Finally, they should consider using experimental approaches in allocating practices to interventions to minimize self-selection bias (eg, cluster-randomized trial or randomized stepped-wedge trials).
Conclusion
Our findings suggest that providing external support interventions in implementing quality improvement strategies may improve ABCS quality metrics among smaller primary care practices. While estimated improvements were small, they reflected average changes across a large and diverse sample of practices. Future research examining number of cardiovascular events prevented as a result of these complex, multi-component interventions is needed to justify investments in such large-scale primary care quality improvement programs.
Acknowledgments
Authors would like to thank Leif Solberg, MD, William Miller, MD, MS, and Benjamin Crabtree, PhD for their insightful comments.
Notes
This article was externally peer reviewed.
Funding: This research was supported by a grant from Agency for Healthcare Research and Quality 1R01HS023940-01 (PI: Cohen).
Conflicts of interest: None.
To see this article online, please go to: http://jabfm.org/content/35/5/968.full.
- Received for publication February 2, 2022.
- Revision received April 3, 2022.
- Accepted for publication April 8, 2022.
References
In this issue
Jump to section
Related Articles
Cited By...
- No citing articles found.