Abstract
Introduction: To examine the association of prior investment on the effectiveness of organizations delivering large-scale external support to improve primary care.
Methods: Mixed-methods study of 7 EvidenceNOW grantees (henceforth, Cooperatives) and their recruited practices (n = 1720). Independent Variable: Cooperatives’s experience level prior to EvidenceNOW, defined as a sustained track record in delivering large-scale quality improvement (QI) to primary care practices (high, medium, or low). Dependent Variables: Implementation of external support, measured as facilitation dose; effectiveness at improving (1) clinical quality, measured as practices’ performance on Aspirin, Blood Pressure, Cholesterol, and Smoking (ABCS); and (2) practice capacity, measured using the Adaptive Reserve (AR) score and Change Process Capacity Questionnaire (CPCQ). Data were analyzed using multivariable linear regressions and a qualitative inductive approach.
Results: Cooperatives with High (vs low) levels of prior experience with and investment in large-scale QI before EvidenceNOW recruited more geographically dispersed and diverse practices, with lower baseline ABCS performance (differences ranging from 2.8% for blood pressure to 41.5% for smoking), delivered more facilitation (mean=+20.3 hours, P = .04), and made greater improvements in practices’ QI capacity (CPCQ: +2.04, P < .001) and smoking performance (+6.43%, P = .003). These Cooperatives had established networks of facilitators at the start of EvidenceNOW and leadership experienced in supporting this workforce, which explained their better recruitment, delivery of facilitation, and improvement in outcomes.
Discussion: Long-term investment that establishes regionwide organizations with infrastructure and experience to support primary care practices in QI is associated with more consistent delivery of facilitation support, and greater improvement in practice capacity and some clinical outcomes.
- Cardiovascular Diseases
- Leadership
- Linear Models
- Population Health Management
- Primary Health Care
- Quality Improvement
- Surveys and Questionnaires
- Workforce
Introduction
Over the past 15 years, multiple initiatives have aimed to help primary care practices keep up with rapid changes in technology, knowledge, clinical evidence, and policy. These efforts have included local and federal initiatives to support the implementation and use of electronic health records (EHRs)1 and improve clinical quality measures (eg, Comprehensive Primary Care Initiative).2 There have also been initiatives to assist practices in becoming recognized Patient-Centered Medical Homes.3 Organizations, such as Regional Extension Centers (RECs) and Quality Improvement Organizations (QIOs), have emerged to provide external support to help practices make these changes.4,5 The Agency for Health care Research and Quality (AHRQ) has contributed to these efforts in various ways, including with its EvidenceNOW initiatives. These initiatives have funded grantees to develop regional infrastructure to provide external support to practices aimed at improving practice capacity and clinical outcomes, such as cardiovascular preventive care delivery, guidance to identify and address alcohol misuse,6 and female urinary incontinence.7
External support can span a range of activities: assisting in tailoring new evidence to practice; providing technical expertise to access EHR data to inform quality improvement; and coaching to implement operational changes.8,9 External support may benefit practices that lack internal resources for quality improvement, such as smaller clinician-owned practices and some health-system- and hospital-owned practices located in rural communities and at great distances from central office support.10 External support can ameliorate resource and capacity constraints that make staying current and engaging in ongoing quality improvement challenging.11,12 Although there is evidence that elements of external support, such as working with a coach or facilitator, can be beneficial for practices,13 the assistance provided by external organizations is often not well defined and measured. Studies of the delivery of large-scale external support are limited.
EvidenceNOW aimed to improve the ABCS of heart health (Aspirin use in high-risk individuals, Blood pressure control, Cholesterol management, and Smoking cessation support) and primary care practice capacity (health information technology, quality improvement, ability to adapt to change). We conducted the national evaluation of EvidenceNOW – called Evaluating System Change to Advance Learning and Take Evidence to Scale (ESCALATES; Cohen, PI). The EvidenceNOW grantees (henceforth, called Cooperatives) have published the outcomes of their individual grants.14⇓⇓⇓⇓⇓–20 This article complements that work by reporting the results of a comparative analysis that examined differences among Cooperatives.
Cooperatives are partnerships. Cooperative leads were Principal Investigators affiliated with academic health centers who partnered with regional experts and organizations with the ability to provide health information technology support and expertise (eg, RECs), a practice facilitator workforce (eg, QIOs, Area Health Education Centers), and educational materials and resources.21 Our prior work showed that Cooperatives started EvidenceNOW at different levels of preparedness, with some Cooperatives having partnerships that leveraged a stable facilitator workforce, data infrastructure and a network of relationships with practices, and other Cooperatives that were quite new to this work, and used EvidenceNOW to further develop this experience and infrastructure.22,23 In this article, we examine the association between Cooperatives’ experience level and delivery of facilitation and clinical outcome change. We tested the following a priori hypothesis:
Cooperatives with high levels of experience (pre-EvidenceNOW) will deliver a higher and more consistent amount of facilitation to their practices, and their practices will make larger improvements in practice capacity and in ABCS outcomes as compared to Cooperatives with low experience levels.
Methods
Setting
AHRQ funded 7 EvidenceNOW Cooperatives that operated in 12 states (see map in Online Appendix), defining their regions as either single state (most common), multi-state, or subregions of a state. Cooperatives developed or leveraged existing infrastructure and relationships to recruit practices24 and then delivered external support within a 3-year time frame.23 The initiative’s intended focus was on supporting smaller primary care practices (≤10 clinicians) with limited internal resources for quality improvement.25
Study Design and Conceptual Framework
We evaluated EvidenceNOW through the lens of a natural experiment using an iterative mixed methods design and informed by the Practice Change Model (PCM)26 and the Consolidated Framework for Implementation Research (CFIR).27 Details of the study design have been described elsewhere.21 This study was approved by the Oregon Health & Science University (OHSU) Institutional Review Board and was registered as an observational study at clinicaltrials.gov (NCT02560428).
Study Measures
Independent Variable
The main independent variable was each Cooperative’s experience level before EvidenceNOW, which we defined as a sustained track record in delivering large-scale quality improvement to primary care practices. This measure was derived through qualitative analysis, as described below. Cooperatives with high levels of experience had extensive knowledge of conducting quality improvement at scale in primary care practices, had more than a decade of doing this work in their region, and had regions with centralized infrastructure to support large-scale quality improvement. Cooperatives with medium levels of experience had some knowledge in conducting large-scale quality improvement, had 5 or fewer years of doing this work in their region, and had limited centralized infrastructure to support large-scale quality improvement before EvidenceNOW. Cooperatives with low levels of experience started EvidenceNOW with little knowledge of conducting large-scale quality improvement, little experience doing this work in their region, and no regional infrastructure to support large-scale quality improvement. Table 1 shows how we defined and operationalized this variable and includes supporting qualitative data.
Dependent Variables
There were 3 dependent variables for this study; (1) ability to deliver facilitation, (2) practice capacity, and (3) ABCS quality metrics. Ability to deliver facilitation was assessed by measuring facilitation amount, which was the total number of hours and months of facilitation a Cooperative delivered to each practice. Practice capacity included 2 measures; practices’ ability to adapt to change, measured using the Adaptive Reserve (AR) composite measure,28 and quality improvement capacity measured using part of the Change Process Capacity Questionnaire (CPCQ).30 The ABCS quality metrics focused on clinical improvement in cardiovascular preventive and disease care outcomes. Table 2 details these measures, and the online Appendix includes the AR and CPCQ survey items.
Covariates
Practice characteristics (for details, see online Appendix) included the following: practice location; practice ownership; practice size; and practice patient characteristics, including age and insurance status.
Data Collection
Qualitative data were collected by our team, as described in Table 3. Cooperatives abstracted practice-level ABCS performance metrics from EHRs. Measures of practice capacity and practice demographic data were collected by Cooperatives via a survey pre- and post-intervention. EHR and survey data were shared with our team. Facilitators tracked the frequency, duration, and mode of their practice visits (phone, virtual, in-person). We worked with Cooperatives to harmonize all measures, but the approaches Cooperatives used to collect data varied. More details about data collection, including methods to collect survey and ABCS data, are described elsewhere.21
Analysis
We analyzed qualitative data prospectively using a group analysis process and an inductive approach.21,24,29⇓–31 We immersed ourselves in data (interviews, fieldnotes, and artifacts (eg, toolkits and materials)). We discussed these data to identify insights and summarized emerging observations using written case summaries and matrices. One emerging finding was Cooperative experience– before EvidenceNOW– with large-scale quality improvement. In a second analytic cycle, we reviewed raw and summarized data relevant to this issue. Through consensus, we formulated a definition of Cooperative experience level and, based on data describing each Cooperative’s prior resources, relationships, and infrastructure, rated Cooperative experience level as high, medium, or low.
For quantitative analyses, we calculated descriptive statistics of practice characteristics overall and by each Cooperative’s level of experience, examining baseline and post-intervention outcome performance as well as changes in outcomes between baseline and follow-up. We defined baseline as the first round of practice surveys (for AR; CPCQ) or last quarter before intervention began in each practice (for ABCS). We measured changes in AR and CPCQ as the difference between the first (baseline) and second round (immediately after the intervention) survey responses. For the ABCS, we measured change as the difference between baseline and post-intervention, where post-intervention was the quarter immediately after the end of intervention. For facilitation, we reported total hours as well as number of months with some facilitation provided to practices during interventions.
We compared the influence of Cooperatives’ level of experience on study outcomes using 3 ordinary least squares (OLS) linear regression models. First, we measured unadjusted mean changes in each outcome by each Cooperative’s level of experience (low experience serving as the reference group). Second, we adjusted for practice characteristics in all models. For models where ABCS change was the outcome, we also adjusted for baseline AR because we hypothesized that initial levels of AR might be associated with changes in clinical performance. Third, we reran covariate-adjusted regression models stratified by outcome performance at baseline (below and above the median). This set of analyses assessed if Cooperatives were effective at improving clinical and capacity measures among clinics with low performance (below the median) at baseline. We clustered standard errors at the Cooperative level using bootstrapping with 1000 repetitions to account for correlated response at the Cooperative level.32 All quantitative analyses were performed using R version 3.6.0.
Results
Each EvidenceNOW Cooperative recruited between 209 and 315 practices in their region. Table 4 shows that Cooperatives tended to recruit practices with 10 or fewer clinicians, which was the initiative’s target. Low experience Cooperatives tended to recruit practices from urban settings, near where the academic health center or their partner organizations were located. High experience Cooperatives recruited a more regionally dispersed (see Appendix Figure 2 for visual examples) and demographically diverse group of practices than low experience Cooperatives.
Delivery of Facilitation
Overall, facilitators delivered an average of 18.1 hour of facilitation over an average of 7.2 months (see Table 5). Practices from Cooperatives with high levels of experience received significantly more hours (+20.3 hours, P = .040) and months (+4.3 months, P = .001) of facilitation than practices from Cooperatives with low levels of experience. After adjusting for practice characteristics, differences in facilitation amount by level of Cooperative experience persisted (Table 6). Qualitative data showed that facilitators of high experience Cooperatives were living across their regions and in closer proximity to practices. In addition, these Cooperatives had experience supporting a remote facilitator workforce.
I think the structured system that we have with the communication and the collaboration amongst the [facilitators] is very strong…having that structured system and having the library that we have to just go in and access…there's a whole list of resources on there that we can use. (Facilitator Interview, Cooperative 1)
The combination of organizational infrastructure (eg, toolkits, resource libraries, peer support) to inform and support facilitator work and less travel time translated into more frequent visits.
In contrast, Cooperatives with low levels of experience lacked regionwide facilitator infrastructure and learned – during the EvidenceNOW initiative – that they needed tools and processes to continuously support and monitor facilitators: “Training-wise, yeah, it was really experience in practice. It was not really anything formalized. [We were] thrown in the fire.” (Facilitator Interview, Cooperative 2). Low experience Cooperatives also set overly ambitious and unachievable timelines for practices and facilitators, and they engaged many practices from large health systems to meet recruitment targets. For example, we heard during a site visit with one low experience Cooperative that their recruitment leader asked a large health system to sign-up half of their 100 practices for EvidenceNOW. The other low experience Cooperative also reached targets by recruiting 36% of their practice cohort (n = 76) from two health systems (43 and 33 practices from each system). Health system practices tended to have in-house data analytics and quality improvement teams, and in some cases, leadership volunteered their practices for EvidenceNOW, which affected participation in facilitation.
Change in Practice Capacity and Clinical Quality
Overall, Cooperatives’ practices made small improvements in clinical quality measures during the intervention ranging from 1.9% improvement in the blood pressure metric to a 5.6% improvement in the smoking cessation metric. All capacity measures also improved, AR increased by 0.018 and CPCQ by 6.1 points. For details see Table 5.
Table 5 also shows that practices recruited by Cooperatives with high levels of experience were more likely to have lower average clinical quality metrics at baseline for aspirin (difference: -18.8%, P = .083), blood pressure (difference: -2.8%, P = .04), cholesterol (difference: -17.3%, P = .007) and smoking (difference: -41.5%, P = .005) than practices from Cooperatives with low levels of experience. The 2 high experience Cooperatives used EvidenceNOW funding to expand regional health information exchanges. They intentionally recruited practices that were untethered to health systems and, therefore, did not have “in-house” data teams working on abstracting and cleaning ABCS metrics. For example, their practices might lack the capacity to abstract a metric, which was the case with the cholesterol metric (eg, 37 practices produced this metric in Cooperative 7). This difference in focus and recruitment approach likely explains the low baseline ABCS among the practices in the 2 high-experience Cooperatives.
Improvements in clinical quality metrics by Cooperatives’ level of experience varied (Table 6). Although practices from Cooperatives with high levels of experience showed larger improvements in cholesterol (+5.99%, P = .0141), smoking (+6.43%, P = .0030), and aspirin (+2.89%, P > .05), change in blood pressure was similar between Cooperatives with different experience levels.
With respect to capacity measures, changes in quality improvement capacity were larger for practices from Cooperatives with high levels of experience compared with practices from Cooperatives with low levels (CPCQ difference: 2.04, P < .001), but this was not the case for capacity to adapt to change (AR difference: -0.02, P = .27).
Adjusting for practice characteristics did not substantially affect change estimates for clinical quality and practice capacity, but these changes were no longer statistically significant except for the smoking metric (Table 6). Further stratification by baseline ABCS performance resulted in muted effects for aspirin, cholesterol, and smoking with significant differences observed only for the smoking metric.
Discussion
Cooperatives that had a sustained track record with large-scale quality improvement before EvidenceNOW, which included leaders with the knowledge, regional partnerships, and infrastructure to support this work, recruited a more diverse group of practices to this initiative and delivered significantly more facilitation (hours and months) to their practices than Cooperatives with low experience levels. In addition, the level of improvement observed among the high experience Cooperatives’ practices was greater for all clinical outcomes (except for blood pressure), as well as for quality improvement capacity. These results remained significant for facilitation, quality improvement capacity, and smoking cessation improvement after adjusting for differences in practice characteristics and stratifying for baseline capacity and ABCS levels. What did not change was practices’ adaptive capacity, and this is likely because Cooperatives did not focus on improving the organizational aspects of AR (eg, communication, teamwork, mindfulness) with their practices.
Cooperatives with a sustained track record and investment in large-scale quality improvement support, before EvidenceNOW, had a number of important resources that low-experience Cooperatives lacked. This included robust regionwide facilitator infrastructure (ie, having facilitators located geographically throughout the region) and the knowledge of how to support and deploy a facilitator workforce effectively. Studies show that the ability to support a facilitator workforce is connected to facilitator effectiveness,33,34 and when combined with network infrastructure this likely explains the association we see between Cooperative experience level and facilitation amount; facilitators visited practices more when they lived in closer proximity to their practices, and when they had the tools to be helpful. Practices likely wanted more facilitator visits from effective facilitators.
Cooperatives with high experience levels also had leaders with the confidence and ability to reach recruitment targets without recruiting large numbers of practices from single health systems,23 and this explains why they were comparatively more successful at recruiting a more diverse and dispersed group of smaller, clinician-owned practices that were in need of quality improvement capacity-building and assistance in improving clinical quality metrics, as baseline metrics indicate. Cooperatives with high experience levels recruited precisely the types of practices AHRQ wished to target. In contrast, Cooperatives with low experience levels reached their targets by recruiting large numbers of practices from single hospitals and health systems. This was an important departure from the aim of EvidenceNOW. System-owned practices had data and quality improvement resources in-house. Their clinical teams also had less autonomy and more burnout than clinician-owned practices.35,36 Ownership changed how EvidenceNOW Cooperatives and their facilitators worked with practices,37 and the amount of facilitation and the types of changes practices needed to make to improve clinical outcomes.38 This combination of factors contributes to explaining the association between facilitation amount and Cooperative experience level, and, although not easily mutable, practice ownership must be considered when implementing and evaluating the value of large-scale quality improvement initiatives that involve external support. Our findings suggest that efforts to reach a broad range of practices that may benefit from external support require developing the experience and resources of Cooperative or Cooperative-like organizations.
For clinicians and clinical teams, particularly clinician-owned practices that see a need to improve practice capacity and clinical quality, working with a facilitator who is supported by an experienced Cooperative can help the practice work through an improvement process.34 The benefit of working with an external facilitator is less clear for hospital- and health-system-owned practices or those practices that have internal quality improvement support. This is a novel finding worthy of further investigation.
For researchers, it is important to note that the characteristics for which we adjusted in our statistical models are the same characteristics that our qualitative data suggest made the high experience Cooperatives more successful. Typically, unadjusted findings are not very informative to final results. But in the context of understanding Cooperative experience-level differences with practices nested within Cooperatives, comparing unadjusted and adjusted findings allows us to observe and identify the complexity of these relationships and the influence of Cooperative experience separately from variations by practice types. Thus, we report unadjusted and adjusted results, but more consideration of the best ways to blend and report mixed methods data from natural experiments is needed.39 For researchers in the field of dissemination and implementation science, high-experience Cooperatives were more effective in some aspects of their work because they started this initiative with a developed network of relationships with regional practices and because they worked collaboratively with the people in these practices—on the ground—to foster change. This effort was observable on a large scale and should be considered in future dissemination and implementation study designs and analyses.
This study’s findings must be interpreted in light of some key limitations. First, changes in clinical outcomes are evaluated as prepost changes and do not control for concurrent changes in clinical outcomes in a region. For more on how EvidenceNOW practices changed in relation to an external comparison group see Balasubramanian et al.40 This work shows that EvidenceNOW practices, on average, made small improvements in the ABCS among a large, diverse sample of practices with potential for population-level impact on cardiovascular events avoided. Future researchers could use simulation modeling techniques to estimate the population impact of this level of change on cardiovascular events avoided. Second, none of the EvidenceNOW Cooperatives or their partners could produce a list of all the primary care practices in their region. Although not having these data readily available is an important deficiency for our nation, it meant that we could not assess the regional representativeness of primary care practices that participated in each EvidenceNOW Cooperative. Third, although Cooperatives implemented a range of different external support strategies (eg, audit and feedback, performance benchmarking), facilitation amount was the only strategy that we could harmonize and measure across Cooperatives.41,42 Thus, we could not assess the full impact of Cooperatives’ external support on practices. In addition, facilitation amount was collected by Cooperatives and shared with our team; thus, some of the differences we observe in facilitation amounts may reflect different recording practices. Fourth, our study design does not fully align with the standards of randomized comparative effectiveness trial designs. However, a randomized trial is not the standard for studying broad-based real-life change. Looking at EvidenceNOW through the lens of a natural experiment allowed us to observe some important naturally occurring patterns and relationships that might be obscured using more traditional trial methods, and we have taken care to not overstate this study’s findings. Fifth, low baseline levels43 and changes might partially reflect initial data quality issues that were subsequently addressed by Cooperatives and practices as part of quality improvement.22 We cannot rule out this possibility. However, we eliminated outliers from our analysis of ABCS changes to account for possible data quality concerns.
Conclusion
The recent National Academies of Sciences, Engineering, and Medicine report, Implementing High-Quality Primary Care: Rebuilding the Foundation of Health Care,44 sets forth a plan for rebuilding the US health care system, with a focus on primary care stakeholders. One of the key objectives is to ensure that high-quality comprehensive primary care is implemented and available to every person in the US. Findings from EvidenceNOW suggest that 1 step toward accomplishing this objective45⇓–47 could be funding regional infrastructure to develop organizations with the experience, resources, and relationships to deliver external support to primary care practices at a large scale, particularly if the efforts are directed to enhancing quality improvement more generally, rather than at a narrow disease-oriented target. This will take a level of sustained investment that cannot be achieved solely through grant funding but must be built into state and federal budgets.
Acknowledgments
We are grateful to all the EvidenceNOW Cooperatives and their practices who made this work possible. In addition, the entire ESCALATES study team willingly contributed their thoughtful feedback, particularly Kurt C. Stange, MD, PhD. Andrew Bazemore, MD, MPH, Robert L. Phillips Jr. MD, MSPH, and Larry A. Green, MD, three of our study advisors, generously shared creative thoughts and feedback that informed and improved this work. Thank you to Carrie Tillotson, MPH, with Data Drawn Consulting, LLC, who assisted with data visualization.
Appendix
Appendix.
Notes
This article was externally peer reviewed.
Funding: This research was supported by a grant from Agency for Healthcare Research and Quality 1R01HS023940-01 (PI: Cohen).
Conflict of interest: The authors of this manuscript have no conflicts of interest to report.
To see this article online, please go to: http://jabfm.org/content/35/6/1115.full.
- Received for publication March 2, 2022.
- Revision received June 21, 2022.
- Accepted for publication June 23, 2022.