Abstract
Background: Contextual factors relevant to translating healthcare improvement interventions to different settings are rarely collected systematically. This study articulates a prospective method for assessing and describing contextual factors related to implementation and patient reach of a pragmatic trial in primary care.
Methods: In a qualitative case-series, contextual factors were assessed from the My Own Health Report (MOHR) study, focused on systematic health risk assessments and goal setting for unhealthy behaviors and behavioral health in nine primary care practices. Practice staff interviews and observations, guided by a context template were conducted prospectively at three time points. Patient reach was calculated as percentage of patients completing MOHR of those who were offered MOHR and themes describing contextual factors were summarized through an iterative, data immersion process.
These included practice members' motivations towards MOHR, practice staff capacity for implementation, practice information system capacity, external resources to support quality improvement, community linkages, and implementation strategy fit with patient populations.
Conclusions: Systematically assessing contextual factors prospectively throughout implementation of quality improvement initiatives helps translation to other health care settings. Knowledge of contextual factors is essential for scaling up of effective interventions.
Understanding contextual factors relevant to primary care practice settings is critical for translating findings from health care improvement interventions into practice change.1,2 Researchers are increasingly interested in examining and systematically documenting the specific contexts in which implementation occurs to better explain the mechanisms by which interventions improve outcomes in practice.2⇓⇓⇓⇓–7 Most clinical trials focus exclusively on internal validity8,9 over external validity, thus excluding the variability of contexts in which interventions are conducted.10 Knowledge of contextual factors is necessary, however, to understand both how and why findings fit into a particular setting and to generate the information needed to knowledgeably translate interventions to other settings and situations. Paying attention to contextual factors is especially important for pragmatic implementation trials that are intentionally designed for real-world health care settings.11⇓⇓⇓–15
Several models from the field of implementation science have posited the important influence of context on the successful translation of research and quality improvement findings into practice.16⇓⇓⇓–20 In particular, Stange and Glasgow2 identified domains of contextual factors representing multiple, diverse stakeholder perspectives by synthesizing information from 12 existing frameworks and described a method for collecting data on context. Fourteen research teams successfully applied this method retrospectively to diverse practice improvement projects.15 However, retrospective assessment of contextual findings can suffer from recall bias, especially for interventions implemented in busy, fast-paced clinical practices.
As a part of the protocol for the My Own Health Report (MOHR) study, a pragmatic trial focused on systematic implementation of a health behavior and mental health assessment tool and feedback system in 9 primary care practices, we adapted the method described by Stange and Glasgow2 in order to prospectively assess contextual factors influencing intervention implementation and patient reach, calculated by dividing the number of patients who completed the MOHR assessment by the number of patients offered the MOHR. The aim of this study was to show how contextual factors can be assessed prospectively during a pragmatic trial and to delineate the contextual factors influencing the implementation and patient reach of this intervention.
Methods
The MOHR Pragmatic Trial
The MOHR study was a cluster-randomized pragmatic trial of an evidence-based, patient-centered health behavior and mental health assessment tool paired with a feedback system to promote patient counseling and goal-setting. Details about the intervention, the mixed methods evaluation, and main findings from this study are reported in detail elsewhere.21⇓⇓⇓–25 Briefly, practices were provided with a web-based or Article health risk assessment form, the MOHR, which assessed patients' diet, exercise, smoking, alcohol, drug use, stress, depression and anxiety, and sleep.26 Practices were asked to implement MOHR in a way that was pragmatic and feasible for them. They chose whether they administered MOHR through Article or electronically (in person or online), and created their own workflow to share MOHR reports with clinicians and patients and to facilitate goal-setting discussions.
Practice Sample
Nine primary care practices from 6 states implemented the MOHR intervention. Practices were purposefully selected to enhance generalizability and represented the diversity of primary care practices in terms of type, ownership, location, electronic health record infrastructure, and patient panel demographics. Eight research teams that manage practice-based research networks or participated in the Cancer Prevention and Control Research Network used a convenience sampling approach to recruit these practices. Researchers from both networks used their extensive experience partnering with practices to identify suitable practices for this study. After recruitment, 1 practice withdrew early from the study and was replaced by the research team.
Data Collection
Data on contextual factors were collected using a step-wise approach recommended by Stange and Glasgow.2 This included (1) identifying contextual factors using a “Context Matters” template2,15 (see the Appendix); (2) assessing context at the beginning, middle, and end of the study; and (3) evaluating how contextual factors affected key processes and outcomes through an immersion/crystallization analytic approach.27
The Context Matters template is a tool developed by Stange and Glasgow2 to systematically collect and report data on contextual factors relevant to change interventions. This template includes specific domains informed by an extensive review of theoretical models and frameworks,2 and informs interview questions and clinic observations about topics such as payment systems, health information technology support, practice culture, and staffing (see the Appendix). Data collectors were experienced in qualitative data collection and were those who served as research team liaisons with practice staff and leaders. Data collectors and research team members were trained in the use of the context template for data collection and reporting before baseline data collection.
Contextual data were prospectively collected between March and December 2013. Data collectors used the context template as a guide to conduct brief interviews with multiple stakeholders at each practice site, including clinic leaders, clinicians, and staff. They also conducted observations of clinic activities such as patient flow, practice workflow, and interactions among staff to supplement interview data. Field notes from interviews and observations were recorded by practice number on the context template. Data collectors were encouraged to collect and record direct quotes. They then forwarded the completed templates to members of the MOHR context workgroup22 for further data summary and subsequent thematic analysis. The MOHR context workgroup was multidisciplinary, with members representing primary care medicine, epidemiology, anthropology, and health behavior sciences.
For quality control, conference calls were held with research teams before and halfway through implementation to discuss each practice's approach to collecting qualitative data and to problem-solve challenges to completing the context template. Some variations in data collection methods were identified; for example, some sites completed interviews in person, whereas others completed them by phone. Two sites collected data for only 2 of the 3 time points, leading to some missing data at the midpoint and at the end of the implementation period.
Research teams also collected quantitative data on patient reach, defined as the number, proportion, and representativeness of eligible patients offered and completing the MOHR assessment.28 Patient reach was calculated by dividing the number of patients who completed the MOHR assessment by the number of patients offered the MOHR.15
Data Analysis
The context template served as raw data that were uploaded into Atlas.ti (version 7.0; Scientific Software Development GmbH, Berlin, Germany) for coding and analysis. The coding scheme was chosen a priori based on the model developed by Stange and Glasgow.2 At least 2 workgroup members independently coded each practice's context template. Any coding discrepancies were resolved through discussion among the multidisciplinary team. Data were compared across geographic locations (urban, semiurban, rural), networks (practice-based research network vs federally qualified health center, and MOHR administration types (Article-based, online and faxed to office, online and printed at office). Passages of coded text were coalesced to form higher-level themes through a multistage, iterative data immersion process. Excerpts of field notes provided in the Results section of this article were selected to represent these higher-level themes. The study was approved by institutional review boards at the Virginia Commonwealth University (no. HM12746), University of California, Los Angeles (no. 12-0017900), and 5 other participating institutions.
Results
MOHR practices varied with respect to size, ownership, health system affiliation, geographic location, and patient sociodemographics (Table 1). This variation was intentional to enhance generalizability of the findings.
Descriptive Characteristics of the My Own Health Report Primary Care Practices (March-December 2013)
Contextual factors influencing MOHR implementation and patient reach included factors both internal and external to the practice. Below, we describe in more detail how these factors influenced MOHR implementation and patient reach, and Table 2 provides representative quotations and field notes exemplifying the identified factors.
Contextual Factors Related to Implementation and Patient Reach* of the My Own Health Report Health Risk Assessment
Factors Internal to the Practice
Internal factors included practice staff members' motivation to use MOHR, practice staff's capacity to take on additional responsibilities to facilitate MOHR administration, and practices' information system capacity.
Practice Members' Motivations
Practice leaders and staff members were motivated to adopt the MOHR tool because it would enable them to systematically identify patients with unhealthy behaviors and mental health concerns. In particular, patient and provider reports generated by the MOHR tool helped streamline the goal-setting process by easily identifying patients' risk factors and highlighting the behaviors patients were ready to work on. In addition to facilitating goal setting, some practice leaders perceived that implementing MOHR could be helpful in meeting reporting requirements to external agencies, patient-centered medical home certification, or criteria for meaningful use of the electronic medical record (EMR). Such motivations for adopting MOHR greatly facilitated startup of the intervention in practices.
However, enthusiasm waned over time, even among practices that were initially motivated and that perceived MOHR to be useful to clinicians and patients. This is because MOHR added a significant time burden to visits, which resulted in implementation challenges. In addition, in 2 practices, some of the health assessment questions were already part of existing patient intake forms, and practice members were concerned from the outset that implementing MOHR created an element of redundancy and would add more time to patient visits that were already running over schedule.
Practice Staff Capacity
We observed that existing staff modified their roles or took on additional duties to implement MOHR. For instance, among practices that administered MOHR in the office, the medical assistants took on additional responsibilities in assisting patients who needed help completing it. In practices where patients completed the MOHR online at home, practice staff adapted their roles to locate the completed physician reports and include them in the goal-setting discussion. When practices members did not have internal capacity to take on additional responsibilities to implement MOHR, research teams assisted with implementation. This was often the case among federally qualified health center practices that had large patient volumes and a higher proportion of underserved patients.
Practice Information System Capacity
Practice information systems capacity was related to MOHR implementation. Delays in printing or receiving faxes of the MOHR reports disrupted clinical workflow. This resulted in fewer visits that included goal setting because patient and/or provider reports were not available at the time of the visit. In addition, several practices experienced challenges with their existing technology infrastructure, such as changing EMR systems and low use of patient portals, which hindered MOHR implementation.
External Factors Influencing MOHR Implementation
Availability of external resources to support quality improvement emerged as an important contextual factor relevant to MOHR implementation, in particular, support from larger health care systems, practice linkages with community resources, and fit of the implementation strategy with patient populations.
Linkages with a Larger Health Care System
Practices organized within larger health care systems had additional support for implementation that they could leverage and use. This included information technology support as well as help from research or health system staff members to implement quality improvement initiatives. Significant support from 1 practice's affiliated health care system's nurse operators helped increase patient reach of the MOHR intervention. Practices that had such support often were able to leverage internal and external resources to facilitate implementation and even make midcourse changes to implementation when necessary, as described here. At baseline, 1 practice decided to test a comprehensive implementation approach that included (1) mailing MOHR invitations to patients' homes, (2) inviting patients to complete MOHR before their appointment, or (3) inviting patients to complete MOHR over the phone. In the first few weeks, however, they learned that these approaches were unable to reach a large proportion of their target population. Therefore, they modified their approach midstream by seeking help from their health care system to have additional staff administer the MOHR. This resulted in a substantial increase in the proportion of patients reached over subsequent weeks. On the other hand, when a practice belonged to a larger network of clinics, they could compete for staff time and clinic resources, thereby hindering implementation.
Linkages with Community Resources
Another contextual factor that posed a barrier to goal setting was practices' lack of established linkages with community resources to refer patients who needed additional counseling for unhealthy behaviors or mental health issues. Further, the lack of a systematic referral process for more intensive counseling also contributed to inconsistent use of the MOHR tool.
Fit of Practice Implementation Strategy with Patients
Practices' patient panel characteristics (age, predominant language spoken, and health literacy) were an important contextual factor that influenced practices' decisions on how MOHR would be administered to patients. Practices that served large panels of low-income and non-English-speaking patients administered the MOHR in the office, rather than at home via the web, because they perceived that most of their patients would not have consistent Internet access or access to computers at home. Even when administered in the office, patients of these practices needed significant help from practice members or research staff to complete the MOHR on laptop computers or tablets. Anther concern that affected the mode of administration was the ability of older patients to navigate a web-based tool.
Interaction Between Contextual Factors
The contextual factors described above often interacted, exacerbating the challenges to successful implementation of the MOHR. To illustrate, practice information capacity limitations, along with sustained lack of support for the new quality improvement initiative and demotivated staff, specifically presented challenges to MOHR implementation for practice 6:
The MOHR report often does not get to them [physicians] in time for their visit with the patient so the information is not utilized; when it is scanned into the EMR, it is not easily located and they do not have [the] motivation to search for it at the next visit (they also do not know who has filled it out and who hasn't). (site 6, time point 2)
The “physician champion” did not really turn out to be a champion of the project because he said the reports did not get into their medical record in a timely manner and there was no time within the patient visits to address additional issues. As with the other providers, the lead physician and nursing director were not very impressed with the MOHR and did not feel that it added value to their patient care. (site 6, time point 3)
This example highlights how interaction of several contextual factors resulted in breakdowns in the process of MOHR reports reaching clinicians and patients during the visit and a lack of integration of MOHR into existing clinic workflow for this practice.
Discussion
Prospectively assessing contextual factors in a pragmatic trial conducted in primary care revealed factors both within and external to the practice environment as influencing implementation and patient reach. Quality improvement intervention studies rarely collect systematic data on contextual factors. Even rarer are intervention studies that collect contextual data at multiple points over time. This is especially problematic because numerous rapid quality improvement cycles are needed to implement an improvement initiative, and contextual factors facilitating or hindering these improvement cycles are often lost if not assessed in real time.29 This prospective approach in the MOHR study identified key contextual factors, including practice members' motivations toward using the intervention, practice capacity, quality improvement support available to practices, linkages to community resources, and patient panel characteristics.
The MOHR study was designed and implemented as a pragmatic trial,22 such that practices could and did tailor MOHR implementation to suit their local setting. Despite this flexibility, additional contextual factors hindered implementation, including practices' capacity to take on a new quality improvement initiative, practice members' motivation to change, and resources available to the practice in order to support change. Other primary care change initiatives also identified these factors as particularly salient because they require significant changes to practice workflow and are potentially disruptive to practice functioning, suggesting that these may be important to consider for most practice change initiatives.15,16,30,31
In addition to these general contextual factors, our prospective method helped identify variations across practices in goal setting for unhealthy behaviors, even though the study's main findings demonstrated an increase in goal setting among intervention compared with control practices.25 Technology challenges in accessing patient and clinician MOHR reports at the point of care, coupled with the additional time needed to complete MOHR, hindered goal setting. Practices' limited external linkages with community resources for behavioral health counseling (for example smoking cessation and physical activity counseling) challenged implementation, as clinicians perceived no benefit in setting goals with patients if there was no place to send them for additional counseling. Paying attention to context throughout the study helped explain observed variations in implementation and, more importantly, helped identify conditions under which goal setting was more likely to be successfully implemented. Our findings could enable other practices seeking to implement an electronic health risk assessment tool to identify, in advance, “real-world” trade-offs to integrating it in their workflow. And, as our findings suggest, these trade-offs may change over time as implementation proceeds within the practice.
These findings should be interpreted in light of the study's strengths and limitations. While participating practices were very diverse, they were volunteer practices from research networks and thus are unlikely to be representative of all primary care practices. None of the participating practices sustained MOHR after the study's completion,23 so our observations of implementation were restricted in time. Nevertheless, our study shed light on reasons for the lack of sustainability. For instance, the significant additional time and staff resources needed to administer MOHR made it impossible for practices with large volumes of underinsured and uninsured patients to integrate it into their daily workflow. Thus, MOHR was discontinued once the study ended. Even in the practice-based research network practices, additional, unreimbursed time for goal setting made it difficult to sustain MOHR after the end of the study.
The study's methods, involving case studies, may have further limited generalizability. Nonetheless, randomized controlled trials offer only average measures of effectiveness and are context-specific, whereas a series of case studies in different contexts could provide valuable information about how an intervention operates, as with the findings from this study.15 Another, more subtle limitation of the study concerned the context of the study itself. The MOHR study was developed, funded, and led by a national study team based at the National Institutes of Health, with an academic coordinating center that changed during the course of the project. Other than the impact of national reimbursement and reporting requirements, however, respondents did not report any effects of the study aegis, leadership, or even the research staff itself (except as staff “extenders” to administer the MOHR) on implementation.
Notwithstanding these limitations, the study has some significant strengths, most notably the prospective collection of data over multiple assessment times and the diversity of clinics. Context is increasingly recognized as important, but few primary care implementation studies explicitly collect and report on contextual factors, and fewer still do so prospectively throughout implementation.2,15 Paying attention to contextual factors throughout the course of this study helped identify key factors resulting in implementation challenges that would not have otherwise been recognized.
The methodology used in this study can be helpful to both researchers and practicing clinicians. We recommend the use of the Context Matters template to systematically and prospectively capture data on contextual factors at multiple levels (practice, community, and state). This method will enable researchers to identify factors that may influence implementation differentially by practice context. Reporting on contextual factors using this method may also help practices assess whether identified factors are relevant to them when implementing MOHR or a similar health risk assessment (HRA). Our study findings suggest that enhanced capacity to make quality improvement changes in a practice where practice members are motivated and that has resources to make and sustain changes is critically important for the successful implementation of most quality improvement interventions. Specifically, for practices wishing to implement MOHR, we recommend that they pay attention to their information technology capacities to effectively administer MOHR via patient portals or using web-enabled tablets in the office, and to modify their workflows to account for the additional visit time needed for the important task of goal setting.
Conclusion
Understanding practice contexts can be used to successfully implement HRAs as a part of the Medicare annual wellness visit and as part of routine care. Involvement of diverse stakeholders in gathering and interpreting data on relevant contextual factors over time can advance the understanding of what happened with a particular intervention and why, and can allow others to make reasonable judgments about how an intervention or its implementation might need to be modified in order to be effectively executed in different settings and circumstances.
Appendix
MOHR Context Matters Template
Notes
This article was externally peer reviewed.
Funding: none.
Conflict of interest: none declared.
To see this article online, please go to: http://jabfm.org/content/30/3/337.full.
- Received for publication May 9, 2016.
- Revision received October 17, 2016.
- Accepted for publication January 23, 2017.
