Factors Related to Implementation and Reach of a Pragmatic Multisite Trial: The My Own Health Report (MOHR) Study
=================================================================================================================

* Bijal A. Balasubramanian
* Suzanne Heurtin-Roberts
* Sarah Krasny
* Catherine L. Rohweder
* Kayla Fair
* Tanya T. Olmos-Ochoa
* Kurt C. Stange
* Sherri Sheinfeld Gorin

## Abstract

*Background:* Contextual factors relevant to translating healthcare improvement interventions to different settings are rarely collected systematically. This study articulates a prospective method for assessing and describing contextual factors related to implementation and patient reach of a pragmatic trial in primary care.

*Methods:* In a qualitative case-series, contextual factors were assessed from the My Own Health Report (MOHR) study, focused on systematic health risk assessments and goal setting for unhealthy behaviors and behavioral health in nine primary care practices. Practice staff interviews and observations, guided by a context template were conducted prospectively at three time points. Patient reach was calculated as percentage of patients completing MOHR of those who were offered MOHR and themes describing contextual factors were summarized through an iterative, data immersion process.

These included practice members' motivations towards MOHR, practice staff capacity for implementation, practice information system capacity, external resources to support quality improvement, community linkages, and implementation strategy fit with patient populations.

*Conclusions:* Systematically assessing contextual factors prospectively throughout implementation of quality improvement initiatives helps translation to other health care settings. Knowledge of contextual factors is essential for scaling up of effective interventions.

*   Delivery of Health Care
*   Health Resources
*   Patient Care Team
*   Primary Health Care
*   Quality Improvement

Understanding contextual factors relevant to primary care practice settings is critical for translating findings from health care improvement interventions into practice change.1,2 Researchers are increasingly interested in examining and systematically documenting the specific contexts in which implementation occurs to better explain the mechanisms by which interventions improve outcomes in practice.2⇓⇓⇓⇓–7 Most clinical trials focus exclusively on internal validity8,9 over external validity, thus excluding the variability of contexts in which interventions are conducted.10 Knowledge of contextual factors is necessary, however, to understand both how and why findings fit into a particular setting and to generate the information needed to knowledgeably translate interventions to other settings and situations. Paying attention to contextual factors is especially important for pragmatic implementation trials that are intentionally designed for real-world health care settings.11⇓⇓⇓–15

Several models from the field of implementation science have posited the important influence of context on the successful translation of research and quality improvement findings into practice.16⇓⇓⇓–20 In particular, Stange and Glasgow2 identified domains of contextual factors representing multiple, diverse stakeholder perspectives by synthesizing information from 12 existing frameworks and described a method for collecting data on context. Fourteen research teams successfully applied this method retrospectively to diverse practice improvement projects.15 However, retrospective assessment of contextual findings can suffer from recall bias, especially for interventions implemented in busy, fast-paced clinical practices.

As a part of the protocol for the My Own Health Report (MOHR) study, a pragmatic trial focused on systematic implementation of a health behavior and mental health assessment tool and feedback system in 9 primary care practices, we adapted the method described by Stange and Glasgow2 in order to prospectively assess contextual factors influencing intervention implementation and patient reach, calculated by dividing the number of patients who completed the MOHR assessment by the number of patients offered the MOHR. The aim of this study was to show how contextual factors can be assessed prospectively during a pragmatic trial and to delineate the contextual factors influencing the implementation and patient reach of this intervention.

## Methods

### The MOHR Pragmatic Trial

The MOHR study was a cluster-randomized pragmatic trial of an evidence-based, patient-centered health behavior and mental health assessment tool paired with a feedback system to promote patient counseling and goal-setting. Details about the intervention, the mixed methods evaluation, and main findings from this study are reported in detail elsewhere.21⇓⇓⇓–25 Briefly, practices were provided with a web-based or Article health risk assessment form, the MOHR, which assessed patients' diet, exercise, smoking, alcohol, drug use, stress, depression and anxiety, and sleep.26 Practices were asked to implement MOHR in a way that was pragmatic and feasible for them. They chose whether they administered MOHR through Article or electronically (in person or online), and created their own workflow to share MOHR reports with clinicians and patients and to facilitate goal-setting discussions.

### Practice Sample

Nine primary care practices from 6 states implemented the MOHR intervention. Practices were purposefully selected to enhance generalizability and represented the diversity of primary care practices in terms of type, ownership, location, electronic health record infrastructure, and patient panel demographics. Eight research teams that manage practice-based research networks or participated in the Cancer Prevention and Control Research Network used a convenience sampling approach to recruit these practices. Researchers from both networks used their extensive experience partnering with practices to identify suitable practices for this study. After recruitment, 1 practice withdrew early from the study and was replaced by the research team.

### Data Collection

Data on contextual factors were collected using a step-wise approach recommended by Stange and Glasgow.2 This included (1) identifying contextual factors using a “Context Matters” template2,15 (see the Appendix); (2) assessing context at the beginning, middle, and end of the study; and (3) evaluating how contextual factors affected key processes and outcomes through an immersion/crystallization analytic approach.27

The Context Matters template is a tool developed by Stange and Glasgow2 to systematically collect and report data on contextual factors relevant to change interventions. This template includes specific domains informed by an extensive review of theoretical models and frameworks,2 and informs interview questions and clinic observations about topics such as payment systems, health information technology support, practice culture, and staffing (see the Appendix). Data collectors were experienced in qualitative data collection and were those who served as research team liaisons with practice staff and leaders. Data collectors and research team members were trained in the use of the context template for data collection and reporting before baseline data collection.

Contextual data were prospectively collected between March and December 2013. Data collectors used the context template as a guide to conduct brief interviews with multiple stakeholders at each practice site, including clinic leaders, clinicians, and staff. They also conducted observations of clinic activities such as patient flow, practice workflow, and interactions among staff to supplement interview data. Field notes from interviews and observations were recorded by practice number on the context template. Data collectors were encouraged to collect and record direct quotes. They then forwarded the completed templates to members of the MOHR context workgroup22 for further data summary and subsequent thematic analysis. The MOHR context workgroup was multidisciplinary, with members representing primary care medicine, epidemiology, anthropology, and health behavior sciences.

For quality control, conference calls were held with research teams before and halfway through implementation to discuss each practice's approach to collecting qualitative data and to problem-solve challenges to completing the context template. Some variations in data collection methods were identified; for example, some sites completed interviews in person, whereas others completed them by phone. Two sites collected data for only 2 of the 3 time points, leading to some missing data at the midpoint and at the end of the implementation period.

Research teams also collected quantitative data on patient reach, defined as the number, proportion, and representativeness of eligible patients offered and completing the MOHR assessment.28 Patient reach was calculated by dividing the number of patients who completed the MOHR assessment by the number of patients offered the MOHR.15

### Data Analysis

The context template served as raw data that were uploaded into Atlas.ti (version 7.0; Scientific Software Development GmbH, Berlin, Germany) for coding and analysis. The coding scheme was chosen a priori based on the model developed by Stange and Glasgow.2 At least 2 workgroup members independently coded each practice's context template. Any coding discrepancies were resolved through discussion among the multidisciplinary team. Data were compared across geographic locations (urban, semiurban, rural), networks (practice-based research network vs federally qualified health center, and MOHR administration types (Article-based, online and faxed to office, online and printed at office). Passages of coded text were coalesced to form higher-level themes through a multistage, iterative data immersion process. Excerpts of field notes provided in the Results section of this article were selected to represent these higher-level themes. The study was approved by institutional review boards at the Virginia Commonwealth University (no. HM12746), University of California, Los Angeles (no. 12-0017900), and 5 other participating institutions.

## Results

MOHR practices varied with respect to size, ownership, health system affiliation, geographic location, and patient sociodemographics (Table 1). This variation was intentional to enhance generalizability of the findings.

View this table:
[Table 1.](http://www.jabfm.org/content/30/3/337/T1)

Table 1. 
Descriptive Characteristics of the My Own Health Report Primary Care Practices (March-December 2013)

Contextual factors influencing MOHR implementation and patient reach included factors both internal and external to the practice. Below, we describe in more detail how these factors influenced MOHR implementation and patient reach, and Table 2 provides representative quotations and field notes exemplifying the identified factors.

View this table:
[Table 2.](http://www.jabfm.org/content/30/3/337/T2)

Table 2. 
Contextual Factors Related to Implementation and Patient Reach* of the My Own Health Report Health Risk Assessment

### Factors Internal to the Practice

Internal factors included practice staff members' motivation to use MOHR, practice staff's capacity to take on additional responsibilities to facilitate MOHR administration, and practices' information system capacity.

#### Practice Members' Motivations

Practice leaders and staff members were motivated to adopt the MOHR tool because it would enable them to systematically identify patients with unhealthy behaviors and mental health concerns. In particular, patient and provider reports generated by the MOHR tool helped streamline the goal-setting process by easily identifying patients' risk factors and highlighting the behaviors patients were ready to work on. In addition to facilitating goal setting, some practice leaders perceived that implementing MOHR could be helpful in meeting reporting requirements to external agencies, patient-centered medical home certification, or criteria for meaningful use of the electronic medical record (EMR). Such motivations for adopting MOHR greatly facilitated startup of the intervention in practices.

However, enthusiasm waned over time, even among practices that were initially motivated and that perceived MOHR to be useful to clinicians and patients. This is because MOHR added a significant time burden to visits, which resulted in implementation challenges. In addition, in 2 practices, some of the health assessment questions were already part of existing patient intake forms, and practice members were concerned from the outset that implementing MOHR created an element of redundancy and would add more time to patient visits that were already running over schedule.

#### Practice Staff Capacity

We observed that existing staff modified their roles or took on additional duties to implement MOHR. For instance, among practices that administered MOHR in the office, the medical assistants took on additional responsibilities in assisting patients who needed help completing it. In practices where patients completed the MOHR online at home, practice staff adapted their roles to locate the completed physician reports and include them in the goal-setting discussion. When practices members did not have internal capacity to take on additional responsibilities to implement MOHR, research teams assisted with implementation. This was often the case among federally qualified health center practices that had large patient volumes and a higher proportion of underserved patients.

#### Practice Information System Capacity

Practice information systems capacity was related to MOHR implementation. Delays in printing or receiving faxes of the MOHR reports disrupted clinical workflow. This resulted in fewer visits that included goal setting because patient and/or provider reports were not available at the time of the visit. In addition, several practices experienced challenges with their existing technology infrastructure, such as changing EMR systems and low use of patient portals, which hindered MOHR implementation.

### External Factors Influencing MOHR Implementation

Availability of external resources to support quality improvement emerged as an important contextual factor relevant to MOHR implementation, in particular, support from larger health care systems, practice linkages with community resources, and fit of the implementation strategy with patient populations.

#### Linkages with a Larger Health Care System

Practices organized within larger health care systems had additional support for implementation that they could leverage and use. This included information technology support as well as help from research or health system staff members to implement quality improvement initiatives. Significant support from 1 practice's affiliated health care system's nurse operators helped increase patient reach of the MOHR intervention. Practices that had such support often were able to leverage internal and external resources to facilitate implementation and even make midcourse changes to implementation when necessary, as described here. At baseline, 1 practice decided to test a comprehensive implementation approach that included (1) mailing MOHR invitations to patients' homes, (2) inviting patients to complete MOHR before their appointment, or (3) inviting patients to complete MOHR over the phone. In the first few weeks, however, they learned that these approaches were unable to reach a large proportion of their target population. Therefore, they modified their approach midstream by seeking help from their health care system to have additional staff administer the MOHR. This resulted in a substantial increase in the proportion of patients reached over subsequent weeks. On the other hand, when a practice belonged to a larger network of clinics, they could compete for staff time and clinic resources, thereby hindering implementation.

#### Linkages with Community Resources

Another contextual factor that posed a barrier to goal setting was practices' lack of established linkages with community resources to refer patients who needed additional counseling for unhealthy behaviors or mental health issues. Further, the lack of a systematic referral process for more intensive counseling also contributed to inconsistent use of the MOHR tool.

#### Fit of Practice Implementation Strategy with Patients

Practices' patient panel characteristics (age, predominant language spoken, and health literacy) were an important contextual factor that influenced practices' decisions on how MOHR would be administered to patients. Practices that served large panels of low-income and non-English-speaking patients administered the MOHR in the office, rather than at home via the web, because they perceived that most of their patients would not have consistent Internet access or access to computers at home. Even when administered in the office, patients of these practices needed significant help from practice members or research staff to complete the MOHR on laptop computers or tablets. Anther concern that affected the mode of administration was the ability of older patients to navigate a web-based tool.

### Interaction Between Contextual Factors

The contextual factors described above often interacted, exacerbating the challenges to successful implementation of the MOHR. To illustrate, practice information capacity limitations, along with sustained lack of support for the new quality improvement initiative and demotivated staff, specifically presented challenges to MOHR implementation for practice 6: The MOHR report often does not get to them [physicians] in time for their visit with the patient so the information is not utilized; when it is scanned into the EMR, it is not easily located and they do not have [the] motivation to search for it at the next visit (they also do not know who has filled it out and who hasn't). (site 6, time point 2)The “physician champion” did not really turn out to be a champion of the project because he said the reports did not get into their medical record in a timely manner and there was no time within the patient visits to address additional issues. As with the other providers, the lead physician and nursing director were not very impressed with the MOHR and did not feel that it added value to their patient care. (site 6, time point 3) This example highlights how interaction of several contextual factors resulted in breakdowns in the process of MOHR reports reaching clinicians and patients during the visit and a lack of integration of MOHR into existing clinic workflow for this practice.

## Discussion

Prospectively assessing contextual factors in a pragmatic trial conducted in primary care revealed factors both within and external to the practice environment as influencing implementation and patient reach. Quality improvement intervention studies rarely collect systematic data on contextual factors. Even rarer are intervention studies that collect contextual data at multiple points over time. This is especially problematic because numerous rapid quality improvement cycles are needed to implement an improvement initiative, and contextual factors facilitating or hindering these improvement cycles are often lost if not assessed in real time.29 This prospective approach in the MOHR study identified key contextual factors, including practice members' motivations toward using the intervention, practice capacity, quality improvement support available to practices, linkages to community resources, and patient panel characteristics.

The MOHR study was designed and implemented as a pragmatic trial,22 such that practices could and did tailor MOHR implementation to suit their local setting. Despite this flexibility, additional contextual factors hindered implementation, including practices' capacity to take on a new quality improvement initiative, practice members' motivation to change, and resources available to the practice in order to support change. Other primary care change initiatives also identified these factors as particularly salient because they require significant changes to practice workflow and are potentially disruptive to practice functioning, suggesting that these may be important to consider for most practice change initiatives.15,16,30,31

In addition to these general contextual factors, our prospective method helped identify variations across practices in goal setting for unhealthy behaviors, even though the study's main findings demonstrated an increase in goal setting among intervention compared with control practices.25 Technology challenges in accessing patient and clinician MOHR reports at the point of care, coupled with the additional time needed to complete MOHR, hindered goal setting. Practices' limited external linkages with community resources for behavioral health counseling (for example smoking cessation and physical activity counseling) challenged implementation, as clinicians perceived no benefit in setting goals with patients if there was no place to send them for additional counseling. Paying attention to context throughout the study helped explain observed variations in implementation and, more importantly, helped identify conditions under which goal setting was more likely to be successfully implemented. Our findings could enable other practices seeking to implement an electronic health risk assessment tool to identify, in advance, “real-world” trade-offs to integrating it in their workflow. And, as our findings suggest, these trade-offs may change over time as implementation proceeds within the practice.

These findings should be interpreted in light of the study's strengths and limitations. While participating practices were very diverse, they were volunteer practices from research networks and thus are unlikely to be representative of all primary care practices. None of the participating practices sustained MOHR after the study's completion,23 so our observations of implementation were restricted in time. Nevertheless, our study shed light on reasons for the lack of sustainability. For instance, the significant additional time and staff resources needed to administer MOHR made it impossible for practices with large volumes of underinsured and uninsured patients to integrate it into their daily workflow. Thus, MOHR was discontinued once the study ended. Even in the practice-based research network practices, additional, unreimbursed time for goal setting made it difficult to sustain MOHR after the end of the study.

The study's methods, involving case studies, may have further limited generalizability. Nonetheless, randomized controlled trials offer only average measures of effectiveness and are context-specific, whereas a series of case studies in different contexts could provide valuable information about how an intervention operates, as with the findings from this study.15 Another, more subtle limitation of the study concerned the context of the study itself. The MOHR study was developed, funded, and led by a national study team based at the National Institutes of Health, with an academic coordinating center that changed during the course of the project. Other than the impact of national reimbursement and reporting requirements, however, respondents did not report any effects of the study aegis, leadership, or even the research staff itself (except as staff “extenders” to administer the MOHR) on implementation.

Notwithstanding these limitations, the study has some significant strengths, most notably the prospective collection of data over multiple assessment times and the diversity of clinics. Context is increasingly recognized as important, but few primary care implementation studies explicitly collect and report on contextual factors, and fewer still do so prospectively throughout implementation.2,15 Paying attention to contextual factors throughout the course of this study helped identify key factors resulting in implementation challenges that would not have otherwise been recognized.

The methodology used in this study can be helpful to both researchers and practicing clinicians. We recommend the use of the Context Matters template to systematically and prospectively capture data on contextual factors at multiple levels (practice, community, and state). This method will enable researchers to identify factors that may influence implementation differentially by practice context. Reporting on contextual factors using this method may also help practices assess whether identified factors are relevant to them when implementing MOHR or a similar health risk assessment (HRA). Our study findings suggest that enhanced capacity to make quality improvement changes in a practice where practice members are motivated and that has resources to make and sustain changes is critically important for the successful implementation of most quality improvement interventions. Specifically, for practices wishing to implement MOHR, we recommend that they pay attention to their information technology capacities to effectively administer MOHR via patient portals or using web-enabled tablets in the office, and to modify their workflows to account for the additional visit time needed for the important task of goal setting.

## Conclusion

Understanding practice contexts can be used to successfully implement HRAs as a part of the Medicare annual wellness visit and as part of routine care. Involvement of diverse stakeholders in gathering and interpreting data on relevant contextual factors over time can advance the understanding of what happened with a particular intervention and why, and can allow others to make reasonable judgments about how an intervention or its implementation might need to be modified in order to be effectively executed in different settings and circumstances.

## Appendix

#### MOHR Context Matters Template

![Figure1](http://www.jabfm.org/https://www.jabfm.org/content/jabfp/30/3/337/F1.medium.gif)

[Figure1](http://www.jabfm.org/content/30/3/337/F1)

## Notes

*   This article was externally peer reviewed.

*   *Funding:* none.

*   *Conflict of interest:* none declared.

*   To see this article online, please go to: [http://jabfm.org/content/30/3/337.full](http://jabfm.org/content/30/3/337.full).

*   Received for publication May 9, 2016.
*   Revision received October 17, 2016.
*   Accepted for publication January 23, 2017.

## References

1.  1.Kaplan HC, Provost LP, Froehle CM, Margolis PA. The Model for Understanding Success in Quality (MUSIQ): building a theory of context in healthcare quality improvement. BMJ Qual Saf 2012;21:13–20.
    
    [Abstract/FREE Full Text](http://www.jabfm.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MzoicWhjIjtzOjU6InJlc2lkIjtzOjc6IjIxLzEvMTMiO3M6NDoiYXRvbSI7czoyMDoiL2phYmZwLzMwLzMvMzM3LmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

2.  2.Stange KC, Glasgow RE. Considering and reporting important contextual factors in research on the patient-centered medical home. Rockville, MD: Agency for Healthcare Research and Quality; 2013.
    
    

3.  3.Badger F, Plumridge G, Hewison A, Shaw KL, Thomas K, Clifford C. An evaluation of the impact of the Gold Standards Framework on collaboration in end-of-life care in nursing homes. A qualitative and quantitative evaluation. Int J Nurs Stud 2012;49:586–95.
    
    [CrossRef](http://www.jabfm.org/lookup/external-ref?access_num=10.1016/j.ijnurstu.2011.10.021&link_type=DOI) 
    
    [PubMed](http://www.jabfm.org/lookup/external-ref?access_num=22130508&link_type=MED&atom=%2Fjabfp%2F30%2F3%2F337.atom) 

4.  4.Chaney EF, Rubenstein LV, Liu CF, et al. Implementing collaborative care for depression treatment in primary care: a cluster randomized evaluation of a quality improvement practice redesign. Implement Sci 2011;6:121.
    
    [CrossRef](http://www.jabfm.org/lookup/external-ref?access_num=10.1186/1748-5908-6-121&link_type=DOI) 
    
    [PubMed](http://www.jabfm.org/lookup/external-ref?access_num=22032247&link_type=MED&atom=%2Fjabfp%2F30%2F3%2F337.atom) 

5.  5.Macq J, Solis A, Martinez G, Martiny P. Tackling tuberculosis patients' internalized social stigma through patient centred care: an intervention study in rural Nicaragua. BMC Public Health 2008;8:154.
    
    [CrossRef](http://www.jabfm.org/lookup/external-ref?access_num=10.1186/1471-2458-8-154&link_type=DOI) 
    
    [PubMed](http://www.jabfm.org/lookup/external-ref?access_num=18466604&link_type=MED&atom=%2Fjabfp%2F30%2F3%2F337.atom) 

6.  6.Shaw RJ, Kaufman MA, Bosworth HB, et al. Organizational factors associated with readiness to implement and translate a primary care based telemedicine behavioral program to improve blood pressure control: the HTN-IMPROVE study. Implement Sci 2013;8:106.
    
    

7.  7.Van Herck P, De Smedt D, Annemans L, Remmen R, Rosenthal MB, Sermeus W. Systematic review: effects, design choices, and context of pay-for-performance in health care. BMC Health Serv Res 2010;10:247.
    
    [CrossRef](http://www.jabfm.org/lookup/external-ref?access_num=10.1186/1472-6963-10-247&link_type=DOI) 
    
    [PubMed](http://www.jabfm.org/lookup/external-ref?access_num=20731816&link_type=MED&atom=%2Fjabfp%2F30%2F3%2F337.atom) 

8.  8.Moher D, Hopewell S, Schulz KF, et al. CONSORT 2010 explanation and elaboration: updated guidelines for reporting parallel group randomised trials. J Clin Epidemiol 2010;63:e1–37.
    
    [CrossRef](http://www.jabfm.org/lookup/external-ref?access_num=10.1016/j.jclinepi.2010.03.004&link_type=DOI) 
    
    [PubMed](http://www.jabfm.org/lookup/external-ref?access_num=20346624&link_type=MED&atom=%2Fjabfp%2F30%2F3%2F337.atom) 

9.  9.Schulz KF, Altman DG, Moher D. CONSORT 2010 statement: updated guidelines for reporting parallel group randomized trials. Ann Intern Med 2010;152:726–32.
    
    [CrossRef](http://www.jabfm.org/lookup/external-ref?access_num=10.7326/0003-4819-152-11-201006010-00232&link_type=DOI) 
    
    [PubMed](http://www.jabfm.org/lookup/external-ref?access_num=20335313&link_type=MED&atom=%2Fjabfp%2F30%2F3%2F337.atom) 
    
    [Web of Science](http://www.jabfm.org/lookup/external-ref?access_num=000278560000006&link_type=ISI) 

10. 10.Green LW, Glasgow RE. Evaluating the relevance, generalization, and applicability of research: issues in external validation and translation methodology. Eval Health Prof 2006;29:126–53.
    
    [CrossRef](http://www.jabfm.org/lookup/external-ref?access_num=10.1177/0163278705284445&link_type=DOI) 
    
    [PubMed](http://www.jabfm.org/lookup/external-ref?access_num=16510882&link_type=MED&atom=%2Fjabfp%2F30%2F3%2F337.atom) 
    
    [Web of Science](http://www.jabfm.org/lookup/external-ref?access_num=000235450900006&link_type=ISI) 

11. 11.Dobrow MJ, Goel V, Lemieux-Charles L, Black NA. The impact of context on evidence utilization: a framework for expert groups developing health policy recommendations. Soc Sci Med 2006;63:1811–24.
    
    [CrossRef](http://www.jabfm.org/lookup/external-ref?access_num=10.1016/j.socscimed.2006.04.020&link_type=DOI) 
    
    [PubMed](http://www.jabfm.org/lookup/external-ref?access_num=16764980&link_type=MED&atom=%2Fjabfp%2F30%2F3%2F337.atom) 

12. 12.May CR, Mair FS, Dowrick CF, Finch TL. Process evaluation for complex interventions in primary care: understanding trials using the normalization process model. BMC Fam Pract 2007;8:1–9.
    
    [CrossRef](http://www.jabfm.org/lookup/external-ref?access_num=10.1186/1471-2296-8-1&link_type=DOI) 
    
    [PubMed](http://www.jabfm.org/lookup/external-ref?access_num=17201905&link_type=MED&atom=%2Fjabfp%2F30%2F3%2F337.atom) 
    
    [Web of Science](http://www.jabfm.org/lookup/external-ref?access_num=000251689100001&link_type=ISI) 

13. 13.McDonald KM, Schultz EM, Chang C. Evaluating the state of quality-improvement science through evidence synthesis: insights from the closing the quality gap series. Perm J 2013;17:52–61.
    
    [CrossRef](http://www.jabfm.org/lookup/external-ref?access_num=10.7812/TPP/13-010&link_type=DOI) 
    
    [PubMed](http://www.jabfm.org/lookup/external-ref?access_num=24079357&link_type=MED&atom=%2Fjabfp%2F30%2F3%2F337.atom) 

14. 14.Rojas Smith L, Ashok M, Dy SM, Wines RC, Teixeira-Poit S. Contextual frameworks for research on the implementation of complex system interventions. Rockville, MD: Agency for Healthcare Research and Quality; 2014.
    
    

15. 15.Tomoaia-Cotisel A, Scammon DL, Waitzman NJ, et al. Context matters: the experience of 14 research teams in systematically reporting contextual factors important for practice change. Ann Fam Med 2013;11(Suppl 1):S115–23.
    
    [Abstract/FREE Full Text](http://www.jabfm.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6ODoiYW5uYWxzZm0iO3M6NToicmVzaWQiO3M6MTU6IjExL1N1cHBsXzEvUzExNSI7czo0OiJhdG9tIjtzOjIwOiIvamFiZnAvMzAvMy8zMzcuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

16. 16.Kaplan HC, Brady PW, Dritz MC, et al. The influence of context on quality improvement success in health care: a systematic review of the literature. Milbank Q 2010;88:500–59.
    
    [CrossRef](http://www.jabfm.org/lookup/external-ref?access_num=10.1111/j.1468-0009.2010.00611.x&link_type=DOI) 
    
    [PubMed](http://www.jabfm.org/lookup/external-ref?access_num=21166868&link_type=MED&atom=%2Fjabfp%2F30%2F3%2F337.atom) 
    
    [Web of Science](http://www.jabfm.org/lookup/external-ref?access_num=000285446400005&link_type=ISI) 

17. 17.Damschroder LJ, Aron DC, Keith RE, Kirsh SR, Alexander JA, Lowery JC. Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implement Sci 2009;4:50.
    
    [CrossRef](http://www.jabfm.org/lookup/external-ref?access_num=10.1186/1748-5908-4-50&link_type=DOI) 
    
    [PubMed](http://www.jabfm.org/lookup/external-ref?access_num=19664226&link_type=MED&atom=%2Fjabfp%2F30%2F3%2F337.atom) 

18. 18.Kitson A, Harvey G, McCormack B. Enabling the implementation of evidence based practice: a conceptual framework. Qual Health Care 1998;7:149–58.
    
    [Abstract](http://www.jabfm.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MzoicWhjIjtzOjU6InJlc2lkIjtzOjc6IjcvMy8xNDkiO3M6NDoiYXRvbSI7czoyMDoiL2phYmZwLzMwLzMvMzM3LmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

19. 19.Bergstrom A, Skeen S, Duc DM, et al. Health system context and implementation of evidence-based practices-development and validation of the Context Assessment for Community Health (COACH) tool for low- and middle-income settings. Implement Sci 2015;10:120.
    
    

20. 20.Harvey G, Kitson A. PARIHS revisited: from heuristic to integrated framework for the successful implementation of knowledge into practice. Implement Sci 2016;11:33.
    
    

21. 21.Glasgow RE, Kessler RS, Ory MG, Roby D, Gorin SS, Krist A. Conducting rapid, relevant research: lessons learned from the My Own Health Report project. Am J Prev Med 2014;47:212–9.
    
    [CrossRef](http://www.jabfm.org/lookup/external-ref?access_num=10.1016/j.amepre.2014.03.007&link_type=DOI) 
    
    [PubMed](http://www.jabfm.org/lookup/external-ref?access_num=24953520&link_type=MED&atom=%2Fjabfp%2F30%2F3%2F337.atom) 

22. 22.Krist AH, Glenn BA, Glasgow RE, et al. Designing a valid randomized pragmatic primary care implementation trial: the my own health report (MOHR) project. Implement Sci 2013;8:73.
    
    [CrossRef](http://www.jabfm.org/lookup/external-ref?access_num=10.1186/1748-5908-8-73&link_type=DOI) 
    
    [PubMed](http://www.jabfm.org/lookup/external-ref?access_num=23799943&link_type=MED&atom=%2Fjabfp%2F30%2F3%2F337.atom) 

23. 23.Krist AH, Phillips SM, Sabo RT, et al. Adoption, reach, implementation, and maintenance of a behavioral and mental health assessment in primary care. Ann Fam Med 2014;12:525–33.
    
    [Abstract/FREE Full Text](http://www.jabfm.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6ODoiYW5uYWxzZm0iO3M6NToicmVzaWQiO3M6ODoiMTIvNi81MjUiO3M6NDoiYXRvbSI7czoyMDoiL2phYmZwLzMwLzMvMzM3LmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

24. 24.Phillips SM, Glasgow RE, Bello G, et al. Frequency and prioritization of patient health risks from a structured health risk assessment. Ann Fam Med 2014;12:505–13.
    
    [Abstract/FREE Full Text](http://www.jabfm.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6ODoiYW5uYWxzZm0iO3M6NToicmVzaWQiO3M6ODoiMTIvNi81MDUiO3M6NDoiYXRvbSI7czoyMDoiL2phYmZwLzMwLzMvMzM3LmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

25. 25.Krist AH, Glasgow RE, Heurtin-Roberts S, et al; MOHR Study Group. The impact of behavioral and mental health risk assessments on goal setting in primary care. Transl Behav Med 2016;6:212–9.
    
    

26. 26.National Cancer Institute. MyOwnHealthReport. 2016. Available from: [http://myownhealthreport.org/](http://myownhealthreport.org/). Accessed January 11, 2017.
    
    

27. 27.1.  Crabtree BF, 
    2.  Miller WL
    
    Borkan J. Immersion/crystallization. In: Crabtree BF, Miller WL, editors. Doing qualitative research. Thousand Oaks, CA: Sage Publications; 1999. pp. 179–94.
    
    

28. 28.Glasgow RE, Vogt TM, Boles SM. Evaluating the public health impact of health promotion interventions: the RE-AIM framework. Am J Public Health 1999;89:1322–7.
    
    [CrossRef](http://www.jabfm.org/lookup/external-ref?access_num=10.2105/ajph.89.9.1322&link_type=DOI) 
    
    [PubMed](http://www.jabfm.org/lookup/external-ref?access_num=10474547&link_type=MED&atom=%2Fjabfp%2F30%2F3%2F337.atom) 
    
    [Web of Science](http://www.jabfm.org/lookup/external-ref?access_num=000082214600005&link_type=ISI) 

29. 29.Balasubramanian BA, Cohen DJ, Davis MM, et al. Learning evaluation: blending quality improvement and implementation research methods to study healthcare innovations. Implement Sci 2015;10:31.
    
    [PubMed](http://www.jabfm.org/lookup/external-ref?access_num=25889831&link_type=MED&atom=%2Fjabfp%2F30%2F3%2F337.atom) 

30. 30.Nutting PA, Crabtree BF, Stewart EE, et al. Effect of facilitation on practice outcomes in the National Demonstration Project model of the patient-centered medical home. Ann Fam Med 2010;8(Suppl 1):S33–44; S92.
    
    [Abstract/FREE Full Text](http://www.jabfm.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6ODoiYW5uYWxzZm0iO3M6NToicmVzaWQiO3M6MTM6IjgvU3VwcGxfMS9TMzMiO3M6NDoiYXRvbSI7czoyMDoiL2phYmZwLzMwLzMvMzM3LmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

31. 31.Shaw EK, Chase SM, Howard J, Nutting PA, Crabtree BF. More black box to explore: how quality improvement collaboratives shape practice change. J Am Board Fam Med 2012;25:149–57.
    
    [Abstract/FREE Full Text](http://www.jabfm.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NToiamFiZnAiO3M6NToicmVzaWQiO3M6ODoiMjUvMi8xNDkiO3M6NDoiYXRvbSI7czoyMDoiL2phYmZwLzMwLzMvMzM3LmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==)