Abstract
Background: Health-related scientific discoveries are often not applied in clinical settings after publication, even when recommended by a trusted journal or professional association. This article describes an assessment tool we developed for use by primary care clinicians and practice administrators to evaluate whether to implement recommended evidence-based interventions in their practices.
Methods: We used dissemination and implementation theory to develop a worksheet to guide decision making about whether interventions are suitable for implementation in primary care practice settings. We tested the tool by analyzing how members of a primary care practice-based research network rated 4 evidence-based interventions.
Results: The median likelihood of implementation ranged from 2 to 3.5 on a scale of 1 (low) to 5 (high). Raters' level of agreement with statements about 3 intervention characteristics was associated (P < .05) with a higher likelihood of implementation using Spearman rank-order correlation: simple to implement, testable before fully implementing, and modifiable to meet the needs of the practice. Raters found the worksheet helpful in thinking through potential implementation, especially the prompts about modifiability and relevance to the practice's patients and priorities.
Conclusions: The Decision-to-Implement Worksheet provides a new resource for primary care practices that want to assess whether evidence-based interventions are suitable to adopt or adapt to meet their needs.
- Decision Making
- Evidence-Based Medicine
- Practice-based Research
- Primary Health Care
- Translational Medical Research
Implementation of health-related research findings into primary care clinical practice is an enduring challenge. The differing priorities of researchers and primary care practitioners has resulted in a body of evidence that is not primarily designed with external validity or end-user applicability in mind.1 At the same time, the competing demands of day-to-day health care delivery make it challenging for frontline practitioners and staff in primary care practice to identify, evaluate, and implement new evidence.2
A range of conceptual frameworks have been developed both to guide researchers as they test dissemination and implementation strategies and to enhance the value of dissemination and implementation research for its end users.3,4 While the frameworks' purpose and components vary, there is a cross-cutting emphasis on the idea of the fit of evidence with practice setting, which is influenced by the details of how a study is conducted and reported as well as practice organizational readiness and resources.5 This theoretical work has resulted in methods to identify studies that are methodologically rigorous and include attention to external validity and pragmatic design considerations.6,7 In addition, efforts to disseminate evidence-based innovations are increasing in journals and services such as Patient-Oriented Evidence that Matters (POEM),8 plain-language summaries from the Cochrane Library,9 and Strength of Recommendation Taxonomy.10 Professional organizations (e.g., the American Academy of Family Physicians) and panels like the US Preventive Services Task Force review and interpret evidence to help primary care decision makers access the most relevant findings from the vast array of new research published every month. There are also a number of approaches and tools to guide researchers and program planners to identify and tailor evidence-based programs to meet practices' needs.11 For example, Canada has a National Collaborating Centre for Methods and Tools to support evidence-informed public health through functions including a registry of methods and tools.12
Missing from the literature and guidance are evidence-based methods and tools to help practitioners and practice administrators—the end users of research—think through whether potentially applicable innovations that they identify through evidence-synthesis summaries such as those described above are applicable and adaptable to their local needs. Theoretical models about the uptake and adoption of knowledge focus on the implementation process but tend to be one-directional, from researchers to implementers/users.13 Further, a “decision maker” is often framed as someone operating at the policy level, typically through public health units.14 Clinical practices' decision making process about whether to adopt innovations is much less studied than researchers' recommendations to implement. End-user decision making entails a different and more granular process, rooted in specific clinical contexts and based on local needs, than that which is supported in the literature and by existing knowledge translation tools.
This article describes the development and testing of the Decision-to-Implement Worksheet to guide decision making by primary care clinicians and practice administrators about whether evidence-based interventions are suitable to implement in their clinical practices to meet specific local needs. This initiative is part of broader evidence translation efforts in the Institute of Translational Health Sciences: the Clinical and Translational Science Award (CTSA) program, which works to speed the translation of scientific discoveries into clinic practice for the benefit of patients and communities throughout Washington, Wyoming, Alaska, Montana, and Idaho (WWAMI).
Methods
Theoretical Basis for the Decision-to-Implement Worksheet
Theory provides an important foundation for evidence-based practice implementation strategies and tools.15 We drew on conceptual insights from the Consolidated Framework for Implementation Research16 and the Model of Diffusion in Service Organizations17 as the basis for a tool to guide decision making about the implementation of findings, as described above, into primary care practice. Both of these frameworks combine existing theory, and they highlight that spread occurs in the context of a system of interdependent factors and depends on the fit of an innovation within the context of a new setting. These frameworks suggest that implementation and spread develop from interactions between the intervention, the inner and outer settings, the individuals involved, and the process by which implementation is accomplished.16 Aspects of each of these interacting components can promote or impede innovation. For example, Fleuren et al11 identified 15 determinants related to the individuals involved.
We considered which aspects of these frameworks are most relevant to support the implementation component considered in this study—primary care providers' and administrators' evaluation of new evidence in their local setting. Greenhalgh et al17 emphasize that compatible interventions have to meet perceived needs and ways of working, drawing on the insights of Everett Rogers18 that diffusion of innovation depends on potential end-user perceptions of relative advantage, compatibility, complexity, observability, and trialability (the ability to try innovations on a limited basis before full implementation). Several theories also highlight that interventions require adaptation, but those requiring less adaptation are simpler to implement because they involve less of an active change process to incorporate the innovation into the organizational ecosystem.16 In addition to characteristics of the intervention, compatibility depends on characteristics of the setting, such as available resources, and supports for adoption play an important role in a decision to implement.16,17
Decision-to-Implement Worksheet Item Development
We searched for publications and white papers describing existing decision making tools relevant to the conceptual domains discussed above. We looked for review articles available through PubMed using a combination of search terms and key words related to primary care, tools (tools, methods, resources, models, frameworks), and knowledge transfer (knowledge transfer, evidence-based medicine, innovation, dissemination, translation into practice, spread, translational science, adoption, diffusion of innovation, implementation, quality improvement). We also identified articles referenced in select manuscripts and included instruments nominated by health services researchers and leaders in the Institute of Translational Health Sciences network. We located a number of tools related to evidence-based medicine and program implementation in the National Coordinating Center for Methods and Tools' Registry of Methods and Tools.12 These tools support many components of the evidence implementation process, for example, critical appraisal of the methodological quality of evidence by clinicians and evaluation of a practice's organizational capacity for change by an implementation program. However, none relate to the specific process of end-user decision making about implementation that factors in local needs and workflows.
Therefore we developed new questions based on the key theoretical constructs identified above. These questions asked about the intervention (eg, relevance) and system resources and supports: additional training, changes in team workflow/tasks, modifications to information technology systems, new/additional financial support, and support from organizational leadership. Two potentially relevant theoretical constructs were not included in the worksheet after careful consideration. First, strength of evidence, a factor contributing to an intervention's relative advantage, was excluded because there are a number of existing efforts to appraise the quality of evidence (as described earlier); we postulate that end-user consideration of the fit of already-established strong evidence is more pertinent than additional evaluation of study quality. Second, observability is a facilitator of decision making but would not be relevant when reviewing a published article; instead, this would be evaluated when considering evidence when learning from peers, for example, at site visits.
To triangulate responses to worksheet items, a single 5-point Likert scale item asked about the overall likelihood of implementation of the research finding within the next year. The other questions were standardized into 4-point Likert scale items designed for use by primary care providers and practice administrators. The resulting Decision-to-Implement Worksheet was pilot tested with 3 colleagues at other CTSAs who are practicing clinicians with expertise in evidence translation. We gave the pilot testers a sample article to review and asked them to focus on whether the questions made sense in the context of the exercise. We made minor changes to question order and wording based on their feedback and added an item asking about “relevance to my patient population,” expanding on the previous operationalization of compatibility as relevant to primary care in general.
Testing the Tool
Mirroring the process that a professional association might use to identify relevant evidence for its constituents, we conducted a prescreening process to narrow the evidence base. We identified a convenience sample of interventions with potential relevance to challenges faced by primary care practices. The prescreening tool (available at researchtoolkit.org) included questions about study type, significance of findings, and relevance to primary care practices. Studies were included if they were systematic reviews or randomized controlled trials, had significant findings, and focused on outcomes that are a common or serious problem in primary care. Given that this is an exploratory study, we did not assess the methodological quality of individual studies.
The source we used to identify the convenience sample of research findings was publications in 2013 from Group Health Research Institute–associated researchers. The year 2013 was selected as the most recent complete year available, making this set of publications the most timely and appropriate for this pilot. Of 446 articles, a total of 28 potentially implementable innovations were identified (Figure 1). To narrow interventions to those that pertained to local needs, a subset of 4 (Table 1) were identified as most compatible with the clinical priorities that had previously been identified by practices within the WWAMI region Practice and Research Network (WPRN), a network of >50 primary care practices in WWAMI that have committed to research collaboration.
Steps used to identify evidence-based interventions from Group Health 2013 publications. *From database of peer-reviewed publications in 2013 by Group Health Research Institute–affiliated researchers; **This number worked best with the configuration of the practice-based research network's annual meeting; depending on a group's objectives, a larger or smaller number could be selected. WPRN, WWAMI region Practice and Research Network.
Innovations Identified from 2013 Group Health Publications and Reviewed by 26 WWAMI Region Practice and Research Network Clinicians Using the Decision-to-Implement Worksheet
Each of the 4 selected interventions was then rated in person using the Decision-to-Implement Worksheet at the WPRN′s annual meeting by a group of primary care–based clinicians from practices affiliated with organizations such as hospital-affiliated health centers, community-based health care systems, and federally qualified health centers. Each participant reviewed a synopsis of an article describing the innovation, with the article available as a reference if needed; completed the worksheet individually; and then joined a group for a conversation that focused on their article to discuss their reasons for ratings. A facilitator recorded these comments. We used SPSS to calculate descriptive statistics and correlation between worksheet items based on the Spearman rank-order test (SPSS Statistics for Windows; IBM Corp., Armonk, NY). Correlations were calculated for all interventions combined because of the small sample size for individual interventions. Statistical inference was based on 2-sided tests with α = 0.05. Blank responses and those noted as not applicable were treated as missing data in the statistical analysis but were tabulated in the presentation of results. We also reviewed the discussion notes and synthesized themes that emerged. The Group Health human subjects review office determined that the procedures did not require review by the institutional review board.
Results
A total of 26 primary care clinicians each reviewed 1 of the 4 studies in groups of 6 or 7. The median likelihood of implementation within the next year ranged from 2 to 3.5 on a scale of 1 (low) to 5 (high) (Table 2). Raters' level of agreement with worksheet statements about 3 intervention characteristics was associated with the reported likelihood of implementation: simple to implement, testable before fully implementing, and modifiable to meet the needs of the practice.
Decision-to-Implement Worksheet Ratings, by Intervention and Concept
Themes from the Discussions
The themes from the discussions fell into the categories of relevance to patients, level of effort and fit with workflow, cost/return on investment, clinician/leader buy-in, and effectiveness. Guide to Decide,19 a Web-based decision aid about breast cancer chemoprevention, received the lowest rating for likelihood of implementation. The group summarized their view as “we have so many other battles to fight,” and did not identify breast cancer chemoprevention as a priority. Several expressed concerns about the utility of the decision aid for patients with limited English proficiency or Internet access. One clinician noted a concern that the Web-based decision aid was not the same as a 1-on-1 discussion with a provider.
Ask-Advise-Connect,20 which enables practices to connect patients who smoke directly to the state quitline, was rated as most likely to be implemented. Participants saw value in how the intervention addressed a high-priority issue in a manner that fit with practice flow. They raised a number of questions about the effectiveness and sustainability of the intervention, for example, how often treatment resulted in smoking cessation, how quitlines might vary by state, how to calculate return on investment, and what would happen if the state quitline was defunded.
The group that discussed Electronic Communications and Home Blood Pressure Monitoring to Improve Blood Pressure Control,21 a pharmacist-based collaborative care approach to hypertension control, noted that it improved outcomes for hypertension but not necessarily in a cost-effective manner. One rater pointed out that the intervention could be adapted to use personnel receiving lower salaries. The group identified some potential limits to generalizability, including that not all patients have access to a patient portal and that patients with comorbidities were excluded from the study.
The group discussing TEAMcare,22 a collaborative care approach for patients with multiple chronic conditions, noted positive features of the intervention, including improved outcomes and the opportunity to train existing staff to conduct the intervention. They raised concerns about how to replicate the intervention in the context of current reimbursement strategies and limitations on their patients' literacy level and ability to travel to the clinic to participate in the program.
Raters said they found the worksheet helpful in thinking through potential implementation of the interventions, especially the prompts about modifiability and relevance to the practice's patients and priorities. Suggestions for changes included prompting the respondent to think through options for how to adapt the intervention, not just whether it could or could not be implemented “off the shelf,” and adding more detail to the worksheet items about relevance to the practice, including salience to clinicians and usability by patients (eg, in terms of literacy level). In observing the discussions, we noticed that the question about whether the intervention was described clearly did not seem necessary; this question conflated components of the intervention that other questions asked with the quality of the writing in the source article or evidence synopsis. Edits based on these suggestions and observations were incorporated into a revised version of the worksheet (available at researchtoolkit.org).
Discussion
The Decision-to-Implement Worksheet is a tool that provides a new resource for primary care practices seeking to assess whether evidence-based interventions are suitable to adopt or adapt to meet their needs or address a specific problem. Our approach to prescreening did not address all evidence or predict implementation within complex systems, nor was it designed to do so; other approaches or organizations may serve these purposes. However, the development of this worksheet contributes to the call by Greenhalgh et al17 for more research on the process through which innovations in health service organizations can be implemented and sustained (p. 620) and how innovations arising as good ideas can be reinvented and adapted to be perceived as more compatible for spread (p. 617).
In preliminary testing, we learned that 3 intervention characteristics correlated with a higher likelihood of implementation: simple to implement, testable before fully implementing, and modifiable to meet the needs of the practice. In addition, relevance to patients and fit with practice priorities, including their financial realities, were key aspects of discussion about how testers evaluated the likelihood of implementation. While we are aware of few comparable studies, 1 study of leaders of substance abuse disorder treatment organizations who had recently implemented interventions found that interventions consistent with the program's treatment philosophy and linked to client satisfaction and potential reputation benefits were most strongly endorsed.23
The low likelihood of implementation for 3 of 4 interventions that had been prescreened as designed for primary care settings is surprising. The identification of barriers to implementation, including fit with workflow, return on investment, clinician/leader buy-in, and effectiveness, reinforced the guidance by Damschroder et al16 that, “without adaptation, interventions usually come to a setting as a poor fit, resisted by individuals who will be affected by the intervention, and requiring an active process to engage individuals to accomplish implementation.” These findings suggest that spread of even highly relevant evidence into practice requires local tailoring, local implementation support, and attention to return on investment.
The Decision-to-Implement Worksheet was designed to be used after a prescreening process to narrow the evidence base, which typically would be beyond the resources of a primary care decision maker and thus would be conducted by a trusted body such as a professional organization or Agency for Healthcare Research and Quality evidence-based practice centers. Even though we were unfamiliar with the POEM process24 when we designed our method, our approach to prescreening mirrors it. POEM finds that about 1 in 40 studies (2.5%) that they review meets criteria for validity and relevance to clinical practice, compared with 6.3% of the articles we reviewed, likely reflecting different inclusion criteria. The use of a synopsis to highlight operational details that may not be clear in articles was helpful. The discussions during the WPRN meeting also reinforced previous recommendations25 that evidence summaries should include information on clinical effectiveness as well as cost-effectiveness and local applicability.
This methods development work was exploratory and raises opportunities for further enhancement. Identifying the most effective application of the worksheet is an important next step. The development of the worksheet used a convenience sample of interventions, but an ideal starting point for a practice might be to identify its own needed area of improvement as well as potentially relevant findings or innovations, then assessing each using this tool. Another next step is to learn more about how to implement the tool in busy practices and what practices think about when and why they would use the tool. One aspect to test further is what type of information and format gives practices the information they need to evaluate an innovation. In our study, meeting participants still turned to the full article as they completed the worksheet, so further testing of the relative value of a synopsis versus instructing authors to include additional operational detail in the article would be helpful to inform future publications and evidence translation programs.
A limitation of this work is that the methods used to identify the theoretical frameworks and compile existing instruments did not include a systematic review, so we may have omitted other relevant aspects of intervention assessment. However, the fact that the Decision-to-Implement Worksheet was pretested by clinicians and by colleagues with expertise in implementation research in CTSAs and practice-based research networks provides reassurance that the tool contributes to the field. Another limitation is that the correlation testing combined ratings of 4 different interventions; further testing with a larger number of participants would provide more information about correlations between ratings of specific intervention characteristics and the overall likelihood of implementation.
Conclusions
This work adds research and tools about how to evaluate evidence-based interventions that are ready for implementation in primary care. An important first step to efficiently and effectively improve health care delivery and patient health outcomes is to equip providers and practice administrators with a synopsis of ready-to-implement evidence-based findings. Some approaches and tools for this step are described at the beginning of this article. Here we address a critical subsequent step by providing an assessment method and tool that can help practices expeditiously assess the characteristics of the intervention and the resources needed to implement the innovation. The revised version of the Decision-to-Implement Worksheet, along with a sample evidence synopsis profile and literature prescreening protocol, is available at researchtoolkit.org in the “Primary Care Evidence Review Toolkit.” We encourage others to use, adapt, and report on the tool to build shared learning about best practices for fostering the uptake of evidence into community practice. For example, journal editors could further test evidence synopses as part of, or as companions to, articles to guide the application of findings in practice. Translational research networks including CTSAs and PBRNs could use such a tool to identify important network-wide implementation and dissemination projects based on network priorities. In addition to supporting decision making by primary care practices, the tool provides parameters that researchers can use to inform how they design and report research to make it relevant to the needs of primary care practices.
Acknowledgments
The authors are grateful to their colleagues who directed them to relevant resources and to the WWAMI region Practice and Research Network (WPRN) practice representatives who tested the form.
Notes
This article was externally peer reviewed.
Funding: This work was supported by the National Center For Advancing Translational Sciences of the National Institutes of Health under award UL1TR000423.
Conflict of interest: none declared.
Disclaimer: The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.
- Received for publication October 6, 2015.
- Revision received March 1, 2016.
- Accepted for publication March 21, 2016.
