Research ArticleOriginal Research

Clinical Decision Support Systems for Opioid Prescribing for Chronic Non-Cancer Pain in Primary Care: A Scoping Review

Sheryl Spithoff, Stephanie Mathieson, Frank Sullivan, Qi Guan, Abhimanyu Sud, Susan Hum and Mary Ann O'Brien
The Journal of the American Board of Family Medicine July 2020, 33 (4) 529-540; DOI: https://doi.org/10.3122/jabfm.2020.04.190199

Abstract

Background and Objectives: Clinical decision support systems (CDSSs) may help clinicians prescribe opioids for chronic noncancer pain (CNCP) more appropriately. This scoping review determined the extent and range of the current evidence on CDSSs for opioid prescribing for CNCP in primary care, and whether investigators followed best evidence and current guidance in designing, implementing and evaluating these complex interventions.

Methods: We searched 9 electronic databases and other data sources for studies from January 1, 2008 to October 11, 2019. Two reviewers independently screened the citations. One reviewer extracted data and a second verified for accuracy. Inclusion criteria: study of a CDSS for opioid prescribing for CNCP in a primary care clinical setting. We reported quantitative results in tables and qualitative results in narrative form.

Results: Our search yielded 5068 records, of which 14 studies met our inclusion criteria. All studies were conducted in the United States. Six studies examined local (eg, health center) CDSSs and 8 examined prescription drug monitoring program CDSSs. Three CDSSs incorporated evidence-based components. Study aims were heterogeneous and study designs included both quantitative and qualitative methodologies. No studies assessed patient health outcomes. Few studies appeared to be following guidance for evaluating complex interventions.

Conclusions: Few studies have rigorously assessed the use of CDSSs for opioid prescribing for CNCP in primary care settings. Going forward, investigators should include evidence-based components into the design of CDSSs and follow guidance for the development and evaluation of complex interventions.

Introduction

Two countries at the epicenter of the opioid crisis, Canada and the United States,1⇓⇓–4 recently released clinical practice guidelines for opioid prescribing for chronic noncancer pain (CNCP).5,6 These guidelines recommend against using opioid analgesics for CNCP as a first-line medication because the harms frequently outweigh the benefits.7⇓⇓–10 When opioids are prescribed for CNCP, the guidelines recommend risk mitigation strategies and opioid dose tapering. Both guidelines target primary care providers (PCPs), since they write about half of all opioid analgesic prescriptions in North America.11⇓–13 However, evidence shows that PCPs may have difficulty adopting recommended clinical practices.14⇓⇓⇓⇓⇓⇓–21 Clinical decision support may provide assistance.

Clinical decision support systems (CDSSs) are electronic systems that assist health care providers in clinical decision-making, by providing patient-specific data at the point-of-care.14⇓–16 Studies show that CDSSs lead to improvements in clinician performance (a care process measure), such as ordering appropriate tests and safer prescribing.17⇓⇓⇓⇓⇓⇓⇓–25 Some CDSS design components are evidence-based, including requiring a reason for an override, activating automatically (ie, the CDSS runs without requiring provider initiation), integrating into the electronic medical record (EMR), and providing advice to patients (eg, written materials), as well as clinicians.14,20,26⇓–28 These components lead to improvements in care process outcomes. Studies in which the CDSS evaluators are also the developers tend to show positive impact on process outcomes.26,27

However, the impact of CDSSs on important patient or population health outcomes is unclear,17⇓⇓–20 and widespread adoption is often limited by implementation issues.29⇓⇓⇓⇓–34 In addition, CDSSs can be difficult to develop and evaluate because they are complex interventions that seek to change the functioning of a complex adaptive system, such as a primary care clinic.35 The Medical Research Council in the United Kingdom (UK) recommends that researchers design and evaluate complex interventions through a carefully staged series of studies targeting key uncertainties, as well as a definitive evaluation.35,36 All steps should include process evaluations and assess for unintended consequences.37

CDSSs can have a variety of roles in improving adherence to opioid prescribing guidelines for CNCP. They can be used to reduce the number of new opioid prescriptions for acute pain38 and to reduce the initiation of opioid prescribing for CNCP. They can also be used to improve prescribing and other measures, like risk mitigation strategies for patients already receiving opioids for CNCP. This is the most challenging role for a CDSS as these patients are at high risk of harms and changing prescribing is very difficult.39,40

Several studies have evaluated CDSSs for opioid prescribing for CNCP in primary care settings.41⇓⇓–44 These studies report that the use of a CDSS led to a reduction in opioid prescribing or improved adherence to clinical practice guidelines.41⇓⇓–44 Several studies have also evaluated prescription drug monitoring program (PDMP) CDSSs for opioid prescribing for CNCP in primary care settings. PDMP CDSSs are large, centralized, government-run databases that prescribers can access for point-of-care information on a patient's opioid prescriptions.45,46 While 1 PDMP CDSS study found that physicians wrote fewer opioid prescription in 61% of cases,47 another study reported no association between PDMP implementation status and requirement levels (from no requirements to a mandatory requirement to check the PDMP before prescribing) and physicians' opioid prescribing for CNCP.48 Four other PDMP CDSS studies examined PCPs' use of, and views on PDMPs.49⇓⇓–52 To date, however, the literature in this emerging field has not been systematically summarized and analyzed so the benefits and risks of implementing a CDSS are unclear.

This scoping review determined the extent and range of the current evidence on CDSSs for opioid prescribing for CNCP in primary care. Our secondary aim was to determine whether researchers followed best evidence for the design of the CDSSs and current guidance for the evaluation of complex interventions.

Methods

We conducted a scoping review using the frameworks53,54 described by Colquhoun et al,55 and the methods outlined by The Joanna Briggs Institute.56 We followed the reporting guidelines from the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) Extension for Scoping reviews.57 We created an a priori protocol and used an iterative approach. Modifications included a secondary research aim and a change to the data extraction plan.

Study Eligibility

We included peer-reviewed and nonpeer-reviewed studies that used quantitative, qualitative, and mixed-methods methodologies. We excluded nonsystematic reviews, letters, opinion articles, analysis articles, clinical practice guidelines, and policy documents. We included all studies where the population was PCPs (ie, family physicians, emergency medicine physicians, nurse practitioners, and primary care internists) working in a primary care setting. Studies that reported less than 50% PCPs or did not report the percentage of PCPs were excluded, unless results were reported by subgroup. We included all studies that assessed a CDSS that sought to improve opioid prescribing for CNCP patients in a primary care clinical setting. We excluded studies where PCPs were working in a secondary and tertiary settings, such as a pain clinic or addiction clinic. We excluded primary care pediatric clinics. We defined a CDSS as an electronic system that assisted health care providers in clinical decision-making, by providing patient-specific data at the point-of-care.14⇓–16 We included studies where the CDSS was integrated into the EMR, or functioned independently (eg, Web-accessed), or was embedded within a larger intervention. We excluded studies where CDSS use was not specified, where it was used for another reason, or where it was not implemented in clinical settings.

Data Sources and Searches

We searched electronic databases (MEDLINE [via OVID], EMBASE, CINAHL, CENTRAL, PsycINFO and International Pharmaceutical Abstracts [via OVIDSP]) from January 1, 2008 to October 11, 2019. CDSSs developed before this period likely evolved or became obsolete.59 We built a comprehensive search strategy, including the terms “opioid,” and “clinical decision support systems.” Since studies used a large number of different keywords and medical subject headings for a CDSS, we had to conduct a broad search using a large variety of terms, including computer systems, health informatics, clinical decision-making (Appendix 1 Medline search strategy). The Medline strategy (Appendix 1) was adapted for the other databases. We used the Canadian Agency for Drugs and Technologies approach to our gray literature search (Appendix 2 Gray literature search).60 We also searched trial registries (ClinicalTrials.gov, World Health Organization International Clinical Trials Registry Platform), checked reference lists of additional eligible studies and contacted experts (ie, lead authors on included studies, registered protocols and systematic reviews of CDSSs).

Screening and Selection

Two researchers independently screened abstracts to determine whether they met inclusion criteria. Two researchers then independently screened the full text of all relevant articles. For both steps, after we screened 10 to 15 titles and articles, we checked inter-reviewer agreement to ensure it was at least 80% before continuing further. When there were disagreements, a third researcher (MAO) assisted in making the final decision. We contacted authors for more information when full text was not available online.58

Data Extraction

We created and pilot tested a data extraction form to record the following items: study population and setting, description of the intervention and implementation process, type of CDSS, inclusion of evidence-based CDSS components (components that the literature has consistently found to have an impact on outcomes: requiring a reason for an override; activating automatically; integrating into the EMR; and providing advice to patients and clinicians),14,20,26⇓–28 study aims, methodology and design, study outcomes, funding information, conflicts of interest, and adherence to guidance for complex interventions (eg, study was part of a stepped approach to development and evaluation), assessment for unintended consequences, planned process evaluation, process and outcome measures, theoretical approach to guide implementation and/or evaluation. One reviewer extracted data and another researcher reviewed their work (SMS, MAO, QG, SM, SH). This was a modification from our protocol that specified that 2 researchers would independently extract the data.

Data Synthesis

We used a flow diagram to report on study selection. We reported quantitative data in tabular format. We wrote narrative summaries using contextual and process-oriented data. We did not conduct a detailed assessment of study quality, assess for reporting bias, or risk of bias consistent with current guidance on conducting scoping reviews.55⇓–57

Results

Our literature search identified 5068 citations from which 14 were included in the scoping review (Figure 1). Six studies examined local CDSSs (eg, specific health system, center or clinic)41,43,44,61⇓–63 while 8 examined state-run, Web-based, central PDMP CDSSs47,49⇓⇓–52,64⇓–66 Results using these 2 typologies are summarized in Table 1. Study descriptions are detailed in Appendix 3.

Figure 1.
Figure 1.

PRISMA Flow Diagram.

View this table:
Table 1.

Study Setting, Participants, Clinical Decision Support System (CDSS) Type, and Inclusion of Evidence-Based Components

CDSS Description

Types of CDSSs included protocols (ie, forms that guide clinical management) in the EMR, Intranet dashboards, EMR alerts, data repositories and Web-based clinical tools. Four local CDSSs were integrated into the EMR43,44,62 and 2 automatically activated.44,62 The other 2 required the PCP to activate the CDSS. Studies assessing PDMP CDSSs did not report any evidence-based design components.

Study Characteristics

All studies occurred in the United States and practice settings were mostly primary care clinics. Three were set in the emergency department.44,47,49 All the local CDSSs, and 3 of the PDMP CDSS studies47,64,66 were designed to assess whether a CDSS alone or incorporated into a multi-faceted intervention improved prescribing or adherence to guidelines. The remaining PDMP CDSS studies determined providers' behavior, knowledge of, attitudes toward and use of CDSSs. Local CDSS study designs included 4 prepost interventions, a cluster randomized controlled trial (RCT) and a mixed-methods evaluation. The 8 PDMP CDSS studies included a wide variety of study designs including: 3 prepost interventions, a cross-sectional survey, 2 qualitative, 1 mixed-methods, and 1 retrospective cohort. Study aims and designs are summarized in Table 2 and described in detail in Appendix 3. One study was part of a stepped approach in evaluating a complex intervention.63 About half of the studies that assessed the impact of an intervention included a process evaluation (measures assessing if program components had been implemented as intended).41,43,47,49,62⇓–64 Two studies reported using a theoretical approach in implementation and evaluation processes.61,63

View this table:
Table 2.

Aims and Designs of Included Studies

Implementation Processes

All the studies on local CDSSs described their implementation process, but provided little detail. None of the PDMP CDSS studies described implementation processes.

Study Findings

Local CDSSs

Anderson et al41 found that the CDSS and summary reports improved compliance with guidelines; Canada et al43 reported that a CDSS plus monetary incentives improved adherence to guidelines; Downes et al62 found that a CDSS and electronic reports reduced opioid prescribing and increased urine drug testing and use of pain contracts; Gugelmann et al44 found that the CDSS reduced opioid prescribing; Liebschutz et al61 reported that a multi-faceted intervention that included a CDSS in both study arms also reduced opioid prescribing; and Seal et al63 found in a multi-component intervention (with CDSS in both arms) that providers “abandoned use” of the CDSS.

PDMP CDSSs

Baehren et al47 found that physicians who used PDMP data wrote fewer opioid prescriptions in 61% of cases and more opioid prescriptions in 39% of cases; Binswanger et al64 found that a multi-component intervention improved adherence to guidelines; Chaudhary et al52 found that most PCPs reported always checking the PDMP before prescribing opioids to new patients. Click et al50 found that providers have positive views about PDMPs, but reported barriers in using them. Coleman51 found that in 5 of 7 records of patient prescribed opioids, providers accessed the PDMP. Freeman et al65 reported that PDMPs are key tools for PCPs and that barriers included a lack of integration; Kohlbeck et al49 reported that an educational intervention increased providers' knowledge of, behavior, and attitudes toward PDMP CDSSs; Patchett et al66 reported that a multi-component intervention increased use of a PDMP and led to a reduction in opioid prescribing.

Funding and Conflict of Interest

All but 2 local CDSS studies reported on funding for CDSS evaluation;44,62 and 3 others were missing information on funding for CDSS development.44,63 All PDMP studies except 166 provided information on funding for evaluation, but none provided information on funding for development. For all 6 local CDSS studies, the developers were also the evaluators or the relationship was unclear or not stated. No evaluators of PDMPs provided information on their relationship to the PDMP developer (Table 3).

View this table:
Table 3.

Funding and Relationship between Developers and Evaluators

Discussion

We identified 14 studies published between 2009 and 2019 that examined CDSSs for opioid prescribing for CNCP in primary care clinical settings. Six of the studies examined local CDSSs (that were used locally within a specific health center, health system or clinic) and 8 examined PDMP CDSSs. Studies evaluating CDSS impact found that the CDSS (alone or more commonly, part of a dual or multi-component intervention) led to more appropriate prescribing practices and/or adherence to guidelines. Several PDMP CDSS studies assessed providers' views on, and/or their use of PDMP CDSSs. These studies reported frequent use of the PDMP CDSS and positive views toward the CDSS with some acknowledgment of the barriers and limitations. These findings are similar to a recent qualitative rapid review that asked providers about the use of PDMPs.67 No study in our review, however, contained an assessment of patient health outcomes or assessed for unintended consequences. In addition, in 4 studies the evaluators were also the CDSS developers, a potentially useful situation, but one that presents a potential conflict of interest,26,27 that was not addressed by the investigators. We also found that few CDSSs included evidence-based components and that in only 1 study investigators reported following current guidance for development and evaluation of complex interventions.35,36

Our finding that there were only 14 studies, and only 1 RCT, which met our inclusion criteria is surprising. In contrast, a 2015 systematic review found 7 RCT studies of CDSSs for antibiotic prescribing by primary care providers.28 There may be several contributing factors. The prescription opioid crisis only gained widespread attention in the last decade,68 and it takes time to develop a complex intervention like a CDSS.36 It is also possible that some CDSSs failed to show promise early on and development was subsequently stalled or halted. Accordingly, there are a number of reports on the development of a CDSSs for opioid prescribing for CNCP where clinical outcomes have not been reported yet.69⇓⇓–72 And finally, it is possible that CDSSs are being used without an evaluation plan, as has occurred with many PDMP CDSSs.73 This may be because of a demand for immediate solutions to the opioid crisis and an evaluation of a CDSS takes significant time and money. However, since CDSSs frequently do not improve patient outcomes,17⇓⇓–20 and may lead to unintended consequences, a comprehensive evaluation is essential.74

Most studies in our review that assessed the impact of the CDSS reported an improvement in prescribing or better adherence to clinical practice guidelines. This aligns with previous research in other fields: CDSSs have a modest impact on clinician performance (a care process outcome).17⇓⇓⇓⇓⇓⇓⇓–25 However, these results need careful interpretation. Most studies were prepost, nonrandomized control or observational designs. Although—consistent with guidance for scoping reviews55,56—we did not conduct a quality assessment; these types of study designs have greater threats to validity.75 In addition, in most of the studies, the CDSS was part of a larger intervention, so its specific impact was unclear. Another reason for caution is that no studies assessed patient health outcomes, such as quality of life, morbidity, and mortality.76⇓–78 Reductions in opioid prescribing and better adherence to guidelines may have unintended consequences.36 For example, studies report that patients often turn to illicit sources of opioids when they have reduced access to prescribed opioids, increasing their risk of overdose.79⇓⇓⇓⇓–84 Several studies in a systematic review found that heroin overdoses increased after a PDMP CDSS was implemented.74 A more recent systematic review, however, found no consistent association between population-level opioid-related harms (including heroin use and overdoses) and PDMP CDSSs.85 We also noted a conflict of interest in some studies where the developers were also the evaluators. Systematic reviews in other fields have demonstrated that when the CDSS evaluator is also the developer, outcomes are better.26,27 It is possible that developers achieve better outcomes because they design effective implementation plans,26 but it is possible that the conflict of interest leads to conscious or unconscious bias.26,86⇓⇓⇓⇓⇓–92 Interestingly, none of the studies reported funding from or involvement of for-profit entities. It is possible that CDSSs developed by for-profit entities are not undergoing a publicly reported evaluation. This is problematic, and as a recent criminal case demonstrated, can lead to potential harm to patients.93

We found that few of the CDSSs incorporated evidence-based design components. In only 1 study did researchers follow guidance for designing and evaluating complex interventions. Developers may not have incorporated evidence-based components because of the lag time between development and evaluation: when the CDSS was created the developers may not have had access to systematic reviews on the various components. The developer may also feel that the evidence does not apply to this particular subspecialty or setting.94 Another reason may be a general excitement and overconfidence in e-health technologies.95 Funders and developers may be too eager to solve the problem of unsafe opioid prescribing using eHealth technologies and are not ensuring that developers are building on information from the medical literature.95 Changes are occurring. Between 2012 and 2016, the Substance Abuse and Mental Health Services Administration funded 9 projects to integrate PDMP data into EMRs.96 Investigators might not follow guidance for complex evaluations because it is a lengthy and expensive iterative process before a definitive evaluation.35⇓–37,97 This is a widespread issue—few complex interventions seem to undergo modeling, pilot and feasibility testing,98 and many lack process evaluations.99,100 This is problematic. If researchers conduct a trial without testing components, possible causal pathways, uncertainties, contextual factors, and implementation approaches, they risk wasting resources on an expensive trial and perhaps causing harm.35,37,101 Conversely, if the evaluation takes too long, the technology could become obsolete before it gains widespread uptake.59 Adopting rapid, concurrent and iterative pilot and feasibility studies may be the best approach.102⇓–104

Limitations

There are 2 main limitations in our review. In the gray literature search we may have missed non-English-language studies, as we conducted the searches only in English. Second, several of the studies included both PCPs and other provider types (we excluded those with less than 50% PCPs), and, as these studies only reported aggregate outcomes, they may not accurately reflect the PCP population.

Conclusion and Next Steps

Our review reveals that few studies have rigorously assessed the use of CDSSs in the context of opioid prescribing for CNCP in the primary care setting. More high-quality studies are needed. Going forward, investigators should include evidence-based components into the design of CDSSs and follow guidance for the development and evaluation of complex interventions, including pilot studies, process evaluations, and an assessment for unintended consequences.

Acknowledgments

The authors would like to thank Kaitlin Fuller, Education & Liaison Librarian for the MD Program and the Institute of Medical Science; Gerstein Science Information Centre, University of Toronto.

Appendix

View this table:
Appendix 1:

Medline Search Strategy

View this table:
Appendix 2:

Gray Literature Search

View this table:

Databases Gray Literature

View this table:

Search Engine Searching

Notes

  • This article was externally peer reviewed.

  • Disclosures: None of the authors have any competing interests to declare.

  • Funding: This project did not receive specific funding. SS was supported by a graduate research award from the University of Toronto Department of Family and Community Medicine; SM received fellowship support in 2018 from a National Health and Medical Research Council (NHMRC) program Grant (#AP1113532) and 2019 NHMRC Early Career Fellowship (APP1158463); FS was supported by NYGH (North York General Hospital) as the Gordon F. Cheesbrough Research Chair in Family and Community Medicine; AS was supported by a graduate research award from the University of Toronto, Department of Family and Community Medicine and a research fellowship from the Medical Psychiatry Alliance, University of Toronto; QG was supported by a graduate research award from the Institute of Health Policy, Management and Evaluation, University of Toronto.

  • To see this article online, please go to: http://jabfm.org/content/33/4/529.full.

  • Received for publication May 30, 2019.
  • Revision received February 20, 2020.
  • Accepted for publication February 23, 2020.

References

  1. 1.↵
    1. Fischer B,
    2. Keates A,
    3. Bühringer G,
    4. Reimer J,
    5. Rehm J
    . Non-medical use of prescription opioids and prescription opioid-related harms: why so markedly higher in North America compared to the rest of the world?. Addict Abingdon Engl 2014;109:177–81.
    OpenUrl
  2. 2.↵
    Report of the International Narcotics Control Board for 2017 [Internet]. International Narcotics Control Board. 2017. Available from: https://www.incb.org/incb/en/publications/annual-reports/annual-report-2017.html.
  3. 3.↵
    1. Seth P
    . Overdose Deaths Involving Opioids, Cocaine, and Psychostimulants—United States, 2015–2016. MMWR Morb Mortal Wkly Rep [Internet]. 2018 [cited 2018 Apr 23];67. Available from: https://www.cdc.gov/mmwr/volumes/67/wr/mm6712a1.htm.
  4. 4.↵
    Government of Canada. National Report: Apparent opioid-related deaths in Canada [Internet]. 2018 [cited 2018 Apr 23]. Available from: www.canada.ca/en/public-health/services/publications/healthy-living/national-report-apparent-opioid-related-deaths-released-march-2018.html.
  5. 5.↵
    1. Dowell D,
    2. Haegerich TM,
    3. Chou R
    . CDC Guideline for prescribing opioids for chronic pain–United States, 2016. JAMA 2016;315:1624–45.
    OpenUrlCrossRefPubMed
  6. 6.↵
    1. Busse J
    . The 2017 Canadian guideline for opioids for chronic non-cancer pain [Internet]. National pain center. 2017. Available from: http://www.cmaj.ca/content/suppl/2017/05/03/189.18.E659.DC1/170363-guide-1-at-updated.pdf.
  7. 7.↵
    1. Dunn KM,
    2. Saunders KW,
    3. Rutter CM,
    4. et al
    . Overdose and prescribed opioids: Associations among chronic non-cancer pain patients. Ann Intern Med 2010;152:85–92.
    OpenUrlCrossRefPubMed
  8. 8.↵
    1. Bohnert AB,
    2. Valenstein M,
    3. Bair MJ,
    4. et al
    . Association between opioid prescribing patterns and opioid overdose-related deaths. JAMA 2011;305:1315–21.
    OpenUrlCrossRefPubMed
  9. 9.↵
    1. Gomes T,
    2. Mamdani MM,
    3. Dhalla IA,
    4. Paterson JM,
    5. Juurlink DN
    . Opioid dose and drug-related mortality in patients with nonmalignant pain. Arch Intern Med 2011;171:686–91.
    OpenUrlCrossRefPubMed
  10. 10.↵
    1. Chou R,
    2. Turner JA,
    3. Devine EB,
    4. et al
    . The effectiveness and risks of long-term opioid therapy for chronic pain: A systematic review for a National Institutes of Health Pathways to Prevention Workshop. Ann Intern Med 2015;162:276–86.
    OpenUrlCrossRefPubMed
  11. 11.↵
    1. Volkow ND,
    2. McLellan TA,
    3. Cotto JH,
    4. Karithanom M,
    5. Weiss S
    . Characteristics of opioid prescriptions in 2009. JAMA 2011;305:1299–301.
    OpenUrlCrossRefPubMed
  12. 12.↵
    1. Levy B,
    2. Paulozzi L,
    3. Mack KA,
    4. Jones CM
    . Trends in Opioid Analgesic-Prescribing Rates by Specialty, U.S., 2007–2012. Am J Prev Med 2015;49:409–13. Sep.
    OpenUrlCrossRefPubMed
  13. 13.↵
    Health Quality Ontario. Starting on Opioids in Ontario [Internet]. 2017 [cited 2018 Jul 12]. Available from: http://startingonopioids.hqontario.ca/.
  14. 14.↵
    1. Kawamoto K,
    2. Houlihan CA,
    3. Balas EA,
    4. Lobach DF
    . Improving clinical practice using clinical decision support systems: A systematic review of trials to identify features critical to success. BMJ 2005;330:765.
    OpenUrlAbstract/FREE Full Text
  15. 15.↵
    1. Sim I,
    2. Gorman P,
    3. Greenes RA,
    4. et al
    . Clinical Decision support systems for the practice of evidence-based medicine. J Am Med Inform Assoc JAMIA 2001;8:527–34.
    OpenUrl
  16. 16.↵
    1. Haynes RB,
    2. Wilczynski NL
    . Effects of computerized clinical decision support systems on practitioner performance and patient outcomes: Methods of a decision-maker-researcher partnership systematic review. Implement Sci IS 2010;5;5:12.
    OpenUrl
  17. 17.↵
    1. Jia P,
    2. Zhang L,
    3. Chen J,
    4. Zhao P,
    5. Zhang M
    . The Effects of Clinical Decision Support Systems on Medication Safety: An Overview. PLoS One 2016;11:e0167683.
    OpenUrl
  18. 18.↵
    1. Lloyd R,
    2. Ramakrishna A,
    3. Mackay JA,
    4. et al
    . Computerized clinical decision support systems for acute care management: A decision-maker-researcher partnership systematic review of effects on process of care and patient outcomes. Implement Sci IS 2011;6:91.
    OpenUrl
  19. 19.↵
    1. Misra S,
    2. Gerstein HC,
    3. Garg AX,
    4. et al
    . Computerized clinical decision support systems for chronic disease management: A decision-maker-researcher partnership systematic review. Implement Sci IS 2011;6:92.
    OpenUrl
  20. 20.↵
    1. Moja L,
    2. Kwag KH,
    3. Lytras T,
    4. et al
    . Effectiveness of Computerized decision support systems linked to electronic health records: A systematic review and meta-analysis. Am J Public Health 2014;104:e12–22.
    OpenUrl
  21. 21.↵
    1. Jaspers MWM,
    2. Smeulers M,
    3. Vermeulen H,
    4. Peute LW
    . Effects of clinical decision-support systems on practitioner performance and patient outcomes: A synthesis of high-quality systematic review findings. J Am Med Inform Assoc 2011;18:327–34.
    OpenUrlCrossRefPubMed
  22. 22.↵
    1. Nieuwlaat R,
    2. Connolly SJ,
    3. Mackay JA,
    4. et al
    . Computerized clinical decision support systems for therapeutic drug monitoring and dosing: A decision-maker-researcher partnership systematic review. Implement Sci IS 2011;6:90.
    OpenUrl
  23. 23.↵
    1. Sebaldt RJ,
    2. Mackay JA,
    3. Prorok JC,
    4. et al
    . Computerized clinical decision support systems for primary preventive care: A decision-maker-researcher partnership systematic review of effects on process of care and patient outcomes. Implement Sci IS 2011;6:87.
    OpenUrl
  24. 24.↵
    1. Hemens BJ,
    2. Holbrook A,
    3. Tonkin M,
    4. et al
    . Computerized clinical decision support systems for drug prescribing and management: A decision-maker-researcher partnership systematic review. Implement Sci IS 2011;6:89.
    OpenUrl
  25. 25.↵
    1. Bright TJ,
    2. Wong A,
    3. Dhurjati R,
    4. et al
    . Effect of clinical decision-support systems: A systematic review. Ann Intern Med 2012;157:29–43.
    OpenUrlCrossRefPubMed
  26. 26.↵
    1. Roshanov PS,
    2. Fernandes N,
    3. Wilczynski JM,
    4. et al
    . Features of effective computerised clinical decision support systems: Meta-regression of 162 randomised trials. BMJ 2013;346:f657–f657.
    OpenUrlAbstract/FREE Full Text
  27. 27.↵
    1. Garg AX,
    2. Adhikari NKJ,
    3. McDonald H,
    4. et al
    . Effects of computerized clinical decision support systems on practitioner performance and patient outcomes: A systematic review. JAMA 2005;293:1223–38.
    OpenUrlCrossRefPubMed
  28. 28.↵
    1. Holstiege J,
    2. Mathes T,
    3. Pieper D
    . Effects of computer-aided clinical decision support systems in improving antibiotic prescribing by primary care providers: A systematic review. J Am Med Inform Assoc 2015;22:236–42.
    OpenUrlCrossRefPubMed
  29. 29.↵
    1. Stultz JS,
    2. Nahata MC
    . Computerized clinical decision support for medication prescribing and utilization in pediatrics. J Am Med Inform Assoc 2012;19:942–53.
    OpenUrlCrossRefPubMed
  30. 30.↵
    1. Moxey A,
    2. Robertson J,
    3. Newby D,
    4. Hains I,
    5. Williamson M,
    6. Pearson SA
    . Computerized clinical decision support for prescribing: Provision does not guarantee uptake. J Am Med Inform Assoc 2010;17:25–33.
    OpenUrlCrossRefPubMed
  31. 31.↵
    1. Patterson ES,
    2. Doebbeling BN,
    3. Fung CH,
    4. Militello L,
    5. Anders S,
    6. Asch SM
    . Identifying barriers to the effective use of clinical reminders: Bootstrapping multiple methods. J Biomed Inform 2005;38:189–99.
    OpenUrlCrossRefPubMed
  32. 32.↵
    1. Arts DL,
    2. Medlock SK,
    3. van Weert H,
    4. Wyatt JC,
    5. Abu-Hanna A
    . Acceptance and barriers pertaining to a general practice decision support system for multiple clinical conditions: A mixed methods evaluation. PLoS ONE 2018;13:e0193187. [Internet]. 2018 Apr [cited 2018 19] Available from: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5908177/.
    OpenUrl
  33. 33.↵
    1. Dalleur O,
    2. Seger DL,
    3. Slight SP,
    4. et al
    . Inappropriate overrides of age-related alerts in prescriber order entry. J Gen Intern Med. 2015;S189–S190.
  34. 34.↵
    1. Slight SP,
    2. Beeler PE,
    3. Seger DL,
    4. et al
    . An evaluation of drug-allergy interaction alert overrides in inpatients. J Gen Intern Med 2015;S81.
  35. 35.↵
    1. Craig P,
    2. Dieppe P,
    3. Macintyre S,
    4. Michie S,
    5. Nazareth I,
    6. Petticrew M
    . Developing and evaluating complex interventions: the new Medical Research Council guidance. BMJ. 2008;337: a1655.
    OpenUrlFREE Full Text
  36. 36.↵
    1. Craig P,
    2. Dieppe P,
    3. Macintyre S,
    4. Michie S,
    5. Nazareth I,
    6. Petticrew M
    . Developing and evaluating complex interventions: The new Medical Research Council guidance. Int J Nurs Stud 2013;50:587–92.
    OpenUrlCrossRefPubMed
  37. 37.↵
    1. Catwell L,
    2. Sheikh A
    . Evaluating eHealth interventions: The need for continuous systemic evaluation. PLoS Med 2009;6:e1000126. [Internet] Aug 18 [cited 2018 Sep 24] Available from: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2719100/
    OpenUrlCrossRefPubMed
  38. 38.↵
    1. Shah A
    . Characteristics of Initial prescription episodes and likelihood of long-term opioid use—United States, 2006–2015. MMWR Morb Mortal Wkly Rep [Internet]. 2017 [cited 2020 Feb 14];66. Available from: https://www.facebook.com/CDCMMWR.
  39. 39.↵
    1. Dasgupta N,
    2. Beletsky L,
    3. Ciccarone D
    . Opioid crisis: No Easy fix to its social and economic determinants. Am J Public Health 2018;108:182–6.
    OpenUrlPubMed
  40. 40.↵
    1. Brooks EA,
    2. Unruh A,
    3. Lynch ME
    . Exploring the lived experience of adults using prescription opioids to manage chronic noncancer pain. Pain Res Manag J Can Pain Soc 2015;20:15–22.
    OpenUrl
  41. 41.↵
    1. Anderson D,
    2. Zlateva I,
    3. Khatri K,
    4. Ciaburri N
    . Using health information technology to improve adherence to opioid prescribing guidelines in primary care. Clin J Pain 2015;31:573–9.
    OpenUrl
  42. 42.↵
    1. Patel S,
    2. Carmichael JM,
    3. Taylor JM,
    4. Bounthavong M,
    5. Higgins DT
    . Evaluating the Impact of a clinical decision support tool to reduce chronic opioid dose and decrease risk classification in a veteran population. Ann Pharmacother 2018;52:325–31.
    OpenUrl
  43. 43.↵
    1. Canada RE,
    2. DiRocco D,
    3. Day S
    . A better approach to opioid prescribing in primary care. J Fam Pract 2014;63:E1–E8.
    OpenUrl
  44. 44.↵
    1. Gugelmann H,
    2. Shofer FS,
    3. Meisel ZF,
    4. Perrone J
    . Multidisciplinary intervention decreases the use of opioid medication discharge packs from 2 urban EDs. Am J Emerg Med 2013;31:1343–8.
    OpenUrl
  45. 45.↵
    1. Sproule BA
    . Prescription Monitoring programs in canada: best practice and program review [Internet]. Canadian Centre on Substance Abuse, Ottawa, ON 2015;[cited 2018 Jun 14]. Available from: http://www.ccsa.ca/Resource%20Library/CCSA-Prescription-Monitoring-Programs-in-Canada-Report-2015-en.pdf.
  46. 46.↵
    1. Rutkow L,
    2. Smith KC,
    3. Lai AY,
    4. Vernick JS,
    5. Davis CS,
    6. Alexander GC
    . Prescription drug monitoring program design and function: A qualitative analysis. Drug Alcohol Depend 2017;180:395–400.
    OpenUrl
  47. 47.↵
    1. Baehren DF,
    2. Marco CA,
    3. Droz DE,
    4. Sinha S,
    5. Callan EM,
    6. Akpunonu P
    . A statewide prescription monitoring program affects emergency department prescribing behaviors. Ann Emerg Med 2010;56:19–23.e1-3.
    OpenUrlCrossRefPubMed
  48. 48.↵
    1. Lin HC,
    2. Wang Z,
    3. Boyd C,
    4. Simoni-Wastila L,
    5. Buu A
    . Associations between statewide prescription drug monitoring program (PDMP) requirement and physician patterns of prescribing opioid analgesics for patients with non-cancer chronic pain. Addict Behav 2018;76:348–54.
    OpenUrl
  49. 49.↵
    1. Kohlbeck S,
    2. Akert B,
    3. Pace C,
    4. Zosel A
    . A multistep approach to address clinician knowledge, attitudes, and behavior around opioid prescribing. Wis Med J 2018;117:38–41.
    OpenUrl
  50. 50.↵
    1. Click IA,
    2. Basden JA,
    3. Bohannon JM,
    4. Anderson H,
    5. Tudiver F
    . Opioid Prescribing in rural family practices: A qualitative study. Subst Use Misuse 2018;53:533–40.
    OpenUrl
  51. 51.↵
    1. Coleman CD
    . Evidence-Based strategies to minimize risk for opioid pain medication misuse among patients with chronic pain in a primary care setting. Diss Abstr Int Sect B Sci Eng 2016;64:
  52. 52.↵
    1. Chaudhary S,
    2. Compton P
    . Use of risk mitigation practices by family nurse practitioners prescribing opioids for the management of chronic nonmalignant pain. Subst Abuse 2017;38:95–104.
    OpenUrl
  53. 53.↵
    1. Arksey H,
    2. O'Malley L
    . Scoping studies: towards a methodological framework. Int J Soc Res Methodol 2005;8:19–32.
    OpenUrlCrossRef
  54. 54.↵
    1. Levac D,
    2. Colquhoun H,
    3. O'Brien KK
    . Scoping studies: Advancing the methodology. Implement Sci 2010;5:69.
    OpenUrlCrossRefPubMed
  55. 55.↵
    1. Colquhoun HL,
    2. Levac D,
    3. O'Brien KK,
    4. et al
    . Scoping reviews: time for clarity in definition, methods, and reporting. J Clin Epidemiol 2014;67:1291–4.
    OpenUrlCrossRefPubMed
  56. 56.↵
    1. Peters M,
    2. Godfrey C,
    3. McInerney P,
    4. Soares C,
    5. Khalil H,
    6. Parker D
    . The Joanna Briggs Institute Reviewers' Manual 2015: Methodology for JBI scoping reviews. 2015 Jan 1 [cited 2018 Feb 7]; Available from: https://espace.library.uq.edu.au/view/UQ:371443.
  57. 57.↵
    1. Tricco AC,
    2. Lillie E,
    3. Zarin W,
    4. et al
    . PRISMA extension for scoping reviews (PRISMA-ScR): Checklist and explanation. Ann Intern Med 2018;169:467. [Internet] Sep 4 [cited 2018 Sep 21]; Available from: http://annals.org/article.aspx?doi=10.7326/M18-0850.
    OpenUrlCrossRefPubMed
  58. 58.↵
    1. Li G,
    2. Abbade LPF,
    3. Nwosu I,
    4. et al
    . A scoping review of comparisons between abstracts and full reports in primary biomedical research. BMC Med Res Methodol 2017;17:181.
    OpenUrl
  59. 59.↵
    1. Main C,
    2. Moxham T,
    3. Wyatt JC,
    4. Kay J,
    5. Anderson R,
    6. Stein K
    . Computerised decision support systems in order communication for diagnostic, screening or monitoring test ordering: systematic reviews of the effects and cost-effectiveness of systems [Internet]. NIHR J Library 2010; [cited 2018 Jan 7]. Available from: https://www.ncbi.nlm.nih.gov/books/NBK56829/.
  60. 60.↵
    Grey matters: A practical tool for searching health-related grey literature | CADTH.ca [Internet]. [cited 2018 Sep 24]. Available from: https://www.cadth.ca/resources/finding-evidence/grey-matters.
  61. 61.↵
    1. Liebschutz JM,
    2. Xuan Z,
    3. Shanahan CW,
    4. et al
    . Improving adherence to long-term opioid therapy guidelines to reduce opioid misuse in primary care: A cluster-randomized clinical trial. JAMA Intern Med 2017;177:1265–72.
    OpenUrl
  62. 62.↵
    1. Downes JM,
    2. Klepser DG,
    3. Foster J,
    4. Nelson M
    . Development of a standardized approach for managing opioids in adults with chronic noncancer pain. Am J Health-Syst Pharm 2018;75:321–6.
    OpenUrlAbstract/FREE Full Text
  63. 63.↵
    1. Seal KH,
    2. Borsari B,
    3. Tighe J,
    4. et al
    . Optimizing pain treatment interventions (OPTI): A pilot randomized controlled trial of collaborative care to improve chronic pain management and opioid safety-rationale, methods, and lessons learned. Contemp Clin Trials 2019;77:76–85.
    OpenUrl
  64. 64.↵
    1. Binswanger IA,
    2. Joseph N,
    3. Hanratty R,
    4. et al
    . Novel opioid safety clinic initiative to deliver guideline-concordant chronic opioid therapy in primary care. Mayo Clin Proc Innov Qual Outcomes 2018;2:309–16.
    OpenUrl
  65. 65.↵
    1. Freeman PR,
    2. Curran GM,
    3. Drummond KL,
    4. et al
    . Utilization of prescription drug monitoring programs for prescribing and dispensing decisions: Results from a multi-site qualitative study. Res Soc Adm Pharm RSAP 2019;15:754–60.
    OpenUrl
  66. 66.↵
    1. Patchett D,
    2. Grover M,
    3. Kresin M,
    4. et al
    . The benefits of a standardized approach to opioid prescribing. J Fam Pract 2019;68:E1–E7.
    OpenUrl
  67. 67.↵
    Prescribing and Dispensing policies to address harms associated with prescription drug abuse | CADTH.ca [Internet]. [cited 2018 Aug 20]. Available from: https://www.cadth.ca/prescribing-and-dispensing-policies-address-harms-associated-prescription-drug-abuse.
  68. 68.↵
    1. McCarthy M
    . Containing the opioid overdose epidemic. BMJ 2012;345:e8340–e8340.
    OpenUrlFREE Full Text
  69. 69.↵
    1. Trafton J,
    2. Martins S,
    3. Michel M,
    4. et al
    . Evaluation of the acceptability and usability of a decision support system to encourage safe and effective use of opioid therapy for chronic, noncancer pain by primary care providers. Pain Med 2010;11:575–85.
    OpenUrl
  70. 70.↵
    1. Furlan AD,
    2. Reardon R,
    3. Salach L
    . The opioid manager: a point-of-care tool to facilitate the use of the Canadian Opioid Guideline. J Opioid Manag 2012;8:57–61.
    OpenUrlCrossRefPubMed
  71. 71.↵
    1. Midboe AM,
    2. Lewis ET,
    3. Cronkite RC,
    4. et al
    . Behavioral medicine perspectives on the design of health information technology to improve decision-making, guideline adherence, and care coordination in chronic pain management. Behav Med Pract Policy Res 2011;1:35–44.
    OpenUrl
  72. 72.↵
    1. Harle CA,
    2. Bauer SE,
    3. Hoang HQ,
    4. Cook RL,
    5. Hurley RW,
    6. Fillingim RB
    . Decision support for chronic pain care: how do primary care physicians decide when to prescribe opioids? a qualitative study. BMC Fam Pract 2015;16:48.
    OpenUrl
  73. 73.↵
    1. Finley EP,
    2. Garcia A,
    3. Rosen K,
    4. McGeary D,
    5. Pugh MJ,
    6. Potter JS
    . Evaluating the impact of prescription drug monitoring program implementation: a scoping review. BMC Health Serv Res 2017;17:420.
    OpenUrl
  74. 74.↵
    1. Fink DS,
    2. Schleimer JP,
    3. Sarvet A,
    4. et al
    . Association between prescription drug monitoring programs and nonfatal and fatal drug overdoses: A systematic review. Ann Intern Med 2018;168:783. Jun 5.
    OpenUrl
  75. 75.↵
    Grading quality of evidence and strength of recommendations. BMJ 2004;328:1490.
    OpenUrlAbstract/FREE Full Text
  76. 76.↵
    Health Quality Ontario. Recommendations for adoption: Opioid Prescribing for chronic pain [Internet]. 2018 [cited 2018 Jul 20]. Available from: http://www.hqontario.ca/Portals/0/documents/evidence/quality-standards/qs-opioid-chronic-pain-recommendations-for-adoption-en.pdf.
  77. 77.↵
    1. Ciani O,
    2. Buyse M,
    3. Drummond M,
    4. Rasi G,
    5. Saad ED,
    6. Taylor RS
    . Time to review the role of surrogate end points in health policy: State of the art and the way forward. Value Health 2017;20:487–95.
    OpenUrl
  78. 78.↵
    1. Wittes J,
    2. Lakatos E,
    3. Probstfield J
    . Surrogate endpoints in clinical trials: Cardiovascular diseases. Statist Med 1989;8:415–25.
    OpenUrl
  79. 79.↵
    1. Gomes T,
    2. Khuu W,
    3. Martins D,
    4. et al
    . Contributions of prescribed and non-prescribed opioids to opioid related deaths: Population based cohort study in Ontario. Canada. BMJ 2018;362:k3207.
    OpenUrl
  80. 80.↵
    1. Mars SG,
    2. Bourgois P,
    3. Karandinos G,
    4. Montero F,
    5. Ciccarone D
    . “Every ‘never’ I ever said came true”: Transitions from opioid pills to heroin injecting. Int J Drug Policy 2014;25:257–66.
    OpenUrlCrossRefPubMed
  81. 81.↵
    1. Compton WM,
    2. Jones CM,
    3. Baldwin GT
    . Relationship between nonmedical prescription-opioid use and heroin use. N Engl J Med 2016;374:154–63.
    OpenUrlCrossRefPubMed
  82. 82.↵
    1. Baldwin N,
    2. Gray R,
    3. Goel A,
    4. Wood E,
    5. Buxton JA,
    6. Rieb LM
    . Fentanyl and heroin contained in seized illicit drugs and overdose-related deaths in British Columbia, Canada: An observational analysis. Drug Alcohol Depend 2018;185:322–7.
    OpenUrlCrossRefPubMed
  83. 83.↵
    1. Rubin R
    . Illicit fentanyl driving opioid overdose deaths. JAMA 2017;318:2174.
    OpenUrlPubMed
  84. 84.↵
    1. O'Donnell J,
    2. Gladden RM,
    3. Seth P
    . Trends in deaths involving heroin and synthetic opioids excluding methadone, and law enforcement drug product reports, by census region—United States. [Internet] 2017;2006–2015. [cited 2018 Oct 1]. Available from: https://www-cdc-gov.myaccess.library.utoronto.ca/mmwr/volumes/66/wr/mm6634a2.htm?s_cid=mm6634a2_e.
  85. 85.↵
    1. Rhodes E,
    2. Wilson M,
    3. Robinson A,
    4. Hayden JA,
    5. Asbridge M
    . The effectiveness of prescription drug monitoring programs at reducing opioid-related harms and consequences: A systematic review. BMC Health Serv Res 2019;19:784.
    OpenUrl
  86. 86.↵
    1. Heres S,
    2. Davis J,
    3. Maino K,
    4. Jetzinger E,
    5. Kissling W,
    6. Leucht S
    . Why olanzapine beats risperidone, risperidone beats quetiapine, and quetiapine beats olanzapine: An exploratory analysis of head-to-head comparison studies of second-generation antipsychotics. AJP 2006;163:185–94.
    OpenUrl
  87. 87.↵
    1. Lexchin J,
    2. Bero LA,
    3. Djulbegovic B,
    4. Clark O
    . Pharmaceutical industry sponsorship and research outcome and quality: Systematic review. BMJ 2003;326:1167–1170.
    OpenUrlAbstract/FREE Full Text
  88. 88.↵
    1. Lesser LI,
    2. Ebbeling CB,
    3. Goozner M,
    4. Wypij D,
    5. Ludwig DS
    . Relationship between funding source and conclusion among nutrition-related scientific articles. PLoS Med 2007;4:e5.
    OpenUrlCrossRefPubMed
  89. 89.↵
    1. Procyshyn RM,
    2. Chau A,
    3. Fortin P,
    4. Jenkins W
    . Prevalence and outcomes of pharmaceutical industry-sponsored clinical trials involving clozapine, risperidone, or olanzapine. Can J Psychiatry 2004;49:601–6.
    OpenUrlCrossRefPubMed
  90. 90.↵
    1. Turner EH,
    2. Matthews AM,
    3. Linardatos E,
    4. Tell RA,
    5. Rosenthal R
    . Selective publication of antidepressant trials and its influence on apparent efficacy. N Engl J Med 2008;358:252–60.
    OpenUrlCrossRefPubMed
  91. 91.↵
    1. Doshi P,
    2. Dickersin K,
    3. Healy D,
    4. Vedula SS,
    5. Jefferson T
    . Restoring invisible and abandoned trials: A call for people to publish the findings. BMJ 2013;346:f2865–f2865.
    OpenUrlFREE Full Text
  92. 92.↵
    1. Chan AW,
    2. Hróbjartsson A,
    3. Haahr MT,
    4. Gøtzsche PC,
    5. Altman DG
    . Empirical evidence for selective reporting of outcomes in randomized trials: Comparison of protocols to published articles. JAMA 2004;291:2457–65.
    OpenUrlCrossRefPubMed
  93. 93.↵
    Electronic health records vendor to pay $145 million to resolve criminal and civil investigations [Internet]. 2020 [cited 2020 Feb 14]. Available from: https://www.justice.gov/opa/pr/electronic-health-records-vendor-pay-145-million-resolve-criminal-and-civil-investigations-0.
  94. 94.↵
    1. Mair FS,
    2. May C,
    3. O'Donnell C,
    4. Finch T,
    5. Sullivan F,
    6. Murray E
    . Factors that promote or inhibit the implementation of e-health systems: an explanatory systematic review. Bull World Health Organ 2012;90:357–64.
    OpenUrlCrossRefPubMed
  95. 95.↵
    1. Black AD,
    2. Car J,
    3. Pagliari C,
    4. et al
    . The impact of eHealth on the quality and safety of health care: A systematic overview. PLoS Med 2011;8:e1000387. [Internet] Jan 18 [cited Jul 18] Available from: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3022523/.
    OpenUrlCrossRefPubMed
  96. 96.↵
    National Center for Injury Prevention and Control. Integrating & Expanding Prescription Drug Monitoring Program Data: Lessons from Nine States. Centre for Disease Control and Prevention 2017.
  97. 97.↵
    1. Moore GF,
    2. Audrey S,
    3. Barker M,
    4. et al
    . Process evaluations of complex interventions: UK Medical Research Council (MRC) guidance [Internet]. MRC Population Health Science Research Network 2014;[cited 2018 Jul 23]. Available from: https://mrc.ukri.org/documents/pdf/mrc-phsrn-process-evaluation-guidance-final/
  98. 98.↵
    1. Chan CL,
    2. Leyrat C,
    3. Eldridge SM
    . Quality of reporting of pilot and feasibility cluster randomised trials: A systematic review. BMJ Open 2017;7:e016970.
    OpenUrl
  99. 99.↵
    1. Liu H,
    2. Muhunthan J,
    3. Hayek A,
    4. et al
    . Examining the use of process evaluations of randomised controlled trials of complex interventions addressing chronic disease in primary health care—A systematic review protocol. Syst Rev 2016;5. [cited 2018 Sep 24] Available from: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4986376/.
  100. 100.↵
    1. Oakley A,
    2. Strange V,
    3. Bonell C,
    4. Allen E,
    5. Stephenson J
    . Process evaluation in randomised controlled trials of complex interventions. BMJ 2006;332:413–6.
    OpenUrlFREE Full Text
  101. 101.↵
    1. Greenhalgh T,
    2. Russell J
    . Why do evaluations of eHealth programs fail? An alternative set of guiding principles. PLoS Med 2010;7:e1000360. Available from: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2970573/.
    OpenUrlCrossRefPubMed
  102. 102.↵
    1. Basit MA,
    2. Baldwin KL,
    3. Kannan V,
    4. et al
    . Agile acceptance test–driven development of clinical decision support advisories: feasibility of using open source software. JMIR Med Inform 2018;6:e23. [Internet] Apr 13 [cited 2019 May] Available from: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5924365/.
    OpenUrl
  103. 103.↵
    1. Kannry J,
    2. McCullagh L,
    3. Kushniruk A,
    4. Mann D,
    5. Edonyabo D,
    6. McGinn T
    . A Framework for Usable and Effective Clinical Decision Support: Experience from the iCPR Randomized Clinical Trial. eGEMs 2015;3:10.
    OpenUrl
  104. 104.↵
    1. Kannan V,
    2. Fish J,
    3. Mutz J,
    4. et al
    . Rapid Development of specialty population registries and quality measures from electronic health record data: An agile framework. Methods Inf Med 2017;56:e74–83.
    OpenUrl
Back to top

In this issue

Print
Download PDF
Article Alerts
Email Article
Citation Tools

Jump to section

Keywords