The Accuracy of Trigger Tools to Detect Preventable Adverse Events in Primary Care: A Systematic Review
=======================================================================================================

* Joshua Davis
* Nicole Harrington
* Heather Bittner Fagan
* Barbara Henry
* Margot Savoy

## Abstract

*Purpose:* To understand the ability of trigger tools to detect preventable adverse events (pAEs) in the primary care outpatient setting using the Institute for Healthcare Improvement's (IHI) Outpatient Adverse Event Trigger Tool (IHI Tool).

*Methods:* The OVID MEDLINE and OVID MEDLINE In-process and non-Indexed citations databases were queried using controlled vocabulary and Medical Subject Headings related to the concepts “primary care” and “adverse events.” Included articles were conducted in the outpatient setting, used at least 1 of the triggers identified in the IHI Tool, and identified pAEs of any type. Articles were selected for inclusion based first on assessment of titles then abstracts by 2 trained reviewers independently, followed by full text review by 2 authors.

*Results:* Our search identified 6435 unique articles, and we included 15 in our review. The most common studied trigger was laboratory abnormalities. The most common pAEs were medication errors followed by unplanned hospitalizations. The effectiveness of triggers in identifying AEs varied widely.

*Conclusion:* There is insufficient data on the IHI Tool and its use to identify pAEs in the general real-world outpatient setting. Health care providers of the primary care setting may benefit from better trigger tools and other methods to help them detect pAEs. More research is needed to further evaluate the effectiveness of trigger tools to reduce barriers of cost and time and improve patient safety.

*   Ambulatory Care
*   Family Physicians
*   Medical Subject Headings
*   Medication Errors
*   MEDLINE
*   NHS
*   Patient Safety
*   Primary Health Care

Since the 1999 landmark report by the Institute of Medicine (IOM), *To Err is Human*1, patient safety has become a priority in health care systems.2 Although, most work has focused on acute, inpatient settings3,4, a much larger number of patients seek care in the outpatient setting.5,6 Therefore, identifying preventable adverse events (pAEs) in the outpatient setting is important.

Medical errors are mistakes that may or may not cause harm (eg, a minor error in dosing a medication). Not all medical errors are adverse events (AEs). An AE is harm caused from medical care and not the disease process itself (eg, a rash in response to an antibiotic vs a rash in response to an infection). A pAE is an AE or harm to a patient that is the “result of care that fell below the standard expected of physicians in their community”7 or “avoidable by any means currently available unless that means was not considered standard care.”8 (Figure 1). Voluntary reporting and chart review are common, traditionally accepted methods used to identify medical errors and pAEs in the hospital setting.9,10 These methods do not translate easily to the outpatient setting for several reasons. First, patients are not under constant observation in the outpatient setting. In fact, the provider-patient contact time is relatively limited. In addition, it is well documented that voluntary reporting vastly underestimates the rate of medical errors.11,12 Finally, extensive chart review is not only costly, subjective, and unsustainable but also extremely time consuming and inefficient.13⇓–15 One method for streamlining error identification of “high-risk” charts that should be reviewed for errors and AEs relies uses a collection high-risk situation or events called “triggers.”16,17 Several tools have been proposed18⇓–20, but one of the most recognized is that from the Institute for Healthcare Improvement (IHI).19 This IHI Outpatient Adverse Event Trigger Tool (IHI Tool) was developed by experts at the IHI using malpractice claims data and outlines eleven triggers to identify patient records at risk for an AE.19 The list of triggers can be found in Table 1. This tool was chosen because it has been previously described as particularly relevant to the outpatient and primary care setting and it represents an established list of predefined triggers.20

![Figure 1.](http://www.jabfm.org/https://www.jabfm.org/content/jabfp/31/1/113/F1.medium.gif)

[Figure 1.](http://www.jabfm.org/content/31/1/113/F1)

Figure 1. 
Venn diagram of relationship between medical errors, adverse events (AEs), and preventable adverse events (pAEs).

View this table:
[Table 1.](http://www.jabfm.org/content/31/1/113/T1)

Table 1. 
Summary of Studies included in a Systematic Review of Trigger Tools to Identify Preventable Adverse Events in the Outpatient Setting

The tool developed by the IHI has been reportedly validated in outpatients to detect any AEs, preventable or not.19 Although there are reasons why assessing preventability can be problematic21, pAEs and medical errors are opportunities for many safety interventions22 and are often the most easily accepted improvement opportunities by providers due to their intuitive appeal as targets for improvement. We identified 5 systematic reviews that focused on safety events in the outpatient setting.15,23⇓⇓–26 However, none of these focused on the accuracy of tools in identifying pAEs.

Therefore, the goal of this systematic review was to determine the accuracy of each of the components of the IHI Tool, alone or in combination, in identifying pAEs in the outpatient primary care setting.

## Methods

This review was conducted by following the Preferred Reporting for Systematic Reviews and Meta-Analyses (PRISMA) Guidelines.27 (online Appendix 1) A master's prepared medical librarian (BH) conducted a comprehensive literature search for English-language articles published on the use of a trigger tool to identify AEs in the outpatient setting. Dates searched were 1946 to Present in Ovid MEDLINE and Ovid MEDLINE In-Process & Other Non-Indexed Citations.

An initial search was run in December 2015. Following discussions among team members, the search was modified several times, with the final search strategy finalized and performed in February 2016. The search was updated in February 2017. We chose relevant controlled vocabulary (Medical Subject Headings and keywords) to capture the concepts of the outpatient setting and AEs. The search for these 2 concepts yielded 6435 articles. (Full search strategy available in online Appendix 2).

All article titles and abstracts were independently reviewed for inclusion by at least 2 trained reviewers (JD, NH, KB). If either reviewer selected a reference, they ordered the full text for further review. Using this strategy, 158 articles were obtained. The percent agreement on initial independent selection of articles for further review was 97.4%. Interrater reliability using Cohen's Kappa was κ = 0.34 (95% CI, 0.26 to 0.41). To identify other relevant articles, the reference sections of all included articles were checked by one of the authors (JD or NH). As we only included primary research studies in our systematic review, we checked reference sections of systematic reviews identified during the search for potentially relevant primary studies.

### Inclusion and Exclusion Criteria

At the outset, we developed a comprehensive systematic review protocol, including operational definitions, inclusion and exclusion criteria, and search strategy details. Our operational definition of a trigger was “a signal for detecting likely AEs.”28 Figure 1 illustrates the components of patient safety that factor into deliverables of medical care. A priori, we determined that although the IHI Outpatient Trigger Tool was designed to detect any AEs, the focus of our systematic review would be on pAEs to have a more clinically relevant impact. Because the reporting and assessment of AEs is highly variable, we determined that pAEs detected by any means in the outpatient setting would be eligible for inclusion. Preventability was assessed using the original description in the article and whether the study assessed for preventability.

Articles meeting all 3 of the following criteria were eligible for review: research that used any of the triggers identified in the IHI's Tool19, assessed pAEs by any means, occurred primarily in the primary care outpatient setting. There were a number of exclusion criteria, including articles that were non-English language, did not use outpatient data, did not assess preventability of AEs, greater than 50% pediatric patients, specialty-focused, fewer than 10 patients or case reports, or no primary data (reviews, systematic reviews, editorials, newsletters).

### Abstraction Process

The team used an iterative process to develop and pilot test an abstraction form designed to confirm final eligibility for full review, assess article characteristics, and extract data relevant to the study. Each article was independently abstracted by 2 trained reviewers (JD and NH). These 2 reviewers along with 2 authors who were not involved in the abstraction process (HBF and MS) discussed and combined the 2 summaries into a final version. All abstraction disagreements were minor and were resolved during discussions among all 4 reviewers. More than half of these were typographic or human error on the part of the reviewer. On rereading the article, a clear “correct” answer was found and agreed on.

### Quality Assessment

Systematic review guidelines recommend the use of a measure of risk of bias within studies.27 Heterogeneity among studies, including disparate methods and training of reviewers has been cited as a reason to forego quality assessment altogether.28 In addition, there is no accepted standard for quality assessment of patient safety studies of this type. We used a tool from the National Heart, Lung, and Blood Institute (NHLBI) to assess the quality of the articles included in our review.29 It should be noted, though, that this tool is for use with observational studies in general, and not patient safety articles.29 We therefore modified the tool and categorized the trigger as the exposure and the pAE as the outcome. As recommended by the developers of the tool, the results were not tallied, but the interpretation of the results and final categorizations of articles was based on the perceived overall risk of bias. According to statements from the Standards for Reporting of Diagnostic Accuracy Studies (STARD), scales that numerically summarize multiple components into a single number are misleading and unhelpful.30 Thus, the authors applied the NHLBI Quality Assessment Tool and the STARD tool to our selected group of 15 articles to help characterize quality.29,30 To make comparisons between the 2 tools comparable, the STARD checklist was transformed into a rating of good, fair, and poor using a cutoff of >75% of applicable checklist items as a cutoff for good, and less than 60% for poor. No outliers were identified and all 15 were included in our review. This was done independently by 2 authors who met and agreed on categorization by consensus with little disagreement. Given the heterogeneity expected, it was determined that a funnel plot or other assessment of publication bias would be impossible.

## Results

Our initial search strategy identified 6,435 articles (Figure 2). After our initial review of titles and abstracts, we requested 158 full-text articles. Of these full-text articles, 10 fulfilled inclusion and exclusion criteria and are included in this review. One review article was published in English but cited a study originally published in French.31 We did obtain the original French study32 and had it translated by a native French speaker to ensure the data in the English version was an accurate representation of the original study. In addition, reviews of reference sections of included articles and relevant systematic reviews identified 4 new articles17,33⇓⇓–36 for inclusion. Therefore, 15 studies with a total of 278,212 patients and 126,197 incidents of positive triggers were included in this review.17,20,31,33⇓⇓⇓⇓⇓⇓⇓⇓⇓⇓⇓–45

![Figure 2.](http://www.jabfm.org/https://www.jabfm.org/content/jabfp/31/1/113/F2.medium.gif)

[Figure 2.](http://www.jabfm.org/content/31/1/113/F2)

Figure 2. 
Flow diagram for selection of articles in a systematic review of trigger tools for identifying preventable adverse events (pAEs) in the outpatient setting.

Of the 15 included studies, 9 were conducted in the United States, 3 in Scotland, and 1 each in Brazil, France, and Australia (Table 2). Eight studies were published since 2010 and 2 were published before 2000. Most were retrospective, but 3 were prospective. Most studies involved only 1 site, but the number ranged from 1 to 40.

View this table:
[Table 2.](http://www.jabfm.org/content/31/1/113/T2)

Table 2. 
Institute for Healthcare Improvement18 Outpatient Adverse Event Trigger Explanations

### Adverse Events

Only 1 study evaluated a control group, using a case control methodology and a random selection of patients without a trigger.38 Therefore, the quantification of the accuracy of the triggers is limited to the positive predictive value (PPV). The value ranged widely from 0.05% for drug laboratory rules39 to 92% for hospitalization for new rectal cancer.42 The PPV of polypharmacy ranged from 19.5% to 29.3% (depending on the number of medications used to define polypharmacy), for laboratory abnormalities ranged from 0.05%39 to 30.6%36, for hospital admissions ranged from 1.8%33 to 9%37, and for combined tools ranged from 1.8%33 to 24%.36 International normalized ratio (INR), in general, had a particularly high detection rate of adverse drug event (ADEs), as high as 96% in 1 study.38

The overall detection of AEs in the samples studied varied widely, from 1.4%39 to 14.6%38 (Table 2). In the 2 studies that examined fixable (ameliorable) events, all the AEs that were not preventable were ameliorable.38,44 The most commonly studied pAEs were medication related (preventable adverse drug events).33,36,38⇓⇓–41,44 Preventable or unplanned hospital admission was the second most commonly studied type of pAE.42,43,45 Three studies looked at any type of pAE identified in the chart.20,31,34,35,37

The most commonly used method to identify pAEs was chart review.20,31,33⇓⇓⇓⇓⇓⇓⇓⇓⇓⇓–44 Four studies used physicians only as reviewers35,37,38,40, 3 used a combination of physicians and pharmacists.33,36,41 1 used nurses34, 1 used pharmacists44, and 1 used a combination of physicians and nurses20, 5 studies used more than 1 reviewer and all these used at least 2 independent reviewers.20,33,37⇓–39 Interrater reliability of chart review was assessed in 4 studies33,37,40,44 and ranged from a Cohen's kappa of 0.3337 to 0.89.44 Bigby et al37 showed an intrarater reliability from 0.34 to .75 (*P* < .05). All 3 studies showed that interrater agreement on presence of an AE was higher than agreement on preventability/causation. Besides chart review, the other methods to identify pAEs included information from a research database43 and public data for the National Health System.45

### Triggers

The incidence of triggers in a random cohort of charts ranged from 45.6% (507/1111)42 to 1 study that had 1342 triggers in 622 episodes of care38 (Table 2). Several studies cited the same patients having multiple triggers within a given period of review. Honigman et al39 noted that “The relationship between computer-identified incidents and ADEs was often ‘many to 1′.” Across all studies, most of the triggers were identified by either computerized notification (8/14; 57.1%) or manual record review (6/14; 42.9%). One study also interviewed patients, in addition to manually obtaining lab values from the primary care physician.44

Several triggers in the IHI trigger tool were not used in any of the included studies: nursing home placement, surgical procedure, physician change, and complaint letter. Several studies assessed more than 1 of the triggers at the same time, but none of the studies used all of them. The most IHI triggers used in a study was 5. Eight studies17,20,33,36,39,40,42,44 paired triggers from the IHI tool with other triggers, such as free text searches of charts33,39,40, new allergy or “high-priority” codes20,35,40, ICD-9 codes39,40, antidote administration33,36, symptom review44, and incident reporting.33

Triggers that were studied individually included lab abnormality38⇓–40,44, polypharmacy43,45, and hospital admission.31,37,42 The definition of polypharmacy provided by the IHI tool is >5 medications; however, studies in this review used numbers from 4 to 12.43,45 Lab abnormalities evaluated across all studies included international normalized ratio (INR),33,34,36,38⇓⇓–41 creatinine17,33⇓⇓–36,38⇓⇓–41, hemoglobin17,33⇓⇓–36,40, potassium,33,36,39,40 aminotransferase enzymes33,35,37, eosinophilia33,35,39, platelet count33,36,38, toxic drug levels33,36,40, bilirubin36,40, blood urea nitrogen36,38, hemoglobin A1c33,36, thyroid-stimulating hormone (TSH)36,38, positive C. difficile33,36, and white blood cell count.40 Brenner et al37 found no patients with an undetectable TSH while on levothyroxine and Singh et al17 found that toxic drug levels, hemoglobin, platelet count, eosinophilia, potassium, bilirubin, alkaline phosphatase, hemoglobin A1c, and C. difficile had incredibly low PPVs.

### Related Findings

Several studies had findings that were related to our research question and would be interesting to those interested in studying triggers further. Elderly patients were often the focus of many studies17,33⇓⇓–36,43,44, and 4 studies found that older patients were more likely to have pAEs.20,37,43,45 One study that utilized 2 sites found that while implementation at 2 sites were successful, there were variable rates in success of using abnormal laboratory values in identifying pAEs across 2 sites.36 Bigby et al36 found that 9% of emergency admissions were preventable: 6.8% due to a combination of iatrogenic and patient factors, and 2.2% due exclusively to patient factors, for example, noncompliance. Brenner et al37 noted “that most ADEs [not only preventable ones] occurred during the self-management and monitoring stages of medication use, rather than being prescribing or dispensing errors”; however, Singh et al17 noted that preventable ADEs “most commonly originated during the prescribing or administration of medications.” Payne et al44 found that number of conditions/diagnoses was more predictive of preventable hospitalization than number of medications.

Of the studies, 4 reported average time to collect data20,34,36,42, and none reported cost involved in detecting either the triggers or both the pAEs and the triggers. In 1 study by Macnee et al41, the initial time investment from information technology to establish a computerized tool to detect hospital admissions for cancer ranged from “a few hours to 2 days” across 5 clinical sites. The maximum time reported to review the outpatient records for pAEs of about 20 minutes per record.35 One study reported that the “duration of time taken was ok,”17 while another reported, “…the low PPVs suggest that extensive investigator time will be required to use any of the sources investigated.”33

### Quality Assessment

Overall, quality assessment showed a moderate risk of bias. Using the NHLBI tool, we rated 4 articles (27%) as “Good,”20,38,40,45 6 (40%) as “Fair”33,35,36,39,43,44, and 5 (33%) as “Poor.”17,31,37,41,42 Using the STARD tool, we rated 4 articles (27%) as “good”33,36,38,40, 6 articles (40%) as “fair”20,35,36,39,42⇓–44, and 5 articles (33%) as “poor.”31,37,41,43,45 Percent agreement among reviewers was 84%. Complete details are available from the authors on request.

## Discussion

An ideal trigger tool would efficiently detect, monitor, and measure harm and identify AEs. As studies have shown, trigger tools can identify a number of different levels of harm and are tailored to health care organizations and health professionals that can effectively use the tools in conjunction with other patient safety interventions. The IHI Global Trigger tool is among the more developed, recommended tools for health organizations. The tool was designed to help health organizations implement quality improvement approaches and complement other information sources regarding potential patient harm. Moreover, the ideal tool (or system of tools) would be able to detect AEs in advance thereby preventing harm.

### Summary of Findings

Our study identified 15 articles that assessed the accuracy of any of the triggers in the IHI Tool in identifying pAEs in the general outpatient setting. The studies had a moderate risk of bias according to the NHLBI tool. The outcome measurements were remarkably heterogeneous and precluded our ability to quantitatively compare the studies. Our data suggest that accurate trigger tools remain elusive, primarily because of the high number of false positives detected with current tools. Certain items from the IHI tool had no studies on their accuracy in the primary care setting and deserve specific further research: nursing home placement, surgical procedure, change of physician, and patient complaint letter.

In another recent systematic review, the IHI Global Trigger Tool had only 4 studies conducted in the outpatient setting.21 Despite this, the role of outpatient medical care in patient safety is being increasingly recognized as equally if not more important than that of inpatient medicine.3,4 A different approach to identifying AEs in the outpatient setting is needed. Inpatient care can use a single admission as representative of an entire episode, whereas outpatient care is a time-limited representation of a continuum of care, much of which is unobserved. Identifying AEs in the outpatient setting will require a combination of robust identification mechanisms such as trigger tools, scoring systems, chart reviews, and perhaps patient reporting.

### Comparison of Other Review Studies

Robust outpatient safety programs will rest on such mechanisms but will need to address the barriers of time and cost.45 Very few studies included in our study evaluated time or cost of implementing triggers. A successful trigger tool must add value and efficiency in identifying patients or charts that are a high risk of AEs. Preventable AEs are those that are currently actionable for improvement among family medicine clinicians. A recent systematic review summarized the rate of AEs in the outpatient setting and identified an overall median AE rate of 4% (range, <1% to 24%) in over 100 primary studies.25 A quality trigger tool should identify a significantly higher percentage of AEs than those that could be identified by random chart review. In addition, random chart review would undoubtedly miss relevant and preventable AEs. In our included studies, conclusions on the use of trigger tools for identifying pAEs varied. For example, DeWet et al20 stated that “the tool may have greater utility as a research rather than an audit technique”; however, Macnee et al41 stated that their approach “may be 1 of the most effective, low-cost methods of identifying critical occurrences in ambulatory care.” Indeed, further work on improving the identification of AEs, including pAEs, is desperately needed. A systemic review by Hatoun et al23 notes that “…trigger events are used increasingly in the inpatient setting, and although others now exist for ambulatory surgery, few exist for ambulatory primary care.” To better understand the value of trigger tools, it will be important for future research to use control groups (like voluntary reported and random chart review) and to evaluate the time and cost necessary to identify triggers. Some combination of triggers and chart review and patient input in a standardized, multi-step process may also improve the accuracy of identification of pAEs.

### Implications

This study was limited to the identification of pAEs to be more relevant to real practitioners looking to improve the safety of real practice. However, the concept of preventability is not well defined, let alone practical, for research purposes. There is often confusion among both health-care practitioners and researchers about preventability. The determination of preventability and the value of pAEs versus AEs as a quality measure is a hotly debated topic in patient safety.47 Proponents of preventability assessment argue that the area for the greatest current potential improvement is in pAEs and that nonpreventable AEs are of little value. They also argue that the use of overall AEs as a measure of safety seems to overestimate the number of safety events that are preventable versus those that are inherent (as in the IOM reports).1,48,49 pAEs are also the easiest errors for those who are unfamiliar with patient safety terminology to identify as areas for improvement. Those against the use of preventability measures cite that its assessment is unreliable50⇓–52, the definition of preventable changes as technology and research evolve, most AEs are preventable53, and determinations of preventability fail to take into account the existing disease and surrounding circumstances.48 To that end, our results do support the notion that reliability of preventability assessment is lower than that of AE identification, although the interrater reliability for both varied widely.

### Limitations

This review has limitations. First, we chose to focus our search on the triggers listed in the IHI Tool rather than a more comprehensive list. Various definitions of triggers and various methods of identifying pAEs limits the conclusions that can be drawn from this analysis, given that results could have been missed or misclassified. The possibility of missing potentially relevant articles always exists, although we used a robust search strategy, multiple databases, an a priori protocol, and a trained medical librarian to attempt to mitigate this, although publication bias is likely to be especially important as much of this work may be conducted under the banner of quality improvement. In addition, the quality, limited reporting, and heterogeneity of the original studies precluded formal meta-analysis and limited quality assessment. Finally, no minimum standards of effectiveness for trigger tools are established. Our authors agreed that a tool should identify more events (perhaps twice as many) as random chart review. Yet feasibility, a more subjective measure, matters a great deal and feasibility would depend on who was trying to use the tool and for what purpose.

## Conclusion

Given the limited data available, it is premature to endorse the universal use of the IHI Tool to identify outpatient pAEs. This work highlights a starting point for future research on the topic of patient safety in the outpatient primary care setting. Greater emphasis on patient safety research will help direct health care providers to consider the use of triggers to reduce barriers of cost and time associated with a large number of random chart reviews. An increased focus on identification of pAEs in the outpatient setting is urgently needed to improve patient safety.

## Acknowledgments

The authors acknowledge the work of Kameron Brown for his help in obtaining and screening articles for this study.

## Appendix

View this table:
[Appendix 1.](http://www.jabfm.org/content/31/1/113/T3)

Appendix 1. 
Preferred Reporting Items for Systematic Reviews and Meta-Analyses Checklist*

View this table:
[Appendix 2.](http://www.jabfm.org/content/31/1/113/T4)

Appendix 2. 
Search Strategy for a Systematic Review of Trigger Tools to Detect Preventable Adverse Events in the Outpatient Setting

## Notes

*   This article was externally peer reviewed.

*   *Funding:* Work supported by an Institutional Development Award (IDeA) from the National Institute of General Medical Sciences of the National Institutes of Health under grant number U54-GM104941 (PI: Binder-Macleod).

*   *Conflict of interest:* JD, HBF, and MS have received an honorarium from the American Academy of Family Physicians to write a review on patient safety.

*   To see this article online, please go to: [http://jabfm.org/content/31/1/113.full](http://jabfm.org/content/31/1/113.full).

*   Received for publication June 26, 2017.
*   Revision received September 14, 2017.
*   Accepted for publication September 19, 2017.

## References

1.  1.Institute of Medicine. To err is human: Building a safer health system. Washington, DC: The National Academies Press, 2000.
    
    

2.  2.Wachter RM. Patient safety at ten: Unmistakable progress, troubling gaps. Health Aff (Millwood) 2010;29:165–73.
    
    [Abstract/FREE Full Text](http://www.jabfm.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6OToiaGVhbHRoYWZmIjtzOjU6InJlc2lkIjtzOjg6IjI5LzEvMTY1IjtzOjQ6ImF0b20iO3M6MjA6Ii9qYWJmcC8zMS8xLzExMy5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 

3.  3.Gandhi TK, Lee TH. Patient safety beyond the hospital. N Engl J Med 2010;363:1001–3.
    
    [CrossRef](http://www.jabfm.org/lookup/external-ref?access_num=10.1056/NEJMp1003294&link_type=DOI) 
    
    [PubMed](http://www.jabfm.org/lookup/external-ref?access_num=20825311&link_type=MED&atom=%2Fjabfp%2F31%2F1%2F113.atom) 
    
    [Web of Science](http://www.jabfm.org/lookup/external-ref?access_num=000281609700003&link_type=ISI) 

4.  4.Agency for Healthcare Research and Quality. Patient safety in ambulatory care. July 2016. Available from: [https://psnet.ahrq.gov/primers/primer/16](https://psnet.ahrq.gov/primers/primer/16). Accessed June 17, 2017.
    
    

5.  5.American Hospital Association. Chartbook: Trends affecting hospitals and health systems. Chart 3.9 and 3.10. Available from: [http://www.aha.org/research/reports/tw/chartbook/ch3.shtml](http://www.aha.org/research/reports/tw/chartbook/ch3.shtml). 2016. Accessed June 17, 2017.
    
    

6.  6.Center for Disease Control and Prevention: National Center for Health Statistics. Ambulatory care use and physician office visits. June 13, 2016. Available from: [http://www.cdc.gov/nchs/fastats/physician-visits.htm](http://www.cdc.gov/nchs/fastats/physician-visits.htm). Accessed June 17, 2017.
    
    

7.  7.Brennan TA, Leape LL, Laird NM, et al. Incidence of adverse events and negligence in hospitalized patients. Results of the Harvard Medical Practice Study I. N Engl J Med 1991;324:370–6.
    
    [CrossRef](http://www.jabfm.org/lookup/external-ref?access_num=10.1056/NEJM199102073240604&link_type=DOI) 
    
    [PubMed](http://www.jabfm.org/lookup/external-ref?access_num=1987460&link_type=MED&atom=%2Fjabfp%2F31%2F1%2F113.atom) 
    
    [Web of Science](http://www.jabfm.org/lookup/external-ref?access_num=A1991EW29600004&link_type=ISI) 

8.  8.Thomas EJ, Brennan TA. Incidence and types of preventable adverse events in elderly patients: Population based review of medical records. Br Med J 2000;320:741–4.
    
    [Abstract/FREE Full Text](http://www.jabfm.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiYm1qIjtzOjU6InJlc2lkIjtzOjEyOiIzMjAvNzIzNy83NDEiO3M6NDoiYXRvbSI7czoyMDoiL2phYmZwLzMxLzEvMTEzLmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

9.  9.Hanskamp-Sebregts M, Zegers M, Vincent C, van Gurp PJ, de Vet HC, Wollersheim H. Measurement of patient safety: A systematic review of the reliability and validity of adverse event detection with record review. BMJ Open 2016;6:e011078.
    
    [Abstract/FREE Full Text](http://www.jabfm.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NzoiYm1qb3BlbiI7czo1OiJyZXNpZCI7czoxMToiNi84L2UwMTEwNzgiO3M6NDoiYXRvbSI7czoyMDoiL2phYmZwLzMxLzEvMTEzLmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

10. 10.Sari ABA, Sheldon TA, Cracknell A, et al. Sensitivity of routine systems for reporting patient safety incidents in an NHS hospital: Retrospective patient case note review. Br Med J 2007;334:79–82.
    
    [Abstract/FREE Full Text](http://www.jabfm.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiYm1qIjtzOjU6InJlc2lkIjtzOjExOiIzMzQvNzU4NC83OSI7czo0OiJhdG9tIjtzOjIwOiIvamFiZnAvMzEvMS8xMTMuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

11. 11.Cullen DJ, Bates DW, Small SD, Cooper JB, Nemeskal AR, Leape LL. The incident reporting system does not detect adverse drug events: A problem for quality improvement. Jt Comm J Qual Improv 1995;21:541–8.
    
    [PubMed](http://www.jabfm.org/lookup/external-ref?access_num=8556111&link_type=MED&atom=%2Fjabfp%2F31%2F1%2F113.atom) 

12. 12.Murff HJ, Patel VL, Hripcsak G, Bates DW. Detecting adverse events for patient safety research: A review of current methodologies. J Biomed Inform 2003;36(1–2):131–143.
    
    [CrossRef](http://www.jabfm.org/lookup/external-ref?access_num=10.1016/j.jbi.2003.08.003&link_type=DOI) 
    
    [PubMed](http://www.jabfm.org/lookup/external-ref?access_num=14552854&link_type=MED&atom=%2Fjabfp%2F31%2F1%2F113.atom) 
    
    [Web of Science](http://www.jabfm.org/lookup/external-ref?access_num=000186072200013&link_type=ISI) 

13. 13.Localio AR, Weaver SL, Landis JR. Identifying adverse events caused by medical care: Degree of physician agreement in a retrospective chart review. Ann Intern Med 1996;125:457–64.
    
    [CrossRef](http://www.jabfm.org/lookup/external-ref?access_num=10.7326/0003-4819-125-6-199609150-00005&link_type=DOI) 
    
    [PubMed](http://www.jabfm.org/lookup/external-ref?access_num=8779457&link_type=MED&atom=%2Fjabfp%2F31%2F1%2F113.atom) 
    
    [Web of Science](http://www.jabfm.org/lookup/external-ref?access_num=A1996VF88200006&link_type=ISI) 

14. 14.Resar RK, Rozich JD, Classen D. Methodology and rationale for the measurement of harm with trigger tools. Qual Saf Health Care 2003;12(Suppl 2):39–45.
    
    

15. 15.Spencer R, Cambell SM. Tools for primary care patient safety: A narrative review. BMC Fam Pract 2014;15:166.
    
    [CrossRef](http://www.jabfm.org/lookup/external-ref?access_num=10.1186/1471-2296-15-166&link_type=DOI) 
    
    [PubMed](http://www.jabfm.org/lookup/external-ref?access_num=25346425&link_type=MED&atom=%2Fjabfp%2F31%2F1%2F113.atom) 

16. 16.National Health Service. The primary care trigger tool: practical guidance for GP team. Available from: [http://www.nes.scot.nhs.uk/media/6361/Primary%20Care%20Trigger%20Tool%20-%20Practical%20Guidance.pdf](http://www.nes.scot.nhs.uk/media/6361/Primary%20Care%20Trigger%20Tool%20-%20Practical%20Guidance.pdf). 2010. Accessed June 17, 2017.
    
    

17. 17.Singh R, McLean-Plunckett EA, Kee R, et al. Experience with a trigger tool for identifying adverse drug events among older adults in ambulatory primary care. Qual Saf Health Care 2009;18:199–204.
    
    [Abstract/FREE Full Text](http://www.jabfm.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MzoicWhjIjtzOjU6InJlc2lkIjtzOjg6IjE4LzMvMTk5IjtzOjQ6ImF0b20iO3M6MjA6Ii9qYWJmcC8zMS8xLzExMy5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 

18. 18.Institute for Healthcare Improvement. IHI Outpatient Adverse Event Trigger Tool. Available from: [http://www.ihi.org/resources/Pages/Tools/OutpatientAdverseEventTriggerTool.aspx](http://www.ihi.org/resources/Pages/Tools/OutpatientAdverseEventTriggerTool.aspx). October 2006. Accessed June 17, 2017.
    
    

19. 19.1000 Lives. Tools for improvement: How to use trigger tools. April 2012. Available from: [http://www.1000livesplus.wales.nhs.uk/sitesplus/documents/1011/T4I%20%284%29%20How%20to%20use%20Trigger%20Tools%20%28Feb%202011%29%20Web.pdf](http://www.1000livesplus.wales.nhs.uk/sitesplus/documents/1011/T4I%20%284%29%20How%20to%20use%20Trigger%20Tools%20%28Feb%202011%29%20Web.pdf). Accessed June 17, 2017.
    
    

20. 20.de Wet C, Bowie P. The preliminary development and testing of a global trigger tool to detect error and patient harm in primary-care records. Postgrad Med J 2009;85:176–80.
    
    [Abstract/FREE Full Text](http://www.jabfm.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MTI6InBvc3RncmFkbWVkaiI7czo1OiJyZXNpZCI7czoxMToiODUvMTAwMi8xNzYiO3M6NDoiYXRvbSI7czoyMDoiL2phYmZwLzMxLzEvMTEzLmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

21. 21.Hibbert PD, Molloy CJ, Hooper TD, et al. The application of the Global Trigger Tool: A systematic review. Int J Qual Health Care 2016;28:640–9.
    
    

22. 22.1.  Henriksen K, 
    2.  Battles JB, 
    3.  Marks ES, 
    4.  Lewin DI
    
    Kizer KW, Stegun MB. Serious reportable adverse events in health care. In: Henriksen K, Battles JB, Marks ES, Lewin DI, eds. Advances in patient safety: From research to implementation (Volume 4: Programs, tools, and products). Rockville, MD: Agency for Healthcare Research and Quality (US); 2005.
    
    

23. 23.Hatoun J, Chan JA, Yaksic E, et al. A systematic review of patient safety measures in adult primary care. Am J Med Qual 2017;32:237–45.
    
    

24. 24.Lydon S, Cupples ME, Murphy AW, Hart N, O'Connor P. A systematic review of measurement tools for the proactive assessment of patient safety in general practice. J Patient Saf 2017 Apr 4 [Epub ahead of print].
    
    

25. 25.Panesar SS, deSilva D, Carson-Stevens A, et al. How safe is primary care? A systematic review. BMJ Qual Saf 2016;25:544–53.
    
    [Abstract/FREE Full Text](http://www.jabfm.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MzoicWhjIjtzOjU6InJlc2lkIjtzOjg6IjI1LzcvNTQ0IjtzOjQ6ImF0b20iO3M6MjA6Ii9qYWJmcC8zMS8xLzExMy5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 

26. 26.Tsang C, Majeed A, Aylin P. Routinely recorded patient safety events in primary care: A literature review. Fam Pract 2012;29:8–15.
    
    [CrossRef](http://www.jabfm.org/lookup/external-ref?access_num=10.1093/fampra/cmr050&link_type=DOI) 
    
    [PubMed](http://www.jabfm.org/lookup/external-ref?access_num=21878467&link_type=MED&atom=%2Fjabfp%2F31%2F1%2F113.atom) 
    
    [Web of Science](http://www.jabfm.org/lookup/external-ref?access_num=000299416400003&link_type=ISI) 

27. 27.Moher D, Liberati A, Tetzlaff J, Altman DG. Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement. Int J Surg 2010;8:336–41.
    
    [CrossRef](http://www.jabfm.org/lookup/external-ref?access_num=10.1016/j.ijsu.2010.02.007&link_type=DOI) 
    
    [PubMed](http://www.jabfm.org/lookup/external-ref?access_num=20171303&link_type=MED&atom=%2Fjabfp%2F31%2F1%2F113.atom) 

28. 28.Jüni P, Witschi A, Bloch R, Egger M. The hazards of scoring the quality of clinical trials for meta-analysis. JAMA 1999;282:1054–60.
    
    [CrossRef](http://www.jabfm.org/lookup/external-ref?access_num=10.1001/jama.282.11.1054&link_type=DOI) 
    
    [PubMed](http://www.jabfm.org/lookup/external-ref?access_num=10493204&link_type=MED&atom=%2Fjabfp%2F31%2F1%2F113.atom) 
    
    [Web of Science](http://www.jabfm.org/lookup/external-ref?access_num=000082512300029&link_type=ISI) 

29. 29.National Heart, Lung, and Blood Institute. Quality assessment tool for observational cohort and cross-sectional studies. March 2014. Available from:[https://www.nhlbi.nih.gov/health-pro/guidelines/in-develop/cardiovascular-risk-reduction/tools/cohort](https://www.nhlbi.nih.gov/health-pro/guidelines/in-develop/cardiovascular-risk-reduction/tools/cohort). Accessed June 17, 2017.
    
    

30. 30.Essential Items for Reporting Diagnostic Accuracy Studies. Stard-Statement.org, The EQUATOR Network, [www.stard-statement.org/](http://www.stard-statement.org/). Accessed 1 Sept. 2017.
    
    

31. 31.Ambulatory care in France: Too many adverse events. Rev Prescrire 2012;21:40–1.
    
    

32. 32.Michel P, Quenon JL, Djihoud MA, Bru Sonnet R. Les événements indésirables liés aux soins extra-hospitaliers : Fréquence et analyse approfondie des causes (etude EVISA). Available from: [http://solidarites-sante.gouv.fr/IMG/pdf/Rapport\_EVISAfinal_17aout09.pdf](http://solidarites-sante.gouv.fr/IMG/pdf/Rapport_EVISAfinal_17aout09.pdf). 2008. Accessed June 17, 2017.
    
    

33. 33.Field TS, Gurwitz JH, Harrold LR, et al. Strategies for detecting adverse drug events among older persons in the ambulatory setting. J Am Med Inform Assoc 2004;11:492–8.
    
    [CrossRef](http://www.jabfm.org/lookup/external-ref?access_num=10.1197/jamia.M1586&link_type=DOI) 
    
    [PubMed](http://www.jabfm.org/lookup/external-ref?access_num=15299000&link_type=MED&atom=%2Fjabfp%2F31%2F1%2F113.atom) 

34. 34.Hibbert P, Williams H. The use of a global trigger tool to inform quality and safety in Australian general practice: A pilot study. Aust Fam Physician 2014;43:723–6.
    
    [PubMed](http://www.jabfm.org/lookup/external-ref?access_num=25286432&link_type=MED&atom=%2Fjabfp%2F31%2F1%2F113.atom) 

35. 35.McKay J, de Wet C, Kelly M, Bowie P. Applying the trigger review method after a brief educational intervention: potential for teaching and improving safety in GP specialty training? BMC Med Educ 2013;13:117.
    
    [CrossRef](http://www.jabfm.org/lookup/external-ref?access_num=10.1186/1472-6920-13-117&link_type=DOI) 
    
    [PubMed](http://www.jabfm.org/lookup/external-ref?access_num=24000946&link_type=MED&atom=%2Fjabfp%2F31%2F1%2F113.atom) 

36. 36.Bigby J, Dunn J, Goldman L, et al. Assessing the preventability of emergency hospital admissions. A method for evaluating the quality of medical care in a primary care facility. Am J Med 1987;83:1031–6.
    
    [CrossRef](http://www.jabfm.org/lookup/external-ref?access_num=10.1016/0002-9343(87)90938-7&link_type=DOI) 
    
    [PubMed](http://www.jabfm.org/lookup/external-ref?access_num=3503572&link_type=MED&atom=%2Fjabfp%2F31%2F1%2F113.atom) 
    
    [Web of Science](http://www.jabfm.org/lookup/external-ref?access_num=A1987L171600004&link_type=ISI) 

37. 37.Brenner S, Detz A, López A, Horton C, Sarkar U. Signal and noise: Applying a laboratory trigger tool to identify adverse drug events among primary care patients. BMJ Qual Saf 2012;21:670–5.
    
    [Abstract/FREE Full Text](http://www.jabfm.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MzoicWhjIjtzOjU6InJlc2lkIjtzOjg6IjIxLzgvNjcwIjtzOjQ6ImF0b20iO3M6MjA6Ii9qYWJmcC8zMS8xLzExMy5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 

38. 38.Gandhi TK, Seger AC, Overhage JM, et al. Outpatient adverse drug events identified by screening electronic health records. J Patient Saf 2010;6:91–6.
    
    [CrossRef](http://www.jabfm.org/lookup/external-ref?access_num=10.1097/PTS.0b013e3181dcae06&link_type=DOI) 
    
    [PubMed](http://www.jabfm.org/lookup/external-ref?access_num=22130350&link_type=MED&atom=%2Fjabfp%2F31%2F1%2F113.atom) 

39. 39.Honigman B, Lee J, Rothschild J, et al. Using computerized data to identify adverse drug events in outpatients. J Am Med Inform Assoc 2001;8:254–66.
    
    [CrossRef](http://www.jabfm.org/lookup/external-ref?access_num=10.1136/jamia.2001.0080254&link_type=DOI) 
    
    [PubMed](http://www.jabfm.org/lookup/external-ref?access_num=11320070&link_type=MED&atom=%2Fjabfp%2F31%2F1%2F113.atom) 

40. 40.Lederer J, Best D. Reduction in anticoagulation-related adverse drug events using a trigger-based methodology. Jt Comm J Qual Patient Saf 2005;31:313–8.
    
    [PubMed](http://www.jabfm.org/lookup/external-ref?access_num=15999959&link_type=MED&atom=%2Fjabfp%2F31%2F1%2F113.atom) 

41. 41.Macnee CL, Penchansky R. Targeting ambulatory care cases for risk management and quality management. Inquiry 1994;31:66–75.
    
    [PubMed](http://www.jabfm.org/lookup/external-ref?access_num=8168910&link_type=MED&atom=%2Fjabfp%2F31%2F1%2F113.atom) 

42. 42.Mathew R, Young Y, Shrestha S. Factors associated with potentially preventable hospitalization among nursing home residents in New York State with chronic kidney disease. J Am Med Dir Assoc 2012;13:337–43.
    
    [CrossRef](http://www.jabfm.org/lookup/external-ref?access_num=10.1016/j.jamda.2011.01.001&link_type=DOI) 
    
    [PubMed](http://www.jabfm.org/lookup/external-ref?access_num=21450241&link_type=MED&atom=%2Fjabfp%2F31%2F1%2F113.atom) 

43. 43.Obreli-Neto PR, Nobili A, de Oliveira Baldoni A, et al. Adverse drug reactions caused by drug–drug interactions in elderly outpatients: A prospective cohort study. Eur J Clin Pharmacol 2012;68:1667–76.
    
    [PubMed](http://www.jabfm.org/lookup/external-ref?access_num=22644345&link_type=MED&atom=%2Fjabfp%2F31%2F1%2F113.atom) 

44. 44.Payne RA, Abel GA, Avery AJ, Mercer SW, Roland MO. Is polypharmacy always hazardous? A retrospective cohort analysis using linked electronic health records from primary and secondary care. Br J Clin Pharm 2014;77:1073–82.
    
    [CrossRef](http://www.jabfm.org/lookup/external-ref?access_num=10.1111/bcp.12292&link_type=DOI) 
    
    [PubMed](http://www.jabfm.org/lookup/external-ref?access_num=24428591&link_type=MED&atom=%2Fjabfp%2F31%2F1%2F113.atom) 

45. 45.Agency for Healthcare Research and Quality. Measurement of patient safety. July 2016. Available from: [https://psnet.ahrq.gov/primers/primer/35/measurement-of-patient-safety](https://psnet.ahrq.gov/primers/primer/35/measurement-of-patient-safety). Accessed June 17, 2017.
    
    

46. 46.1.  Henriksen K, 
    2.  Battles JB, 
    3.  Keyes MA, 
    4.  Grady ML
    
    Pronovost P. Improving the value of patient safety reporting systems. In: Henriksen K, Battles JB, Keyes MA, Grady ML, eds. Advances in patient safety: New directions and alternative approaches (Vol. 1: Assessment). Rockville, MD: Agency for Healthcare Research and Quality; 2008.
    
    

47. 47.Chaho A. To pay or not to pay: Medicare and the preventable adverse event: A rational decision of dangerous philosophical change. J Law Health 2009;22:91–136.
    
    

48. 48.Institute of Medicine. Crossing the quality chasm: A new health system for the 21st century. July 19, 2001. Available from: [http://www.iom.edu/CMS/8089/5432.aspx](http://www.iom.edu/CMS/8089/5432.aspx). Accessed June 17, 2017.
    
    

49. 49.Jha AK, Kuperman GJ, Teich JM, et al. Identifying adverse drug events: development of a computer-based monitor and comparison with chart review and stimulated voluntary report. J Am Med Inform Assoc 1998;5:305–14.
    
    [CrossRef](http://www.jabfm.org/lookup/external-ref?access_num=10.1136/jamia.1998.0050305&link_type=DOI) 
    
    [PubMed](http://www.jabfm.org/lookup/external-ref?access_num=9609500&link_type=MED&atom=%2Fjabfp%2F31%2F1%2F113.atom) 

50. 50.Van Doormaal JE, Mol PGM, Van Den Bemt PMLA, et al. Reliability of the assessment of preventable adverse drug events in daily clinical practice. Drug Saf 2008;17:645–54.
    
    

51. 51.Michel P, Quenon JL, de Sarasqueta AM, Scemama O. Comparison of three methods for estimating rates of adverse events and rates of preventable adverse events in acute care hospitals. Br Med J 2004;328:199–204.
    
    [Abstract/FREE Full Text](http://www.jabfm.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiYm1qIjtzOjU6InJlc2lkIjtzOjEyOiIzMjgvNzQzMy8xOTkiO3M6NDoiYXRvbSI7czoyMDoiL2phYmZwLzMxLzEvMTEzLmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

52. 52.Rafter N, Hickey A, Condell S, et al. Adverse events in healthcare: Learning from mistakes. QJM 2015;108:273–7.
    
    [CrossRef](http://www.jabfm.org/lookup/external-ref?access_num=10.1093/qjmed/hcu145&link_type=DOI) 
    
    [PubMed](http://www.jabfm.org/lookup/external-ref?access_num=25078411&link_type=MED&atom=%2Fjabfp%2F31%2F1%2F113.atom)