Abstract
Objective: We propose a method of identifying clinical topics for campaigns like Choosing Wisely.
Methods: In the context of an ongoing continuing medication education program, we analyzed ratings on every patient-oriented evidence that matters (POEM) synopsis delivered in 2012 and 2013. Given the objective of the Choosing Wisely campaign, we focused this analysis on 1 specific item in the validated questionnaire used by physicians to rate POEMs. This questionnaire item is about “avoiding an unnecessary diagnostic test or treatment.” For each POEM, we calculated frequencies and proportions for this item, then we identified the 20 POEMs that were most commonly associated with this item in 2012 and 2013. Finally, we determined whether the clinical topic of each of these POEMs was mentioned in the Choosing Wisely master list.
Results: In 2012 and 2013 we received 506,809 completed questionnaires (or ratings) linked to 530 POEMs, for an average of 956 ratings per POEM. In 59% of these POEMs (n = 312), the most commonly expected type of health benefit was “avoiding an unnecessary diagnostic test or treatment.” We then identified the top 20 POEMs most commonly associated with this item in each year by ranking all 312 POEMs from the top down. The clinical topic addressed by 29 of these 40 POEMs was not addressed in the Choosing Wisely master list. These topics fell into 3 categories: diagnostic tests, medical interventions, and surgical interventions.
Conclusion: “Big data” can identify clinical topics relevant to campaigns such as Choosing Wisely. This process represents a new way to inform the expert panel approach.
In the United States and Canada, the Choosing Wisely campaigns seek to engage physicians and their patients in a conversation about diagnostic tests and procedures. Since 2012, multiple organizations and specialty societies on both sides of the border have joined these campaigns. From a societal perspective, Choosing Wisely fits into the context of a need to address the rising costs of health care and improve quality. In a master list of “things physicians and patients should question,” specialty societies—from allergy to vascular medicine—offer hundreds of recommendations for practice.1
To meet their goals, organizations participating in the Choosing Wisely campaigns produce lists of tests, procedures, or treatments that may not be necessary because evidence (1) is not available to demonstrate their worth or (2) shows that the harms outweigh the benefits. Each specialty society uses expert panels to identify topics for inclusion on their list.
In this article, we show how “big data” from thousands of physicians subscribing to an alerting service can help to identify candidate clinical topics for specialty societies in campaigns such as Choosing Wisely. Alerting services raise awareness of new research findings by delivering abstracts or synopses of studies to physicians on a scheduled basis.2 In so doing, alerting services help fill an essential need for lifelong learning. We suggest that topic selection by specialty societies in the Choosing Wisely campaign could be informed by a novel and systematic “bottom-up” process involving thousands of physicians who read synopses in an ongoing continuing medical education (CME) program. For clinicians, the list of topics so identified can serve as a reminder of research findings that can be used to improve practice when applied to patient care.
Methods
Synopses are succinct descriptions of recently published research, including systematic reviews. Since 2005, physician members of the Canadian Medical Association (CMA) can receive by E-mail on weekdays 1 synopsis of clinical research, called a Daily POEM (patient-oriented evidence that matters). POEMs are selected by searching the table of contents of 102 journals for original research or systematic reviews that present new, relevant information. Relevance is determined using the following questions (all criteria must be satisfied):
Did the authors study an outcome that patients would care about? Studies whose results require extrapolation to outcomes that truly matter to patients are not included.
Is the problem studied one that is common to primary care, and is the intervention feasible? Only information that can be implemented in primary care practice is reviewed.
Will the information, if true, require a change in current practice? Information that confirms existing standards of practice is generally not reviewed.3,4
Following this screening step, identified articles are critically appraised for validity using criteria developed by the Evidence-Based Medicine Working Group; these criteria are updated to include new issues related to study quality.5
Although concise like an abstract, a POEM synopsis differs from the abstract of its corresponding research article in format and content. A POEM begins with a clinical question that places the research question into a clinical context. A “bottom line” statement then summarizes the findings of the article and is designed to help clinicians understand how to apply the results. A synopsis provides a brief overview of the study design and results. Unlike the abstract of the research article, the synopsis presents the study design and results to demonstrate that an evaluation of study validity has been performed by the writer. All Daily POEMs are labeled with a level of evidence from the Oxford Centre for Evidence-based Medicine, a description of study design and financial support. The POEM also provides the article citation and a link to the PubMed entry.6
The ongoing POEMs CME program was accredited in 2006. In the context of this program, physician members of the CMA earn a mini-credit for reflecting on each POEM synopsis they read.7 Mini-credits are awarded by the College of Family Physicians of Canada (0.1 Mainpro-M1), as well as the Royal College of Physicians and Surgeons of Canada (0.25 Maincert Section 2). Accreditation is based on meeting the objective of reflective learning and is documented by the completion of a brief questionnaire for each POEM that was read. This documentation is provided through the Information Assessment Method (IAM; http://www.mcgill.ca/iam). The IAM questionnaire has been iteratively refined since 2001 through publicly funded systematic reviews of the literature and qualitative, quantitative, and mixed methods research studies. The IAM questionnaire is unique and its content is validated; it is available in English, French, Spanish, and Portuguese.
Theoretical Framework
The IAM questionnaire operationalizes a model called ACA-LO (Acquisition—Cognition–Application → Levels of Outcome).8 The ACA-LO model extends a previous model of human–information interaction for research on the value of information,9 and it explicates this notion of “value” from the user's perspective (Figure 1). For example, in the context of E-mail alerts, clinicians receive a passage of text (acquisition) that they read and understand (cognition). They may subsequently use this newly understood information for patient care (application). The model then conceptualizes the health benefits physicians can expect to observe if the clinical information is applied to the care of a specific patient(s). Health benefits for the patient are addressed by the following question: For this patient, do you expect any health benefits as a result of applying this information? Following an answer of “yes,” the branching logic of the IAM questionnaire further conceptualizes health benefits in 3 items: (1) this information will help to improve this patient's health status, functioning, or resilience (ie, the ability to adapt to significant life stressors); (2) this information will help to prevent a disease or worsening of disease for this patient; and (3) this information will help to avoid unnecessary treatment, diagnostic procedures, preventive interventions, or a referral for this patient. Thus, when linked to one “object” of clinical information such as a POEM, the IAM provides a brief validated questionnaire to obtain feedback from the reader.
The Acquisition–Cognition–Application → Levels of Outcome (ACA-LO) theoretical model. The ACA-LO theoretical model explains the value of information, that is, how information is valuable from the information users' viewpoint. In this model, 4 “levels of outcomes” (LOs)—situational relevance, cognitive impact, use of information, and subsequent health benefits—are associated with the iterative “acquisition–cognition—application” process. The ACA-LO model is operationalized by the Information Assessment Method (IAM) questionnaire.
Data Collection
In the CME program we have continuously collected ratings of POEMs from participants since 2006. For example, in 2012 about 15% of the 20,375 CMA members receiving POEMs (n = 3056) submitted at least 1 POEM rating. This group included 2343 participants who described themselves as family physicians or general practitioners, as well as 713 participants from 31 other specialties and subspecialties.
Data Analysis
We included ratings of all POEMs delivered in 2012 and 2013, as received by the CMA from January 1, 2012, to December 31, 2013. Using descriptive statistics, we analyzed all ratings and tabulated the frequency of responses to each item on the IAM questionnaire. Because participants in the CME program are not obliged to rate each POEM, the total number of ratings received for each POEM differed.
Following the logic of the IAM questionnaire, a POEM synopsis must first be clinically relevant and then used for a specific patient before any health benefit can be expected. With regard to the health benefits expected by physician participants for their patients, we calculated frequencies and proportions for each of the 3 health benefit items for each POEM. We then identified the 20 POEMs (in each year) with the highest proportion of ratings of item 3 (“this information will help to avoid unnecessary treatment, diagnostic procedures, preventive interventions, or a referral for this patient”). We chose to focus on this item because of its direct link to the objective of the Choosing Wisely campaign, namely, reducing overdiagnosis or overtreatment. One of us (RG) then searched the master list of topics from the Choosing Wisely campaign (as of March 5, 2014) to determine whether the clinical topic addressed by each POEM was included in that list. POEM topics not included in this master list then were grouped by the same author (RG) into categories.
Results
In 2012 and 2013 we received from CMA members 506,809 ratings linked to 530 unique POEMs, for an average of 956 ratings per POEM. In the majority of these POEMs (n = 312; 58.9%), the most commonly expected type of health benefit was avoiding an unnecessary diagnostic test or treatment.
The clinical topic addressed by 11 of our top 40 ranked POEMs was discussed in the master list of the Choosing Wisely campaign. These 11 POEMs addressed the following topics: screening for prostate cancer, osteoporosis, home glucose monitoring, control of type 2 diabetes, treatment of acute bronchitis in children, and otitis media in children.
Among the topics covered in our top 40 POEMs, 29 were not discussed in the master list of the Choosing Wisely campaign. We present in Table 1 the title of each of these POEMs, as well as the test or treatment to consider for de-adoption. Following the evidence presented in the POEM, topics for consideration of de-adoption/discontinuation in clinical practice fell into 3 categories: (1) diagnostic tests (n = 7), (2) medical interventions (n = 19), and (3) surgical interventions (n = 3). As an example, we highlight one of these POEMs in Table 1, describing an important systematic review that found support for restricted use of antibiotics in the treatment of acute bronchitis in otherwise healthy adults.10
Discussion
A process providing structured feedback on valid research findings can identify candidate clinical topics for specialty societies that issue recommendations in campaigns such as Choosing Wisely.11 To inform the expert panel approach from specialty societies, we suggest our process provides an alternative source of topics. At present, national societies freely determine the process they use to create a list of recommendations for their specialty, in accordance with the following principles:
The development process is thoroughly documented and publicly available.
Each recommendation is within the specialty's scope of practice.
Tests, treatments, or procedures included are those that (1) are frequently used and (2) may expose patients to harm or stress.
Each recommendation is supported by evidence.
A focus on avoiding unnecessary tests, treatments, or referrals is needed to begin to address overdiagnosis and overtreatment because these issues can threaten patient well-being and the sustainability of health systems.12 In the United States, many physicians recognize this issue. For example, in a national survey, 42% of primary care physicians believed patients within their own practice received unnecessary medical care.13 The problems associated with overdiagnosis and overtreatment helped to launch Choosing Wisely campaigns in both the United States and Canada. At the level of physician behavior, a focus on de-adopting or discontinuing clinical actions starts with recognizing the importance of negative study findings that identify spurious interventions. Physicians committed to principles of evidence-based medicine and professionalism will recognize the importance of such a focus with respect to their clinical practice.
Identifying topics to consider for discontinuation in clinical practice is a novel way to leverage the collection of “big data” on POEMs distributed in a national CME program. It is also novel in terms of involving thousands of CMA members in topic selection. To our knowledge, this process has not yet been addressed in the literature.14 We acknowledge that this process does present several challenges or limitations. First, our CME program is not meant to be a survey or an observational study of physicians. As volunteer participants in a CME program, CMA members who rate POEMs are not representative of the population as a whole. Nevertheless, POEMs are reviewed and rated by thousands of physicians. A second limitation of this work is related to the process of selecting research articles for the creation of POEMs. A primary research study or systematic review that never became a POEM would not be rated in the ongoing CME program. The clinical topics addressed by such research would therefore be unidentified through the process we describe. However, the POEM selection process, which targets valid and relevant new articles, identifies many practice-changing research findings. The extent of any selection bias in the identification of “POEM-worthy” articles is unknown and therefore is a subject for research. Future studies could also compare new topics identified by the Choosing Wisely campaign against those identified through the POEM CME program. Given that POEMs represent synopses of newly reported research findings, it is possible that topics identified through the POEM CME program would be more “leading edge” than those identified by expert panels.
Conclusion
The analysis of physician ratings of POEMs in a CME program reveals the potential to identify candidate clinical topics relevant for campaigns such as Choosing Wisely. This novel process can provide an alternative source of topics to inform the typical expert panel approach. The topics we identified can also be used to remind clinicians of actions they can consider de-adopting from routine practice.
Notes
This article was externally peer reviewed.
Funding: Financial support for this article was received from Practice Solutions (a subsidiary of the Canadian Medical Association), “Evaluating the effect of information technology on medical practice” project (nominated principal investigator, RG; co-principal investigator, PP).
Conflict of interest: RG and PP codeveloped the Information Assessment Method. McGill University receives funds from commercial license agreements for this method. DS and AS are paid consultants for John Wiley & Sons, associate editors for Essential Evidence Plus, and regular contributors and authors for the Daily POEMs.
- Received for publication August 7, 2014.
- Revision received October 13, 2014.
- Accepted for publication October 20, 2014.
In this issue
Jump to section
Related Articles
Cited By...
- Physician Perspectives on Mammography Screening for Average-Risk Women: "Like a Double-Edged Sword"
- Risk of Inflammatory Bowel Disease in Psoriasis Patients Treated with Anti-Interleukin-17 Agents: A Bayesian Metaanalysis
- Rethinking screening during and after COVID-19: Should things ever be the same again?
- Choosing Wisely: assessment of current US top five list recommendations trustworthiness using a pragmatic approach
- Data Supporting Key Components of Family Medicine--Scope, Continuity, Interprofessional Care, and More