The Need to Systematically Evaluate Clinical Practice Guidelines

The role of clinical practice guidelines has evolved over time. Early guidelines gathered dust on physicians' shelves; now many clinicians use them to guide their medical decisions. However, many feel buffeted by guidelines recommending different approaches to practice.¹ They see guidelines as cudgels used by insurance companies and other payers to incentivize or deter certain practices. Lawyers point to them in courtrooms. Practice guidelines, therefore, may shape medical practice and affect patient outcomes in many ways. Usually untested before or after implementation, guidelines can lead to misuse or overuse of medical services, resulting in harm. It is time to develop a process to ensure their safety and efficacy.

Discrepancy among practice guidelines has been documented² and is well known by clinicians trying to determine the best treatment goals, such as for patients with type 2 diabetes (should the HbA_1c goal be <6.5%?³ 7% to 8%?⁴), or whom to screen for prostate cancer^5,6 or breast cancer.^7,8 For example, at 1 of our institutions, patients receive 2 letters following a normal mammogram: their family medicine physician tells them to follow-up in 2 years, based on guidelines from the US Preventive Services Task Force,⁷ whereas the radiologist performing the screening recommends returning in 1 year, based on American College of Radiology⁸ guidance. What are patients to think?

These guidelines are based on a synthesis and analysis of the same available evidence. Can disparate recommendations all provide equivalent results? Evidence is limited (Table 1) but suggests that some guidelines are more trustworthy than others.

Faith in the effectiveness of guidelines hinges on the premise that benefit and risks identified in controlled clinical trials will automatically translate to similar benefits and risks in the general population. This premise ignores the interpretive turn that occurs as guideline developers formulate a recommendation from the available clinical evidence, for developing guidelines is not solely a scientific endeavor. In moving from the evidence to a specific recommendation, guideline developers must weigh the benefit of an intervention against the likelihood of harm. With clear and explicit methods, and a multidisciplinary guideline development group, the biases that may result from this interpretive turn are minimized, but not eliminated. Thus there is an inevitable aspect of guideline development that makes it subject to value judgments and can be unconsciously colored by intellectual, professional, or financial conflicts of interest.¹⁵

This coloring can produce recommendations that result in misuse or overuse of medical care, particularly when guidelines are produced by speciality societies. As Quanstrum and Hayward¹⁵ noted, “Although it is true that individual medical providers care deeply about their patients, the guild of health care professionals—including their specialty societies—has a primary responsibility to promote its members' interests … it is a fool's dream to expect the guild of any service industry to harness its self-interest and to act according to beneficence alone—to compete on true value when the opportunity to inflate perceived value is readily available.” As the saying goes, never ask a barber if you need a haircut.

Clinical practice guidelines are produced by people—usually well-intentioned people who give their time and energy for the good of society. However, they are subject to the same cognitive biases as all humans.¹⁶ These biases, which are typically not recognized by the person, can result in decisions colored by tunnel vision (job conditioning), “seeing what you want to see” (confirmation bias), decisions limited to the tools at hand (Maslow's hammer), or other inclinations that can affect judgment. These biases can be influenced by financial, professional, and intellectual conflicts of interest.¹⁷

There are >6000 guidelines from 96 groups in 76 countries in the Guidelines International Network database alone, suggesting that there is a “guideline industry” that is similar to that seen the early years of drug commerce. Medicines were initially unregulated until the Pure Food and Drug Act of 1906 established rudimentary standards for drug safety. Yet medicines did not have to demonstrate effectiveness until 1962, and “good manufacturing practices” were not required until even more recently.¹⁸ Regulation of guidelines should not proceed at such a leisurely pace.

Although all guidelines have limitations in applicability (eg, due to the variety of clinical presentations encountered by clinicians), some important lessons can be learned by looking at the standards that have been developed by well-respected guideline producers. For example, in the United Kingdom, the National Institute for Health and Care Excellence has set up rigorous processes to minimize bias and to regulate the guideline development process.¹⁹ In the United States, several organizations such as the Institute of Medicine,²⁰ the US Preventive Services Task Force, and the Guideline International Network²¹ have developed standards for the production of practice guidelines. However, guideline producers infrequently follow these “good manufacturing practice” equivalents.^22,23 As a result, these efforts have not prevented questionable or harmful guidelines from being disseminated, adopted by clinicians, or used in support of so-called quality measures.

Poor-quality guidelines are unlikely to help patient care, and may actually harm it. But even good-quality guidelines may not improve clinical outcomes since many factors are involved in changing outcomes. In addition, because guidelines are dynamic, they should be updated frequently; the Agency for Healthcare Research and Quality recently required that guidelines be updated at least every 5 years or they would not be included on the National Guideline Clearinghouse website. However, there are no such standardized time criteria for guidelines produced by specialty organizations. Therefore, just as pharmaceutical companies are required to conduct postmarketing surveillance, so too must there be a system to determine the effects of new practice guidelines on patient outcomes. Measures are needed to evaluate the safety and effectiveness of guidelines when they are used in practice.

This call for sound governance—whether it comes from within the profession or from a government agency—will be met with little enthusiasm, especially among groups with a vested interest. For guidelines from groups with direct or indirect support from pharmaceutical companies, the reason seems clear.²⁴ Likewise, insurance companies may be reluctant because they may be denying coverage based on faulty guidelines or supporting care that may be causing more harm than good.²⁵ Professional societies may be too invested in their guidelines to want to subject them to independent scrutiny.^26,27

One way out of this dilemma could be to incorporate an analysis of effectiveness and safety into the guideline production process, as is currently recommended.²⁸ Guidelines would be regarded as preliminary until effectiveness and safety analyses are completed using data from real-world experiences. Communicating the preliminary nature of a guideline might help to discourage its widespread adoption until concrete evidence of good results is available.

Another approach is education, whereby physicians are taught how to identify trustworthy guidelines. Several tools exist,^29,30 and we are working on a new one that is easy to use and relevant to clinicians.³¹

The obvious questions then become, Where would the money come from to do the analysis? Who would be recruited to undertake the evaluation? In Australia, the National Health and Medical Research Council approves guidelines written by other groups.³² In the United States, expanding the role of the Agency for Healthcare Research and Quality to include such oversight might be one possibility. An international approach could be taken, perhaps through expansion of the mission of the Cochrane Collaboration. Others have called for a public-private alliance, with independent panels studying the impact of guidelines, with funding and representation coming from a combination of governments, private foundations, and provider and payer groups.¹⁵

“Postmarketing surveillance” of clinical practice guidelines requires appropriate measurement tools. Practice-based research networks and other research collaboratives can collect outcomes data in practice. Payer databases can be used to identify changes in outcomes. The results of these analyses could be used to refine guidelines.

In the past, other producers of medical interventions—from medicine and device manufacturers to diagnostic tests developers—have opposed calls for oversight. Rigorous evaluation through the regulation of these industries, though, provides protection from harm associated with the misuse and overuse of their products. While more rigorous practice guideline evaluation will most likely be met with similar resistance, it should not follow the same timeline medicine regulation has taken; we need to start now. More evaluation and perhaps even regulation will mean less bias in guidelines, less confusion among physicians attempting to adhere to evidence-based practices, and more benefit for the end users: our patients.

• Received for publication April 6, 2016.
• Revision received June 6, 2016.
• Accepted for publication June 13, 2016.