Abstract
American Boards of Medical Specialties have emphasized single point in time testing for summative assessment of cognitive expertise necessary for board certification. In 2016, the American Board of Anesthesiology introduced Maintenance of Certification in Anesthesiology (MOCA), a longitudinal assessment platform that provides diplomates formative feedback with continuous questions over time and adapts questions to areas of knowledge weaknesses over time. This paper describes the rationale, history, and early results of the American Board of Anesthesiology MOCA platform.
Introduction
Although the wording of their mission statements may differ, the member boards of the American Board of Medical Specialties aspire to facilitate the over 900,000 certified physicians in the United States to deliver the very best care to their patients. Specifically, the mission of the American Board of Anesthesiology (ABA) is “to advance the highest standards of the practice of anesthesiology,” with the ultimate goal to improve health and enhance outcomes. As leaders of these boards charged to establish systems for assessing and certifying physicians in our specialties, we have a corresponding responsibility to continually improve our own programs and more fully unlock their power to improve patient care. Over the past 20 years, the ABA (like other boards) has moved from issuing lifetime certificates to issuing time-limited certificates valid for 10 years. Throughout the evolution of Maintenance of Certification in Anesthesiology (MOCA), we heard considerable feedback from our diplomates that while they were absolutely committed to improvement in patient care, their need for professional development did not align with our initial approach to MOCA. Accordingly, the ABA has reconsidered the purpose of MOCA and has evolved our programs to better help anesthesiologists in their desire to improve patient outcomes.
Background
As the ABA has envisioned our assessment systems over the past 10 years, we have come to believe that board certification has two distinct functions, which provide value and relevance to the anesthesiologist and their patients over different phases of a physician’s career—training and practice. The training phase culminates in initial certification. During this phase, the ABA uses information from multiple sources to determine whether the anesthesiologist has the requisite medical knowledge, skills, and abilities to serve as a consultant in anesthesiology, based on established competencies. To earn initial certification, the anesthesiologist must successfully complete a series of four rigorous high-stakes assessments, beginning during training and finishing after successful completion of training. These assessments include two written examinations (BASIC and ADVANCED); and two applied examinations, a newly instituted Objective Structured Clinical Examination (OSCE), and our traditional Standardized Oral Examination (SOE). The SOE consists of two 35-minute guided case based discussion with two examiners that assess attributes of the candidate to manage patients, while the OSCE consists of seven 8-minute stations that assess professionalism, communication, and technical skills with either a standardized actor or an examiner in the room. The content outline of the examination maps to the six core competencies that anesthesiologists are expected to achieve by the end of residency training: patient care and procedural skills, medical knowledge, practice-based learning and improvement, interpersonal and communication skills, professionalism, and systems-based practice. In addition to passing these examinations, anesthesiologists must also be deemed ready for independent practice by their residency programs who have continually assessed their clinical abilities. Thus, initial certification decisions are distinct from Maintenance of Certification (MOC) and are based on all four levels of “Miller’s Pyramid” of assessments: know (BASIC and ADVANCED written examinations), know how (SOE), show (OSCE), and do (assessment by local Accreditation Council for Graduate Medical Education program). Published research demonstrates the different examinations indeed assess different competencies, providing evidence for their external validity in terms of physician performance.1⇓⇓⇓⇓–6
Anesthesiologists who successfully complete this rigorous series of assessments during the initial certification process join the community of practicing diplomate anesthesiologists committed to lifelong learning through MOC. As originally formulated by the American Board of Medical Specialties, the basis for MOC is to mitigate the well-described decline in knowledge and application of knowledge as a physician matures their career after residency (Figure 1).7⇓⇓⇓–11
The original purpose of MOC was to ensure that physicians continue to meet a minimum standard, including participating in various learning activities and periodically passing a high-stakes examination. Those not meeting the standard lose their certification. In this formulation, the focus of MOC is to identify those physicians who have not participated in the required learning and improvement in practice activities; maintained satisfactory professional standing; or met the standard for factual knowledge. The assumption is that meeting MOC requirement is a surrogate for clinical competence, with assessment primarily functioning in a summative role to help identify low-performing physicians. However, this focus is problematic for at least two reasons. First, although there is a general relationship between some measures of clinical performance and MOC examination scores, multiple factors contribute to clinical competence, many of which may not be directly measured by MOC processes (ie, Professionalism and Professional Standing, Lifelong Learning and Self-Assessment, Assessment of Knowledge, Judgment, Skills, and Improvement in Medical Practice). Indeed, the ABA has published evidence showing that passing a periodic high-stakes examination has little utility in predicting which physicians have pronounced practice deficiencies that result in disciplinary actions against their medical license.12 Second, the use of assessments and other MOC elements to “weed out” doctors largely squander the golden opportunity to use MOC assessments as formative tools to effect “learning to improve practice” as opposed to “learning to pass an examination.” Significant advances in cognitive science over the last several decades clearly show that a given assessment can function effectively as either an assessment of learning (summative assessment) or an assessment for learning (formative assessment), but assessments that attempt to serve both purposes compromises the effectiveness of each.13⇓⇓⇓⇓–18
The central goal of our continuing certification paradigm is to use a broad range of innovative assessment methods to support our diplomates to regularly evaluate their knowledge, judgment, and skills and then to connect them to relevant professional development opportunities that afford robust and durable learning and lead to greater quality improvement in their practice. While the diplomate is our central focus in this paradigm, the public is the primary beneficiary. Instead of “overweighting” a summative assessment within our program we endeavor to make professionalism, lifelong learning, assessment, and practice improvement integrated and coherent for the diplomate with greater impact to the patients for whom they care. The predicate of our paradigm is that using formative assessment tools will increase the knowledge, judgment, and skills of the entire cohort of physicians and reduce the variability of their practice of anesthesiology.
In our view, on initial certification, ABA diplomates become part of a learning community that strives collectively to keep their knowledge current and to continually improve their practices. This phase is focused on the ongoing professional development of the diplomate across the spectrum of their career. While the emphasis is on professional development of an engaged diplomate, there are a number of diplomates each year who fail to meet the standard and are subject to an adverse consequential decision regarding board certification. Focusing on professional development transforms the role of the ABA from being a “discriminator” of competence to the specialty to becoming a trusted partner in the professional development of anesthesiologists and improvement of patient care. The focus of MOCA is thus to improve the quality of care for all patients, and consequently shift the “quality curve” of the entire population of practicing diplomate anesthesiologists and to reduce variability (Figure 2). To fulfill this purpose, we have recognized that we need to build a different relationship with diplomates such that we are viewed as trusted partners in their professional development. We strive to build our MOCA program on the fundamental principles about what makes effective learning for adults. These principles include creating learning experiences that are personalized, self directed, and relevant to their individual needs. Moreover, we must recognize that adult learners are internally motivated and bring a richness of experience that must be acknowledged and respected. Participation in this learning community helps assure the public that board-certified physicians are committed to providing high-quality care throughout their careers.
How We’ve Evolved to Date
The ABA has 12 physician directors, each of whom serve up to three 4-year terms. In 2011 we added a public member who may serve up to two 3-year terms. Thus, within a 12-year period, the entire board is replenished. Historically, we’ve intentionally had a “surge” in our sequence of replacing directors. This “surge” ensures against group-think and organizational rigidity, facilitating innovation and a redesign of our strategies and tactics. For instance, the most recent “surge” began in 2008 where 10 of our current 13 directors were onboarded in a 5-year period.
This “surge” has helped the ABA take the long-term view as it evolves its MOCA program. In 2011, the ABA reimagined our original MOCA program by crafting a vision for using technology to foster a culture of lifelong learning. We consulted with leaders in anesthesiology, education, and technology to conceptualize the future program. The vision for the program’s redesign was derived from surveys of diplomates, forums at national meetings, focus groups, and a volunteer MOCA Users’ Group of board-certified anesthesiologists across the career and subspecialty spectrum. The ABA has also stayed abreast of how trends in technology and health care practice may be leveraged to improve the program and add value to diplomates’ practice. Central to this effort has been the process of engaging our diplomates and experts in technology, medical education, and learning science to more fully understand how to harness innovative approaches to create a more relevant, appealing and diplomate-centered program. A program in which the overarching objective is to ensure that anesthesiologists have the knowledge necessary to provide high-quality anesthesia care.
MOCA 2.0® launched in 2016 with MOCA Minute®, our continuous longitudinal assessment for learning, at its core. In short, diplomates access 30 questions each quarter from a computer, tablet, or smartphone. They have 60 seconds to answer a question. Immediately after answering, the diplomate is asked to rate their level of confidence about whether their answer is correct before seeing a screen with the correct answer, key points, and references for learning (Figure 3).19 This confidence rating provides data for the MOCA Minute algorithm to identify areas of knowledge where a diplomate may be confidently wrong and prioritizes these areas for reinforcement through the spaced repetition of subsequent questions on these topics. Our objective is to enhance learning, and reduce the burden that some diplomates told us MOCA caused. This is achieved by integrating the program’s components and delivering assessment for learning in a manner convenient to the diplomate and adaptive to each diplomate’s individual learning needs. MOCA Minute® is not only adaptive to a diplomate’s area(s) of knowledge improvement with direct linkage to related learning opportunities and activities but MOCA Minute® can rapidly adapt to focus learning on new knowledge and national health care priorities. That is, it provides the opportunity to identify in “real time” any areas of knowledge that should be prioritized for attention across the specialty and to guide the development of educational opportunities. As critical new knowledge emerges, MOCA Minute® facilitates the rapid dissemination of this information to all diplomates by pushing out questions that address these topics. It also allows us to track the percentage of diplomates that answer the question correctly on first attempt, as well as those who require a second or third attempt when they have previously answered incorrectly. For example, a novel agent that rapidly reverses neuromuscular blockade was recently approved for use in the United States. With a goal of disseminating critical clinical information about this novel agent, several MOCA Minute® questions were developed. The percentage of diplomates answering correctly was initially low, but increased rapidly as diplomates were directed to learning resources. For one question, the number of diplomates who answered the question correctly on the second attempt nearly doubled from the first attempt, indicated that rapid learning had occurred. These aggregate performances are summarized across a relevant cohort of diplomates and shared with specialty societies to help them identify where to focus educational products that can support physicians’ learning related to these topics. Finally, the utility of aligning MOCA Minute® to national health care priorities is exemplified by rapidly including questions on Zika virus, the opioid epidemic, and COVID-19.
The design of the ABA’s MOCA Minute® and other similar longitudinal assessments leverage what we now know about important cognitive triggers that support learning and memory, including desirable difficulty, spaced repetition, priming, interleaving, metacognition, and personalization.20 We also revised our requirements for Improvement in Medical Practice activities to better allow diplomates to be recognized for how they contribute to such activities in their real day to day setting and practice.
It is important to point out the distinction that the ABA makes between a summative assessment and a summative decision. Summative assessments are in general high-stakes, performance-based examinations that measure knowledge at the conclusion of or at some important milepost of an activity (assessment of learning). In contrast, formative assessments are used throughout an activity, in a low-stakes manner, and are meant to optimize improvement of the learner (assessment for learning). The ABA believes patients are best served by continuous improvement of the learner (diplomate). While the emphasis is on professional development of an engaged diplomate, there are some diplomates each year who fail to meet the standard, and hence a consequential (summative) decision is made regarding board certification based on multiple sources of information.
What Is Next?
As MOCA continues to evolve, we expect to better integrate the program’s components and provide practical feedback to diplomates to help them improve their practice. In early 2019, an Innovation Summit, which included expert educators and a wide range of stakeholders, helped inform and guide future plans for MOC. We also continued our “Users Group” of diplomates participating in MOCA to further guide the development of MOCA 3.0.
Building on the Innovation Summit and Users’ Group feedback, the ABA convened a Learning Theory Workshop in July 2019 to discuss the broader purpose of MOCA and establish guiding principles for implementing some of the innovations discussed during the Innovation Summit. We affirmed that our goal is not simply to “test” diplomates, but to facilitate continuous learning to enhance patient care and improve public health. We discussed the core competencies we want to assess and improve on, as well as prioritized the various learning modalities as we evolve MOCA. Assessment and improvement methods ideally should be more practice-relevant and better integrated into the anesthesiologist’s daily care of patients. We wanted to ensure that we were well grounded in the overarching goal of primacy of our patients as we advanced our programs.
Our community of anesthesiologists, steered by our MOCA 3.0 Users’ Group, will continue to guide the development of MOCA. We will launch the first phase of MOCA 3.0 in 2020 with a redesigned physician’s portal and a new ABA mobile application, which will provide the technical infrastructure we need for future phases of the program. We expect future development to continue to rely heavily on mobile technologies that allow us to meet diplomates where they are professionally, allowing them to chart their course toward personalized learning that positively impacts patient outcomes. We understand we must balance working expeditiously to satisfy the diplomates’ desires for the most relevant program with judiciously ensuring that continuous certification meaningfully contributes to helping anesthesiologists provide the best care possible.
MOCA Minute® has been well received as an assessment of medical knowledge. For example, in a 2017 diplomate survey, 89% of respondents who had taken the former recertification examination and had participated in MOCA Minute® preferred the latter. However, there are opportunities for further improvement in our approach. The current single-best-answer multiple-choice questions may work well for assessing knowledge and some types of clinical judgment, but may not be suited to assess other competencies crucial to success such as decision making, clinical reasoning, communication, professionalism, and practice improvement. Other formats are being actively explored to supplement the current assessment, such as adaptive, case-based testing using clinical scenarios directly relevant to diplomate practice which could provide an opportunity to pause and reflect on the answer with a link to learning. We are also evaluating how we can use technology (eg, artificial intelligence) to capture what and how anesthesiologists are learning on the job rather than making them report it. This approach would integrate learning and practice improvement into the anesthesiologist’s daily work flow, adding value and reducing burden.
We believe this vision of MOC, or whatever the branding term may become, is well aligned with our mission of encouraging the highest standards of the practice of anesthesiology for the primary benefit of our patients. Longitudinal assessment for learning can close knowledge gaps on emerging practices and standards in our specialty with greater speed and efficiency. It can intersect in real time with emerging national health priorities such as the opioid epidemic and perioperative brain health. Working with the diplomates of the ABA, we aspire to significantly advance the quality and value that our diplomates provide to the public, thus improving patient safety and health—the ultimate goal of all physicians. We conclude with the words of Benjamin Franklin:
Tell me and I forget, teach me and I may remember, involve me and I learn.
Notes
This article was externally peer reviewed.
Disclosures: AEH is Chief Assessment Officer of the American Board of Anesthesiology. DOW and DJC serve as a Directors for the American Board of Anesthesiology.
Funding: None.
Conflict of interest: None.
- Received for publication October 29, 2019.
- Revision received April 10, 2020.
- Accepted for publication April 27, 2020.