“The End of the Beginning” for Clinical Simulation in the ABFM Self-Assessment Modules (SAMs)

“Now this is not the end. It is not even the beginning of the end. But it is, perhaps, the end of the beginning.” With these words, Winston Churchill¹ marked the Allied victory of the Second Battle at El Alamein that represented a major turning point in the struggle against the Axis powers in World War II. The American Board of Family Medicine (ABFM) has likewise reached a major turning point in the Maintenance of Certification for Family Physicians (MC-FP) program.

The clinical simulation program began at the ABFM in 1992; the ABFM board intended at the outset to develop the simulation technology as a potential replacement for or enhancement of the MC-FP examination. When ABFM embarked on the MC-FP program in 2004, the board elected to include simulations in the self-assessment process as a means to familiarize diplomates with the interface and functionality in preparation for the appearance of simulations in the Part III examination. Since that deployment, ABFM has delivered >500,000 simulation instances.

Over the past year, ABFM has embarked on several initiatives that have led the board of directors to reconsider the role of simulation in MC-FP. The Data Abstraction and Intelligence Quality Engine for Research and Improvement (DAIQUERI) and Trial of Aggregate Data Exchange for Maintenance of Certification and Raising Quality (TRADEMaRQ) registry projects² will simplify diplomates' participation in MC-FP and potentially provide performance information that might have previously derived from simulations. In addition, ABFM has begun developing a continuous knowledge self-assessment process that involves sending out periodic “mini-quizzes” of 1 to several items (including references and critiques) keyed to the examination content blueprint.³ This process will provide ongoing prospective feedback for diplomates and serve as an alternative option for completing MC-FP self-assessment requirements.

Given these new options, the ABFM board of directors voted at its October 2015 meeting to delink the knowledge assessment component of the MC-FP self-assessment modules from the associated clinical simulations. This means that the knowledge assessments and simulations will now serve as independent options for completing the MC-FP self-assessment requirement.

This new role for simulations provides an opportunity to refocus the simulations' operation and functionality to present a much more formative, rather than summative,⁴ emphasis. Development efforts heretofore have stressed summative scoring models and functionality to support a possible role for simulation in the MC-FP examination. In this summative role, simulation scoring would have necessarily mapped to the existing examination pass/fail threshold, and would not serve easily to motivate and direct higher levels of performance. We can now work to implement formative features such as context-sensitive feedback, quick quizzes, short lectures/discussions, and competitive gaming features that represented inappropriate components for a possible high-stakes examination environment. In this new role scoring can provide feedback on performance, and can motivate higher performance levels. The simulator interface can now evolve to include more guidance (eg, use of dropdown menus and pop-up boxes) than would have been appropriate in the context of using simulation within the examination.

In addition to this more formative emphasis, ABFM, in collaboration with colleagues from Virginia Commonwealth University, recently completed an extensive review of structured and unstructured SAM feedback from the past 10 years' experience (ABFM internal reports).^5,6 That review identified a number of diplomate suggestions for improvements to the simulation interface. During the summer of 2015, these suggestions guided multiple interface revisions that ABFM deployed in August. Subsequent feedback indicated favorable response to the revisions (ABFM internal report).⁷ Work remains on several interface issues, however, including more responsive natural language processing, easier access to diagnostic studies and therapies, and greater use of media resources. The development team met in Lexington, Kentucky, February 23–24, 2016, to begin work on implementing responses to this feedback and the formative features mentioned earlier. The ABFM has also engaged external consultant experts to aid in redesigning the simulation interface.

The October 2015 board action changes the role originally envisioned for the ABFM clinical simulation system. However, this change clearly represents the “end of the beginning”—not the end—of clinical simulation in the ABFM MC-FP program.