Abstract
Background: The member boards of the American Board of Medical Specialties have agreed to expand the scope of certification to include assessment of medical knowledge, practice-based learning and improvement, patient care, interpersonal and communication skills, systems-based practice, and professionalism. Multiple-choice examinations provide limited ability to assess these dimensions.
Methods: The American Board of Family Practice (ABFP) has developed a computer simulation system to facilitate more comprehensive candidate evaluation. The system consists of a knowledge base, a simulation program to create patient scenarios, an interface for presenting simulations to users, and an administrative database to track candidate performance and interactions with the system. The system uses population distributions for disease states to produce cases and evolves patients in response to candidate interventions, such as pharmacological and nonpharmacological therapies. We use Bayesian belief networks to model patient characteristics and comorbid condition interactions.
Results: Simulations have been created for 7 disease states; ultimately simulations will be available for 25 to 30 disease states. Initial testing will take place in regional examination centers but will ultimately use the Internet for convenient access for certification and recertification candidates.
Conclusion: The ABFP will begin field-testing the system in early 2003 and will include simulations in the certification and recertification examination process in 2004.
The medical specialty boards have recently expanded their perspective on physician performance to include six components: medical knowledge, practice-based learning and improvement, patient care, interpersonal and communication skills, systems-based practice, and professionalism1 (detailed descriptions of these components can be found at http://www.acgme.org/outcome/comp/compFull.asp.) Traditional formats such as multiple choice examinations focus on knowledge assessment. The additional dimensions, however, will require alternative techniques, such as standardized patients, chart audit, objective standardized clinical examinations, and simulated patient cases.2
The large number of candidates the American Board of Family Practice (ABFP) examines annually renders standardized patients and objective standardized clinical examinations logistically impractical. Limited chart audits are currently included in the comprehensive office record review, conducted as part of the recertification process. Various groups have experimented with clinical simulation since the 1970s. Barnett and colleagues3 ,4 developed case simulations at the Massachusetts General Hospital, and developers at the University of Illinois created the CASE (Computer Associated Simulation of the Clinical Encounter) at approximately the same time.5 The American Board of Internal Medicine experimented with this system for possible use in physician certification and found this approach to be feasible.6 Another system designed at the University of Wisconsin served as the basis for the National Board of Medical Examiners Step III licensure examination,7 ,8 and their experience provided further validation for the role of simulation in professional assessment.9
These systems use authored case descriptions that, once used, are no longer secure, necessitating a continual process of new case generation. The ABFP recognized in the early 1990s that simulation could improve their ability to make certification decisions but desired to avoid shortcomings associated with previous efforts. The Board established a development team that included computer scientists, mathematicians, psychometricians, and family physician informaticists. After examining existing architectures, the team adopted dynamic simulation,10 driven from a knowledge base, as the means to create unique patient care scenarios appropriate to the family practice domain. This architecture creates unlimited similar, but unique, case scenarios for use in the certification process. The development team described the model architecture in earlier reports,11 ,12 and the ABFP has patented the technology (US patent #6,246,975).
Several assumptions motivated the development effort:
The simulator should generate unique scenarios for each candidate, thus minimizing security considerations associated with the current paper examination.
The simulator should create scenarios from a knowledge base defined by family physicians and appropriate domain experts.
The scenarios should be realistic and evolve patients’ health states prospectively with time and in response to candidate interventions.
The presentation format should provide a friendly interface that depends as little as possible on candidate computer skills.
The system architecture must provide structures suitable for a valid and reproducible scoring process.
Methods
The developers have described the architecture in detail elsewhere.11–16 The simulator currently includes four fundamental pieces: a knowledge base, a simulation engine or program, the presentation interface, and an administrative database (Figure 1). The system uses population distributions for disease states to produce cases and evolves patients in response to candidate interventions. Bayesian belief networks model patient characteristics and comorbid condition interactions.
Schematic of system architecture for case simulation. The four main components consist of the knowledge base, the simulation engine, the presentation system, and the administrative database. The system also uses a third party package (Netica, Norsys Systems) for belief network analysis and manipulation (a fully functioning demonstration version of the software is available at www.norsys.com.) The simulator communicates with the presentation system through the communication module. The presentation system can interface with the outside world by means of the Internet or within a local intranet structure, such as at a test center.
Knowledge Base Development
The knowledge base consists of a software program designed to support entry and maintenance of information (an object-oriented database17 developed for the project) and the actual information needed to support simulation. The development team has created the knowledge base using knowledge development teams. Each knowledge development team consists of several family physicians (at least one of whom must be a current or recent member of the ABFP Board of Directors), a domain expert (such as a specialist or family physician with special expertise in the particular domain), and a knowledge engineer (a family physician trained in or having expertise in medical informatics18 and database design.) The knowledge teams define the content for the simulation and select key references (with particular attention to existing evidence-based guidelines) and concepts to include in the knowledge base.
Each knowledge team undergoes an initial orientation to the simulation architecture, including introduction to Bayesian belief network concepts and an overview of the simulation methodology. The team then constructs the parallel and belief network structures needed for simulation. Additionally, the knowledge teams will periodically review knowledge content for updates and revisions.
The development team has accomplished extensive knowledge instantiation for hypothyroidism, essential hypertension, otitis media, duodenal ulcer disease, major depression, reactive airways disease, and abnormal Papanicolaou smears. Comorbidities, such as lipid disorders, renovascular disease, and allergic disorders, have also been added for support of the primary domain simulations. The Board ultimately intends to create simulations for 25 to 30 disease states; external consultants have advised the development team that this number will provide sufficient breadth to support certification decisions (J. Veloski, personal communication, 1995.)
The Simulator
The simulator consists of a computer program that uses knowledge base information to create credible patient cases. The testing paradigm consists of physician queries and interventions and simulator responses. These cases can evolve as a result of elapsed time or in response to candidate interventions. Bayesian belief networks19 and supporting knowledge base structures interpret, integrate, and update information about a particular patient. As the candidate conducts queries about patient symptoms, physical findings, laboratory results, and clinical test results, the simulator uses belief networks to formulate context-appropriate responses. Similarly, when the candidate orders a particular treatment or therapy, belief networks update the patient’s current status. The simulator program also keeps track of time, and the patient evolves as time passes and in response to the candidate’s therapeutic interventions.
The Interface
The early development efforts coincided with the dominance of the Microsoft DOS operating system and predated the advent of the World Wide Web. The evolution of the Windows operating system, improvements in computer hardware, and availability of de facto standard Internet browser platforms coincided with maturation of the simulation system to enable delivery of patient simulations over the Internet. The World Wide Web provides a means for convenient access to the system for candidate practice and will facilitate the candidate registration process. Standard Web browsers enable access to the system without requiring the installation of dedicated, complex computer software packages. This accessibility will allow candidates to become well acquainted with system function before their certification and recertification examinations.
The system uses the Microsoft Internet Explorer Web browser to provide a familiar-appearing interface for the simulations. The development team has worked with Medical Learning Company (Wellesley, Mass) to create the current interface (the system was demonstrated at the American Academy of Family Physicians Scientific Assembly in San Diego, October, 2002.) The interface presents components found in electronic medical records, but it cannot currently be viewed as a model for such systems. As yet no broadly accepted standards exist for electronic medical records, but we anticipate that such guidance will become available in the near future.20 As standards develop, the interface will evolve to incorporate accepted standard electronic medical record characteristics.
The Administrative Database
The system includes an administrative database to maintain candidate registration information and eligibility for specific examination components. Additionally, this database keeps a record of candidate actions during simulations, thus providing information needed for scoring and for responding to candidate inquiries regarding assessed scores.
Traditional Item Presentation
Family physicians diagnose and manage multiple disease entities21; therefore, the computer-based examination will also include multiple-choice questions similar to those used on the current paper-based test to address less common but important clinical entities for which simulations will not be developed.
The ABFP has developed a multiple-choice examination system that has been used successfully by the American Board of Radiology, the American Board of Pathology, and the American Board of Psychiatry and Neurology for their certification, recertification, and subspecialty examinations (Richard Rovinelli, personal communication, 2002.) The ABFP also used this system to administer the examination for the Certificate of Added Qualification in Sports Medicine in 2002.
The item format mirrors the paper-based examination closely. The questions contain a stem, a question, and possible answers. “Radio buttons” lie adjacent to each question, and the candidate selects the correct answer by clicking the corresponding button with the pointing device (mouse).
Scoring
The dynamic simulation technique poses new challenges for scoring candidate performance. Examinations can be structured so that all candidates start out with identical case presentations. In this context, traditional psychometric models can be used to assess initial candidate actions.22 As the cases evolve, however, individual candidates will experience unique content, and traditional scoring theory might not apply in these circumstances.23
The ABFP has explored several possible alternative approaches. The National Board of Medical Examiners uses expert judgment of possible candidate actions to score USMLE Step 3 simulations.24 This technique has worked for the National Board of Medical Examiners because their simulations represent authored cases that will yield to psychometric models, such as item-response theory.24 The ABFP simulator produces a potentially infinite universe of unique simulation scenarios for which traditional scoring methods do not appear to apply.25 The development team has therefore explored alternative scoring approaches. The most promising techniques derive from information theory26 and decision theory.25–27 The ABFP and the development team are currently conducting research building on analytical constructs described by Downs and Friedman.25 ,27 ,28
Early Results
The work to date has focused on developing and perfecting the simulation algorithms and populating the knowledge base. Knowledge team development efforts have shown that practicing family physicians, irrespective of their familiarity with computer technology, grasp quickly the supporting concepts (such as Bayesian belief network formulations) and structures needed in the knowledge base. Table 1 summarizes knowledge team members’ responses to queries regarding their participation in the knowledge development process.
Number of Team Members Responding to Opinions Regarding Their Participation in Knowledge Development Activities: Results of 7-Point Likert Scale (1 = agree, 7 = disagree) Survey of Knowledge.
A prototype presentation system was submitted to field-testing at a number of residency programs in 2001. Feedback from that experience drove the current Web browser appearance and implementation (internal evaluation documents, Assessment Technologies, Inc, June 2001.)
The ABFP began field-testing the current system in early 2003 and anticipates including simulations in the certification and recertification examinations in late 2003 or 2004. The simulations will ultimately encompass 25 to 30 content areas commonly managed by family physicians. Functioning simulations currently exist for hypertension, hypothyroidism, otitis media, depression, and duodenal ulcer. Knowledge base development for reactive airway disease and abnormal Papanicolaou smears is nearing completion, and work will begin in 2003 on chronic heart failure and ischemic heart disease.
This system represents a substantial change from the traditional ABFP examination format. To become familiar with the system, candidates will have access to the simulator on a secure Web site via the Internet for practice purposes. The actual certification and recertification examinations will occur initially at testing centers throughout the country. Local intranet networks at each center will display traditional multiple-choice test items and patient simulations. This approach enables, indeed necessitates, more flexible timing for individual candidates’ testing sessions. Testing center sites will have limited seating; therefore, the examinations will be administered on multiple dates, providing scheduling options that candidates do not have with the current once-yearly testing format.
Ultimately, a Web-based presentation will allow ABFP certification and recertification candidates to access the process from home, office, or other venues at times convenient to them. This accessibility promises to limit the disruption and cost to candidates’ practices that result from the current use of a limited number of geographically distinct test centers. Moving to a widely distributed process will entail security issues that ABFP has not encountered previously. For example, how will the Board verify that the person who logs in is actually the person sitting at the keyboard? Passwords and biometric methods, such as fingerprint recognition and retinal scanning, can verify the identity of the person accessing the system (a good review of these methods appears at http://www.computer.org/itpro/homepage/jan_feb01/security3.htm) but cannot assure that the same person actually remains at the keyboard to complete the activity. Until this issue can be resolved satisfactorily, the Board will continue to deliver the examination through multiple regional testing centers, but with greater frequency and flexibility than possible with the current paper-based test.
Summary
The family practice certification and recertification examinations have relied on paper-based objective test formats since the inception of the ABFP. This approach produces statistically reliable and reproducible results but unfortunately provides only limited ability to assess a candidate’s abilities beyond cognitive skill. Simulation provides an additional potential probe into decision making and patient management abilities. Whether this method will yield valid and reproducible information distinct from that gained in paper-based examinations represents an as yet unproved hypothesis.25 The ABFP recognizes the need to conduct extensive field-testing of the simulations and scoring method-ologies before using this technology for making high-stakes certification decisions. Nevertheless, computer-administered examinations will promote testing flexibility not available with the current paper-based testing process. Furthermore, although the simulation architecture requires substantial knowledge acquisition and maintenance, and scoring challenges necessitate nontraditional measurement approaches, moving the examination to an electronic format represents a major step toward making certification and recertification a more flexible, prospective, ongoing process for family physicians.
- Received for publication November 14, 2002.
- Revision received November 14, 2002.
