Re: Performance Evaluation of the Generative Pre-Trained Transformer (GPT-4) on the Family Medicine In Training Examination

To the Editor: I am writing in response to the recent article in JABFM by Ting Wang et al. exploring the ability of ChatGPT, a large language model (LLM), to pass the family medicine in training examination.¹ As someone who has also studied this phenomenon and published similar findings in Family Medicine, where our research compared LLMs' proficiency in taking the family medicine in-training examination,² I am compelled to reflect on the implications of these advances.

Board examinations exist fundamentally to uphold public trust in physicians, mostly in physicians' medical knowledge. These rigorous, standardized tests serve as a benchmark, offering assurance to patients, institutions, and colleagues that a physician is competent and merits the responsibility of medical care. As seasoned practitioners, we may come up with our grievances with standardized exams, yet board exams help show the overarching principle: our patients, employers, and professional peers must trust in our expertise and abilities.

Historically, board certification has also provided reassurance to employers seeking to hire qualified, competent doctors. However, the landscape of certification became more nuanced when the American Board of Family Medicine (ABFM) transitioned to a longitudinal assessment model. This change emphasizes a continuous evaluation of medical knowledge through periodic questions, trusting that those who stay engaged are maintaining competence. Herein lies a potential vulnerability: the advent of publicly accessible and increasingly sophisticated LLMs.

These tools, now widely available and impressively accurate, can be leveraged to effortlessly answer board examination questions. Physicians can (with near certainty of passing) use LLMs to complete these assessments. This is not merely an abstract concern but a very real challenge to the integrity of our certification system. Of course, there has always been an implicit honor code—an expectation that physicians will not use extensive references like Harrison's Principles of Internal Medicine or similar resources while testing. However, as I consider the Harrison's textbook on my bookshelf while writing this letter, I recognize that AI models introduce an unprecedented ease and ubiquity of assistance.

For board certification to retain its credibility and fulfill its foundational purpose, I believe we must reevaluate our approach. The original high-stakes examination format, which emphasized unaided knowledge, could perhaps be reinstated. But as a proponent of AI in medicine, I recognize the incredible potential of these technologies to augment our practice and education. Nevertheless, we must draw a firm distinction: if we decide to integrate AI into our assessments, we are setting a precedent for physicians who may become overly reliant on AI and less capable of independent, critical medical decision making. This, to me, is an undesirable outcome that threatens the autonomy and reliability expected of medical professionals.

The board certification questions should evolve to focus on question styles that test medical reasoning and decision making under uncertainty. With scenarios where, even with access to AI, the public still relies on physicians' expertise. The measurement construct of the board certification should align with this goal, ensuring it truly evaluates a physician's ability to navigate complex medical situations independently. In addition, the examination format should explore testing competencies beyond medical knowledge, such as communication and teamwork, through methods like direct observation, simulated patient interactions, or case-based discussions.

Perhaps board certification should evaluate patient cases we’ve already cared for, or maybe an old-fashioned USMLE Step 2 CS-style standardized patient examination? The integration of AI into family medicine³ is inevitable, but our certification processes must carefully weigh the implications. If our goal is to preserve public trust and professional standards, we must ensure that our examinations—and by extension, our physicians—are uncompromising in their demonstration of medical competence.