The American Board of Family Medicine: New Tools to Assist Program Directors and Graduates Achieve Success

Tools for Program Directors

Program directors have been challenged with identifying those residents who lack sufficient medical knowledge, because residents with deficits in that domain can manifest the deficit in many ways, including what may seem to be a lack of judgment or even slowness in clinic or hospital rounds. The In-Training Examination (ITE) has been characterized as a “low-stakes” examination, and program directors have varied in their perspectives regarding how much stock that they should place on the ITE for residents with poor or marginal performance on the examination, despite evidence of the predictive validity of the ITE.⁶ The Bayesian Score Predictor is a wonderful resource that allows program directors to reliably identify residents' future performance on the certification examination and to provide resources to and motivate residents who may be at risk of not passing the certification examination. We believe that moving the certification examination from July to April, coupled with the Family Medicine Certification entry requirements, is particularly useful to ensure that all residents participate in a minimal level of practice improvement and knowledge self-assessment. The new knowledge self-assessment requirement during residency may be an important reason why performance has improved. The fact that many physicians do poorly on their first pass through the knowledge self-assessment questions indicates that, based on objective testing, we often do not know as much as we think we do.

Causality

Puffer et al. are rightly cautious to attribute the variance in the pass rate on the certification examination to individual strategies. Much has changed in the graduate medical education environment over the past decade. As the authors speculate, the “quality of family medicine trainees recruited into training programs” may have been improving during the time frame of their study. Increased numbers of graduates from both US allopathic (MD degrees) and osteopathic (DO degrees) medical schools over the past 15 years create more competition for graduate medical education positions and disproportionately disadvantage international medical graduates (IMGs).⁷ The composition of matriculating family medicine residents has changed over the past decade. While the proportion of US graduates with an MD degree has remained stable at approximately 46% of entering first-year residents, over the past decade the percentage of graduates with a DO degree has increased approximately 1% per year, while the percentage of IMGs has experienced a reciprocal decline.⁸ Scores on the US Medical Licensing Examination Step I and Step II Knowledge have increased over the past decade for US seniors, US IMGs, and non–US IMGs that have matched in family medicine, suggesting that the test-taking ability of family medicine residents may be improving.

Significant changes in the residency accreditation system have also occurred over the same time frame.⁹ Although introduced before the Accreditation Council for Graduate Medical Education Next Accreditation System that was launched in 2013, the Accreditation Council for Graduate Medical Education General Competencies, faculties' growing familiarity with them, and now the early use of Entrustable Professional Activities have standardized the process of resident evaluation and better harmonized that process with the curriculum. Residents have better information with which they can self-evaluate. The more longitudinal process of evaluation reduces opportunities for residents to say, “I did not know I was not doing well.” The formalization of the Clinical Competency Committee evaluation of residents (and related documentation) allows programs to more effectively monitor resident progress, intervene sooner, and be more specific in identifying areas of improvement. Increased attention has been placed on several core requirements within the family medicine accreditation process, including having adequate educational resources (such as family medicine faculty modeling patient care, and resident and faculty scholarship) and outcome measures like patient volumes and demographics and board certification scores. This has drawn the attention of program directors more toward board certification scores. This increased attention may change the behavior of program directors in selecting medical school graduates who perform better on standardized examinations and in making more resources available to their residents when they prepare for the examination.

We believe that more programs are using “board preparation” resources based on monitoring the discussions that take place on the program directors' listserv, where many opine the most effective resources for resident remediation and overall ABFM board certification preparation. The AAFP has a large number of retired board examination questions available for use when practicing, and increased familiarity with the question format/strategy improves performance on the examination in a way similar to board-preparation courses. The AAFP has expanded its portfolio of board review resources and positioned them within the affordable price range of more residencies. The board review courses are updated annually based on changes in the ABFM examination “blueprint,” and that harmonization process makes them very effective.