The American Board of Family Medicine: New Tools to Assist Program Directors and Graduates Achieve Success
==========================================================================================================

* Stanley M. Kozakowski
* Perry A. Pugno

## Abstract

In this commentary we review the improvements in the pass rates for first-time American Board of Family Medicine (ABFM) Certification Examination test takers in the context of new tools and resources for program directors against the backdrop of a changing accreditation system and increased competition for a relatively fixed number of graduate medical education positions in family medicine. While causality cannot be established between the strategic initiatives of the ABFM and higher pass rates, we can all celebrate the new tools and resources provided to residents and program directors, and the improved performance of family medicine graduates on the certification examination.

In this issue, Puffer et al1 demonstrate a steady increase in pass rates among graduating family medicine residents taking the American Board of Family Medicine (ABFM) Certification Examination since 2012, after a steady decline in the preceding years. This improvement follows the ABFM's implementation of several strategies to assist program directors. As former residency directors, and as directors of the Medical Education Division of the American Academy of Family Physicians (AAFP), we have a long-term perspective on family medicine residency education and the changing environment of graduate medical education over the past several decades.

The changes instituted by the ABFM represent significant aid to program directors as they work to ensure that their graduates are prepared to safely and effectively enter independent practice. Board certification is an important metric of clinical knowledge. Other important training domains include application of clinical knowledge, communication and interpersonal skills, professionalism, intellectual curiosity and learning to improve the process and quality of care, and understanding systems of care and the role of the team and practice within the broader medical neighborhood. While these domains are all important, most are challenging to measure, and board certification as a measure of medical knowledge has been correlated with quality of care, overall ratings of a resident's clinical competence, and career satisfaction.2⇓⇓–5 For all these reasons, improvement in board scores is good for residency programs, their graduates, and the public.

## Tools for Program Directors

Program directors have been challenged with identifying those residents who lack sufficient medical knowledge, because residents with deficits in that domain can manifest the deficit in many ways, including what may seem to be a lack of judgment or even slowness in clinic or hospital rounds. The In-Training Examination (ITE) has been characterized as a “low-stakes” examination, and program directors have varied in their perspectives regarding how much stock that they should place on the ITE for residents with poor or marginal performance on the examination, despite evidence of the predictive validity of the ITE.6 The Bayesian Score Predictor is a wonderful resource that allows program directors to reliably identify residents' future performance on the certification examination and to provide resources to and motivate residents who may be at risk of not passing the certification examination. We believe that moving the certification examination from July to April, coupled with the Family Medicine Certification entry requirements, is particularly useful to ensure that all residents participate in a minimal level of practice improvement and knowledge self-assessment. The new knowledge self-assessment requirement during residency may be an important reason why performance has improved. The fact that many physicians do poorly on their first pass through the knowledge self-assessment questions indicates that, based on objective testing, we often do not know as much as we think we do.

## Causality

Puffer et al. are rightly cautious to attribute the variance in the pass rate on the certification examination to individual strategies. Much has changed in the graduate medical education environment over the past decade. As the authors speculate, the “quality of family medicine trainees recruited into training programs” may have been improving during the time frame of their study. Increased numbers of graduates from both US allopathic (MD degrees) and osteopathic (DO degrees) medical schools over the past 15 years create more competition for graduate medical education positions and disproportionately disadvantage international medical graduates (IMGs).7 The composition of matriculating family medicine residents has changed over the past decade. While the proportion of US graduates with an MD degree has remained stable at approximately 46% of entering first-year residents, over the past decade the percentage of graduates with a DO degree has increased approximately 1% per year, while the percentage of IMGs has experienced a reciprocal decline.8 Scores on the US Medical Licensing Examination Step I and Step II Knowledge have increased over the past decade for US seniors, US IMGs, and non–US IMGs that have matched in family medicine, suggesting that the test-taking ability of family medicine residents may be improving.

Significant changes in the residency accreditation system have also occurred over the same time frame.9 Although introduced before the Accreditation Council for Graduate Medical Education Next Accreditation System that was launched in 2013, the Accreditation Council for Graduate Medical Education General Competencies, faculties' growing familiarity with them, and now the early use of Entrustable Professional Activities have standardized the process of resident evaluation and better harmonized that process with the curriculum. Residents have better information with which they can self-evaluate. The more longitudinal process of evaluation reduces opportunities for residents to say, “I did not know I was not doing well.” The formalization of the Clinical Competency Committee evaluation of residents (and related documentation) allows programs to more effectively monitor resident progress, intervene sooner, and be more specific in identifying areas of improvement. Increased attention has been placed on several core requirements within the family medicine accreditation process, including having adequate educational resources (such as family medicine faculty modeling patient care, and resident and faculty scholarship) and outcome measures like patient volumes and demographics and board certification scores. This has drawn the attention of program directors more toward board certification scores. This increased attention may change the behavior of program directors in selecting medical school graduates who perform better on standardized examinations and in making more resources available to their residents when they prepare for the examination.

We believe that more programs are using “board preparation” resources based on monitoring the discussions that take place on the program directors' listserv, where many opine the most effective resources for resident remediation and overall ABFM board certification preparation. The AAFP has a large number of retired board examination questions available for use when practicing, and increased familiarity with the question format/strategy improves performance on the examination in a way similar to board-preparation courses. The AAFP has expanded its portfolio of board review resources and positioned them within the affordable price range of more residencies. The board review courses are updated annually based on changes in the ABFM examination “blueprint,” and that harmonization process makes them very effective.

## Conclusion

We commend Puffer et al1 and the ABFM for their commitment to improving the quality of medical care available to the public, establishing and maintaining standards of excellence in the specialty of Family Medicine, improving the standards for medical education in Family Medicine, and determining through evaluation the fitness of specialists in Family Medicine who apply for and hold certificates.10 The strategies they have developed for program directors are laudable. While these strategies cannot be firmly established as the sole cause of the increasing pass rates for graduating residents on the ABFM Certification Examination, we believe that they have made a large contribution to the increase.

## Notes

*   *Funding:* none.

*   *Conflict of interest:* none declared.

*   To see this article online, please go to: [http://jabfm.org/content/30/5/567.full](http://jabfm.org/content/30/5/567.full).

## References

1.  1.Puffer JC, Peabody MR, O'Neill TR. Performance of graduating residents on the American Board of Family Medicine Certification Examination 2009–2016. J Am Board Fam Med 2017;30:570–571.
    
    [Abstract/FREE Full Text](http://www.jabfm.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NToiamFiZnAiO3M6NToicmVzaWQiO3M6ODoiMzAvNS81NzAiO3M6NDoiYXRvbSI7czoyMDoiL2phYmZwLzMwLzUvNTY3LmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

2.  2.Holmboe ES, Wang Y, Meehan TP, et al. Association between maintenance of certification examination scores and quality of care for medicare beneficiaries. Arch Intern Med 2008;168:1396–403.
    
    [CrossRef](http://www.jabfm.org/lookup/external-ref?access_num=10.1001/archinte.168.13.1396&link_type=DOI) 
    
    [PubMed](http://www.jabfm.org/lookup/external-ref?access_num=18625919&link_type=MED&atom=%2Fjabfp%2F30%2F5%2F567.atom) 

3.  3.Holmboe ES, Weng W, Arnold GK, et al. The comprehensive care project: measuring physician performance in ambulatory practice. Health Serv Res 2010;45(6 Pt 2):1912–33.
    
    [CrossRef](http://www.jabfm.org/lookup/external-ref?access_num=10.1111/j.1475-6773.2010.01160.x&link_type=DOI) 
    
    [PubMed](http://www.jabfm.org/lookup/external-ref?access_num=20819110&link_type=MED&atom=%2Fjabfp%2F30%2F5%2F567.atom) 
    
    [Web of Science](http://www.jabfm.org/lookup/external-ref?access_num=000284066300007&link_type=ISI) 

4.  4.Norcini JJ, Webster GD, Grosso LJ, Blank LL, Benson JA Jr.. Ratings of residents' clinical competence and performance on certification examination. J Med Educ 1987;62:457–62.
    
    [PubMed](http://www.jabfm.org/lookup/external-ref?access_num=3599033&link_type=MED&atom=%2Fjabfp%2F30%2F5%2F567.atom) 
    
    [Web of Science](http://www.jabfm.org/lookup/external-ref?access_num=A1987H761700001&link_type=ISI) 

5.  5.Gray B, Reschovsky J, Holmboe E, Lipner R. Do early career indicators of clinical skill predict subsequent career outcomes and practice characteristics for general internists? Health Serv Res 2013;48:1096–115.
    
    

6.  6.O'Neill TR, Li Z, Peabody MR, Lybarger M, Royal K, Puffer JC. The predictive validity of the ABFM's In-Training Examination. Fam Med 2015;47:349–56.
    
    

7.  7.Mullan F, Salsberg E, Weider K. Why a GME squeeze is unlikely. N Engl J Med 2015;373:2397–9.
    
    [PubMed](http://www.jabfm.org/lookup/external-ref?access_num=26536521&link_type=MED&atom=%2Fjabfp%2F30%2F5%2F567.atom) 

8.  8.Kozakowski SM, Travis A, Bentley A, Fetter G Jr.. Entry of US medical school graduates into family medicine residencies: 2015–2016. Fam Med 2016;48:688–95.
    
    

9.  9.Nasca TJ, Philibert I, Brigham T, Flynn TC. The next GME accreditation system–rationale and benefits. N Engl J Med 2012;366:1051–6.
    
    [CrossRef](http://www.jabfm.org/lookup/external-ref?access_num=10.1056/NEJMsr1200117&link_type=DOI) 
    
    [PubMed](http://www.jabfm.org/lookup/external-ref?access_num=22356262&link_type=MED&atom=%2Fjabfp%2F30%2F5%2F567.atom) 
    
    [Web of Science](http://www.jabfm.org/lookup/external-ref?access_num=000301482800012&link_type=ISI) 

10. 10.About ABFM. Lextington, KY: American Board of Family Medicine. Available from: [https://www.theabfm.org/about/index.aspx](https://www.theabfm.org/about/index.aspx). Accessed June 20, 2017.