Pass Rates on the American Board of Family Medicine Certification Exam by Residency Location and Size

Discussion

In this retrospective study, by definition, we have shown that there are 107 of 429 family medicine residency programs that were at or below the 25th percentile from 2007 to 2011. Even more worrisome is that there are 205 family medicine residency programs (47.8%) that have failed to maintain a ≤10% fail rate over 3- to 5-year time intervals for first-time examinees on the ABFM CE. Approximately half of the residency programs are at risk of being cited for failure to meet published ACGME guidelines.

Interestingly, there were more residency programs that had a ≤10% fail rate in the past 3 years as opposed to the last 4 and 5 years. This is an indication that residency programs have improved their outcomes no recent examination. Notable improvements in examination performance is one of the considerations of the ACGME.⁵ Residency program citation for board performance is common. In 2009, 32 of 114 programs (28.1%) were cited with regard to board scores; this was the second-most common citation.¹⁶ The potential citation rate of 47.8% found in this study is higher than the 28.1% actual recent citation rate. First, this study evaluates all programs, whereas individual programmatic review does not occur annually. Second, the definition of “notable improvement” is not evaluated in this study. Finally, these data suggest that overall program performance has improved the most since 2009. Notably, all these program improvements occurred after the change to the examination blueprint in 2006,² suggesting that changes are not due to the examination itself, but to changes in the examinees or programmatic variables.

Regardless, the high proportion of residency programs that fail to meet the 90% pass rate standard makes us wonder whether the 90% pass rate standard for family medicine should be changed. In internal medicine and pediatrics, for example, the ACGME pass rate standards for first-time examinees are 80% and 60%, respectively.⁵ Given that the overall pass rate for first-time family medicine examinees during the study period was 86.9%, we feel that the 90% pass-rate policy standard should be reevaluated. The only other discipline that has an ACGME program percentile requirement is in radiation oncology (25th percentile).⁵

We also have demonstrated that program size is associated with 5-year pass rates on the ABFM CE; the findings shown through linear regression are consistent with our hypothesis. These results are similar to results shown in pediatrics and surgery.^11,12 However, the magnitude of the Pearson correlation coefficient is very small and is evident in the wide variability shown in Figure 1. It is unlikely that size alone is associated with examinee performance. It is more likely that confounding variables associated with program size account for this difference in educational outcomes. The nature of curricula, the ability to attract specialized faculty, higher-quality residents, and established education programs could be a few potential reasons that larger programs outperform smaller programs in general. There are also individual variables that may be associated with program size, including individual variables such as allopathic or osteopathic training, age, duration of training, and time between residency and taking the ABFM CE. These potential programmatic and individual variables should be studied in the future to determine whether they have any predictive validity for examination performance.

Finally, we have shown clear differences in regional performance on the ABFM CE, consistent with our study hypothesis. The Northeast and the South performed at lower levels than the Midwest and the West. Interestingly, some programs in the Northeast and South do have 100% pass rates for first-time examinees on the ABFM CE. Residency program performance is extremely complex and depends on multiple factors, such as resident selection, curriculum structure, faculty recruitment, and incentives.

Other areas of concern for our study include its retrospective cohort design. We are unable to establish any causation between variables. Second, there are multiple potentially confounding variables related to program size. We could not define exactly which variables are responsible for higher program performance. Importantly, we did not study the programs that have been cited for failing to meet the ACGME examination performance standards; this study was a retrospective analysis of programs that did not meet the published ACGME examination performance standards based on published performance on the ABFM CE. Finally, this study does not evaluate any patient-related outcomes or any clinical outcomes of examinee performance—only binary outcomes on the ABFM CE.

However, among the strengths to this study is that it evaluates the vast majority of family medicine training programs in the United States and Puerto Rico, reviewing the entire examinee population that attempts the ABFM CE as opposed to a smaller cohort of local or regional examinees. Since the data are publically available, this study is easily reproducible. This research also has longitudinal validity in that recent performance over 5 years is evaluated. This study is the first to evaluate the current state of family medicine residency program performance on the ABFM CE and its relation to the important ACGME program criteria and the demographic factors associated with higher performance on the ABFM CE.

The findings of this study have a few implications. First, it notes an alarming potential citation rate for residency training programs that are at risk. This finding might lead to a reevaluation of the ACGME program performance policy standards. This study may also shift the potential application and interview patterns toward residency programs in the West, larger residency programs, and programs that have a recent history of strong performance on the ABFM CE.