Thomas R. O’Neill, PhD; Ting Wang, PhD; Warren P. Newton, MD, MPH
Corresponding Author: Thomas R. O’Neill, PhD; American Board of Family Medicine. Email: toneill@theabfm.org
Section: Original Research
Publication Date: 1/13/2022
Introduction: Differential Item Functioning (DIF) procedures flag examination questions in which examinees from different subpopulations (gender, race, ethnicity, etc.), who are of equal ability, do not have the same probability of answering it correctly. Few medical certification boards employ DIF procedures because they do not collect the needed data on the examinee's race or ethnicity. This article summarizes the American Board of Family Medicine’s (ABFM) combined use of DIF procedures and an expert panel to review certification questions for bias. Methods: ABFM certification exam data from 2013 to 2020 was analyzed using a DIF procedure to flag questions with possible ethnic or racial bias. The flagged questions were next reviewed by a racially and ethnically diverse panel of content experts. If the panel judged the source of the positive DIF was not clinically relevant for the practice of family medicine, the question was removed from the examination. Results: Out of the 3487 questions analyzed, 374 (11%) unique questions including 490 comparisons were flagged by DIF procedures as potentially biased. The flagged questions were balanced between questions that favored the majority group (white) or favored one of three minority groups (Black, Hispanic, or Asian). For the minority groups, the mean net advantage was always positive but not to a statistically significant degree. Of the flagged questions, the review panel removed 4 for unnecessary bias. Discussion: Only a small number of the ABFM board certification examination questions contained unnecessary racial or ethnic bias. Using DIF procedures and panel review can improve the quality of the board certification questions and demonstrate the organization’s commitment to avoid racial or ethnic bias.