The ABFM Begins to Use Differential Item Functioning

Flagging Items

The particular method of DIF detection used by the ABFM is based on the dichotomous Rasch model.^4⇓–6 Using this method, the items are calibrated twice: first using only responses from members of the reference group and next using only responses from members of the focal group. Because the largest self-reported ethnicity among ABFM diplomates is white, the ethnicity reference group is white and the focal groups are the other ethnicity categories. Using this same reasoning, the reference group for sex is male and the focal group is female. Although the fine-tuning of this method to meet the needs of the ABFM is still being developed, the process will largely reflect the procedure described below.

For each item, the 2 calibrations are compared. If the 2 calibrations fall outside of the 95% confidence interval for their mean, then the item is flagged as potentially biased. Please note that the potential bias could be to the advantage or the disadvantage of the focal group. In addition, when using this flagging criterion, it is expected that approximately 5% of the items will be flagged just by chance. Although the criteria could be made more stringent to reduce the number of false positives, it also would reduce the number of false negatives, potentially permitting some biased items to go undetected. The 95% confidence interval seems to be reasonable for use as an initial screening criterion. All items that are flagged as potentially biased in either direction are forwarded to the DIF Review Panel for evaluation. Over time, the screening criteria will likely be better optimized.

Convening a DIF Review Panel

The DIF Review Panel is convened once a year to review the content of items that have been flagged for potential bias. The panel comprises subject matter experts (ABFM diplomates) who represent a diversity of ethnicities and both sexes. The panel also includes a linguist and is moderated by a psychometrician. The panel meeting begins with an explanation of DIF as a concept and the purpose of the panel. The panel is charged with the responsibility of reviewing items for appropriateness for the examination with regard to DIF. The panel may decide that there is no identifiable content that caused the DIF and permit the item to stand. On the other hand, the panel may decide that there is an identifiable source of DIF. If so, the panel must determine whether that source of DIF is related to an important aspect of family medicine. If it is important, then the panel is to let the item stand. If it is not important, then the panel should recommend that the item be deleted or reworked. The items that the panel recommends deleting or reworking are forwarded to the ABFM Examination Committee.

Determining the Final Disposition of the Items

The Examination Committee reviews the recommendations of the DIF panel and makes a final decision on whether an item is sent back to the ABFM content development department for revision/deletion or is permitted to stand. To send the item back for revision/deletion, the Examination Committee should concur that there is likely something in the item causing the difference in relative difficulty that is not an important aspect of family medicine. Of course, the Examination Committee can always send an item back to be reworked or deleted and the reason need not be limited to DIF issues; however, the Examination Committee review is the final step in determining the disposition of an item.

Summary

To defend against claims of discrimination, the certification and licensure testing industry routinely uses DIF to detect items that function differently for protected classes.⁷ While most other American Board of Medical Specialty boards are not yet collecting this information, the ABFM has begun collecting ethnicity data from candidates applying for its examinations so that this kind of bias can be detected. The industry generally regards this type of analysis as a best testing practice that makes the meaning of the examination results more stable across subpopulations.⁸ Documentation of these processes also can be used to show that a test publisher has made a diligent effort to minimize or eliminate sources of irrelevant variance that might have detrimental effects on subpopulations of interest.

On a final note, it is important to underscore that the ABFM does not release ethnicity information to external parties. Furthermore, ethnicity and sex are not used to determine the difficulty of test items with regard to scoring the examination. The operational item calibrations that are used for scoring are based on responses from the entire group, not a particular ethnicity or sex reference group. There are not different passing standards or different scales for the different ethnic groups or sexes: there is only one scale with a single passing standard that applies.