ORIGINAL RESEARCH
Thomas R. O’Neill, PhD; Keith Stelter, MD, MMM; Ting Wang, PhD
Corresponding Author: Thomas R. O’Neill, PhD; The American Board of Family Medicine
Email: toneill@theabfm.org
DOI: 10.3122/jabfm.2023.230443R1
Keywords: Certification, Evaluation Study, Family Medicine, Licensing, Longitudinal Studies, Psychometrics, Research Design, Scales, Statistics
Dates: Submitted: 12-03-2023; Revised: 04-03-2024; Accepted: 04-08-2024
AHEAD OF PRINT: |HTML| |PDF| FINAL PUBLICATION: |HTML| |PDF|
PURPOSE: To determine whether the construct of family medicine clinical decision-making ability was invariant across modes of administration, the one-day exam and the longitudinal assessment. We attempted to identify item characteristics associated with differences in difficulty across modes of administration.
METHODS: The data were item difficulty calibrations based on examinee responses to the one-day exam and the longitudinal assessment. A repeated measures design was employed to identify question calibration differences across modes of administration, so that the stability of the question difficulty across modes of administration could be assessed. A qualitative review of the flagged questions was conducted to identify characteristics associated with questions becoming easier or more difficult.
RESULTS: The correlation between the pairs of calibrations was moderately positive r(298) = .558, p < .001 suggesting that the questions are functioning somewhat similarly across the different modes of administration; however, the scatterplot demonstrates that many of the questions became easier. Of the 298 repeated measures t-tests, 37% (110) did not show a significant difference, 43% (128) became easier on the longitudinal assessment, and 20% (60) became more difficult.
CONCLUSIONS: This study suggests that changes in item difficulty do occur when extra time and the use of external resources are permitted. Usually the questions get easier, but in some cases the question becomes more difficult. Possible reasons for this are presented, and a method to adjust the item difficulty in a way to maintain a single construct is presented.