Re: Family Physicians' Knowledge of Commonly Overused Treatments and Tests

To the Editor: Maurer et al¹ outlined the results of a simple yet elegant study that shows family physicians' knowledge of commonly overused treatments and tests. At first glance, the article seems to demonstrate important gaps in knowledge; however, a detailed reading of the article gives cause for doubt.

Maurer et al¹ and the Journal of the American Board of Family Medicine should be lauded for publishing the actual multiple choice questions used in the study in the appendix of the article. Unfortunately, all the questions leave room for improvement. The second question offers the learner a binary choice: yes or no. Someone with no medical knowledge would have a 50% chance of getting this question correct. The test-wise candidate—who knows that he or she is undergoing an assessment to test his or her ability to “choose wisely”—might also feel prompted to give the answer that the examiner expects, and might thus choose the option with the least number of treatments or tests. The remaining questions ask the candidate to choose the most appropriate scenario for the question. Once again, this is less than optimal as it doesn't simulate clinical reality. Questions should ideally mimic clinical practice—hardly ever in clinical primary care medicine do 4 patients in a row present in relation to a possible Papanicolaou test (as in question 3). The other questions contain similar issues. These problems might seem minor; however, they may create questions that are not as valid and reliable as they should be. The art and science of constructing test questions has advanced over many years, and ignoring the advances that have been made seems a shortcoming.²

The good news is that the core content of the questions is likely to be sound and that judicious editing will make them valid and reliable. The questions could then be used in a variety of contexts outside the framework of the study. The questions, or questions like them, could be used efficiently and at low cost in formative or summative assessments, E-learning resources, or simply in promotional campaigns aiming to encourage wiser choices among physicians.³ Increasingly, the single best answer to such questions might be no investigation, no drug treatment, and no surgery.