Skip to main content

Main menu

  • HOME
  • ARTICLES
    • Current Issue
    • Ahead of Print
    • Archives
    • Abstracts In Press
    • Special Issue Archive
    • Subject Collections
  • INFO FOR
    • Authors
    • Reviewers
    • Call For Papers
    • Subscribers
    • Advertisers
  • SUBMIT
    • Manuscript
    • Peer Review
  • ABOUT
    • The JABFM
    • The Editing Fellowship
    • Editorial Board
    • Indexing
    • Editors' Blog
  • CLASSIFIEDS
  • Other Publications
    • abfm

User menu

Search

  • Advanced search
American Board of Family Medicine
  • Other Publications
    • abfm
American Board of Family Medicine

American Board of Family Medicine

Advanced Search

  • HOME
  • ARTICLES
    • Current Issue
    • Ahead of Print
    • Archives
    • Abstracts In Press
    • Special Issue Archive
    • Subject Collections
  • INFO FOR
    • Authors
    • Reviewers
    • Call For Papers
    • Subscribers
    • Advertisers
  • SUBMIT
    • Manuscript
    • Peer Review
  • ABOUT
    • The JABFM
    • The Editing Fellowship
    • Editorial Board
    • Indexing
    • Editors' Blog
  • CLASSIFIEDS
  • JABFM on Bluesky
  • JABFM On Facebook
  • JABFM On Twitter
  • JABFM On YouTube
NewsBoard News

The Consequential Validity of ABFM Examinations

Kenneth D. Royal and James C. Puffer
The Journal of the American Board of Family Medicine May 2014, 27 (3) 430-431; DOI: https://doi.org/10.3122/jabfm.2014.03.140089
Kenneth D. Royal
PhD
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
James C. Puffer
MD
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • Article
  • References
  • Info & Metrics
  • PDF
Loading

Measurement scholar Samuel Messick,1 defines validity as “an integrated evaluative judgment of the degree to which empirical evidence and theoretical rationales support the adequacy and appropriateness of inferences and actions based on test scores… ” (p. 13). Messick's definition of validity differed from those of previous validity theorists in that he acknowledged that test scores often affect social policy and thus argued social consequences should be examined. Messick referred to this form of validity as “consequential validity.” Shepard2,3 further clarified social consequences to include both the positive/negative and intended/unintended consequences that may result from score-based inferences. The purpose of this article is to discuss consequential validity as it pertains to American Board of Family Medicine (ABFM) examinations.

To date, the ABFM has published numerous articles4⇓⇓⇓⇓⇓–10 that evidence the adequacy and appropriateness of inferences based on examination scores. Many of these articles are validity studies that involve rigorous data analyses with state-of-the-art psychometric methods, whereas others advocate responsible score reporting and interpretation. Given that Messick's1 framework for validity also includes the social consequences that may result from score inferences, it is important to address this aspect of validity as well. Unlike other indicators of validity, consequential validity has less to do with data analysis and more to do with making inferences. Thus, the extent to which ABFM examination scores are appropriately interpreted and used depends largely on others. Our intention is to clarify some key inferences that should and should not be made about ABFM examination score results.

ABFM examinations measure a physician's fund of medical knowledge within the context of the clinical practice of the specialty of family medicine. The examinations do not measure other important aspects of family medicine, such as one's clinical or procedural skills, the ability to communicate with patients, professional attitudes and behaviors, the ability to practice within a system of care, and the ability to learn from the practice of family medicine to continuously improve patient care. Unfortunately, many consumers of ABFM examination score results often make inappropriate inferences about what exactly the scores mean. For example, consumers rightly infer that a passing score conferring certification is a surrogate for quality.11,12 Consumers also rightly infer that a passing score and subsequent certification should facilitate privileges within a hospital setting or credentials within a medical group. Unfortunately, consumers sometimes wrongly infer that a nonpassing score indicates that a physician is not worthy of being certified, and thus by extension, does not or is not capable of providing high-quality care. In addition, some consumers incorrectly infer that a higher examination score is more indicative of a better physician (compared with a physician who has a lower score); it is well understood, however, that multiple factors determine whether a physician is “good.”

It is critical that consumers understand that simply because a physician fails the Maintenance of Certification for Family Physicians (MC-FP) examination does not mean she or he is incapable of providing high-quality care or is incapable of becoming more knowledgeable about the important body of knowledge that defines the specialty of family medicine. Knowledge is fluid; thus everyone has the propensity to become more knowledgeable. In fact, over the years the ABFM staff has heard from hundreds of physicians who initially failed the MC-FP examination and who subsequently developed an improved study plan and passed on their next attempt. Despite the initial stumble, most of these physicians continue to provide quality care to their patients today. Moreover, certification is voluntary. A number of excellent physicians practice family medicine without board certification. Thus, the lack of certification does not imply poor quality; it simply implies the physician has not evidenced his or her knowledge and commitment to continuous improvement by way of a formal certification process.

While fully aware that an examination in and of itself is unable to provide sufficient information about the quality of a physician, the ABFM, along with all American Board of Medical Specialties member boards, adopted a more comprehensive approach to assessing physician performance in 2000. This new paradigm, called Maintenance of Certification, assesses 6 general competencies: professionalism, medical knowledge, communication and interpersonal skills, patient care, systems-based practice, and practice-based learning and improvement. These are assessed by the ABFM within a 4-part construct that (1) assesses professionalism, licensure, and personal conduct; (2) measures the ability of the physician to self-assess and develop a program of life-long learning; (3) assesses by examination cognitive expertise; and (4) assesses the physician's performance in practice and the ability to develop mechanisms to continuously improve quality based on the assessment. We would argue that this expanded approach to physician assessment provides additional information from which appropriate inferences can be made about the quality of care that a physician delivers and has far greater consequential validity within the construct as defined by Messick.1

Conclusion

Empirical data analyses with rigorous research methodologies are critical for providing evidence that an examination is functioning well and measuring the intended construct. The ABFM has produced a considerable body of research that evidences the accuracy and trustworthiness of the score results produced by its examinations. Similarly, the ABFM has continually emphasized that the purpose of the examination is to measure a physician's fund of medical knowledge in clinical family medicine and has emphasized appropriate and responsible score interpretations. Unfortunately, some consumers continue to attach additional meaning to these score results that can affect a physician in unintended ways. To preserve the integrity of the score inferences and their impact on physicians, it is important that all consumers of ABFM examination score results make appropriate and responsible inferences about what exactly the scores mean.

Notes

  • Conflict of interest: The authors are from the ABFM.

References

  1. 1.↵
    1. Linn RL
    1. Messick S
    . Validity. In: Linn RL , ed. Educational measurement. 3rd ed. New York: Macmillan Publishing Co; 1989:13–103.
  2. 2.↵
    1. Shepard LA
    . Evaluating test validity. Rev Res Educ 1993;19:405–50.
    OpenUrlFREE Full Text
  3. 3.↵
    1. Shepard LA
    . The centrality of test use and consequences for test validity. Educ Meas Issues Pract. 1997;16:5–24.
    OpenUrl
  4. 4.↵
    1. O'Neill TR,
    2. Royal KD,
    3. Puffer JP
    . Performance on the American Board of Family Medicine certification examination: are superior test taking skills alone sufficient to pass? J Am Board Fam Med 2011;24:175–80.
    OpenUrlAbstract/FREE Full Text
  5. 5.↵
    1. Royal KD,
    2. Puffer JC
    . Understanding the “sum of subtest to overall score discrepancy” on the Maintenance of Certification for Family Physicians examination. J Am Board Fam Med 2012;25:260–1.
    OpenUrlFREE Full Text
  6. 6.↵
    1. Royal KD,
    2. Puffer JC
    . The reliability of American Board of Family Medicine examinations: implications for test-takers. J Am Board Fam Med 2012;25:131–3.
    OpenUrlFREE Full Text
  7. 7.↵
    1. Royal KD,
    2. Puffer JC
    . Dimensionality of the Maintenance of Certification for Family Physicians examination: evidence of construct validity. J Am Board Fam Med 2013;26:342–4.
    OpenUrlFREE Full Text
  8. 8.↵
    1. Royal KD,
    2. Puffer JC
    . Cheating: its implications for ABFM examinees. J Am Board Fam Med 2012;10:274–5.
    OpenUrl
  9. 9.↵
    1. Royal KD,
    2. Puffer JC
    . Criterion-referenced examinations: implications for the reporting and interpretation of examination results. J Am Board Fam Med 2013;26:225–6.
    OpenUrlFREE Full Text
  10. 10.↵
    1. Royal KD,
    2. Puffer JC
    . A closer look at recertification candidate pass rates. J Am Board Fam Med 2013;26:478–9.
    OpenUrlFREE Full Text
  11. 11.↵
    1. Puffer JC
    . The American Board of Family Medicine certification examination: a proxy for quality. Fam Med 2011;43:436–7.
    OpenUrl
  12. 12.↵
    1. O'Neill TR,
    2. Puffer JC
    . Maintenance of certification and its association with the clinical knowledge of family physicians. Acad Med 2013;88:780–7.
    OpenUrlCrossRefPubMed
PreviousNext
Back to top

In this issue

The Journal of the American Board of Family     Medicine: 27 (3)
The Journal of the American Board of Family Medicine
Vol. 27, Issue 3
May-June 2014
  • Table of Contents
  • Table of Contents (PDF)
  • Cover (PDF)
  • Index by author
  • Back Matter (PDF)
  • Front Matter (PDF)
Print
Download PDF
Article Alerts
Sign In to Email Alerts with your Email Address
Email Article

Thank you for your interest in spreading the word on American Board of Family Medicine.

NOTE: We only request your email address so that the person you are recommending the page to knows that you wanted them to see it, and that it is not junk mail. We do not capture any email address.

Enter multiple addresses on separate lines or separate them with commas.
The Consequential Validity of ABFM Examinations
(Your Name) has sent you a message from American Board of Family Medicine
(Your Name) thought you would like to see the American Board of Family Medicine web site.
CAPTCHA
This question is for testing whether or not you are a human visitor and to prevent automated spam submissions.
15 + 4 =
Solve this simple math problem and enter the result. E.g. for 1+3, enter 4.
Citation Tools
The Consequential Validity of ABFM Examinations
Kenneth D. Royal, James C. Puffer
The Journal of the American Board of Family Medicine May 2014, 27 (3) 430-431; DOI: 10.3122/jabfm.2014.03.140089

Citation Manager Formats

  • BibTeX
  • Bookends
  • EasyBib
  • EndNote (tagged)
  • EndNote 8 (xml)
  • Medlars
  • Mendeley
  • Papers
  • RefWorks Tagged
  • Ref Manager
  • RIS
  • Zotero
Share
The Consequential Validity of ABFM Examinations
Kenneth D. Royal, James C. Puffer
The Journal of the American Board of Family Medicine May 2014, 27 (3) 430-431; DOI: 10.3122/jabfm.2014.03.140089
Twitter logo Facebook logo Mendeley logo
  • Tweet Widget
  • Facebook Like
  • Google Plus One

Jump to section

  • Article
    • Conclusion
    • Notes
    • References
  • References
  • Info & Metrics
  • PDF

Related Articles

  • No related articles found.
  • Google Scholar

Cited By...

  • No citing articles found.
  • Google Scholar

More in this TOC Section

  • What Assessments Are Being Used in Family Medicine Residencies?
  • ABFM Outreach: A Strategic Approach to Creating Genuine Partnerships with Family Physicians
  • Implementing Competency Based ABFM Board Eligibility
Show more Board News

Similar Articles

Navigate

  • Home
  • Current Issue
  • Past Issues

Authors & Reviewers

  • Info For Authors
  • Info For Reviewers
  • Submit A Manuscript/Review

Other Services

  • Get Email Alerts
  • Classifieds
  • Reprints and Permissions

Other Resources

  • Forms
  • Contact Us
  • ABFM News

© 2025 American Board of Family Medicine

Powered by HighWire