Advances in the Cognitive Science and Their Implications for ABFM Knowledge Assessment

Cognitive Skills Must Be Kept Current

The forgetting curve—the exponential decay of memory—was first described more than 130 years ago by Ebbinghaus.² In medicine, as in other professions, development of expertise is marked by acquisition of large amounts of knowledge over time and with a variety of experiences. General aptitude does not necessarily predict expert performance well; rather, ongoing experiences powerfully mediate how information is organized, represented and processed. Here the recent insights of cognitive science are important. A common theme in contemporary cognitive psychology is the existence of 2 distinct systems for information processing—both fast thinking—the rapid pattern recognition that comes with long experience, and “slow thinking”—slow, conscious and under control of the individual¹. Both mediate learning, retention and clinical practice. It is also likely that the quality of practice, rather than just quantity, is necessary to develop and expertise. Mere number of hours of learning on one's own does not adequately explain expert performance. Ample, accurate, and ongoing feedback is also crucial to learning and forgetting.

What are the practical implications for knowledge assessment in board certification? In medicine, of course, keeping up to date is considerably more challenging than retaining what one learned in medical school and residency: keeping up with rapidly changing evidence and standards requires constant attention to learning. This is particularly true for generalists such as family physicians. A point of pride for family medicine is our history of commitment to continuing education, dating back to the AAGP's 1947 commitment to 50 hours of continuing medical education (CME) per year, in service to the health of the public. This historic commitment is reflected in current AAFP and ABFM requirements and is aligned with we know about memory decay. How CME is delivered is also important. Ample evidence suggests that active learning techniques improve learning.^3⇓–5 Yet much of CME is still passive, with credit for CME based on attendance—“butts in the seats.” To address this problem, the College of Family Physicians of Canada has graded CME according to active engagement of learners.⁶

A major policy question is how much CME should be necessary to keep up to date. Medical licensure requirements for CME vary substantially, but generally range from 10 to 40 credits a year; 2 states currently do not require CME. Similarly, requirements for CME across specialty boards are highly variable from none for the American Board of Emergency Medicine to 50 units per year for ABFM. The ideal “right amount” is likely related to a variety of factors, including how CME is delivered, the specialty, current and future scope of practice, and society's expectations, under the social contract, for lifelong learning by physicians. ABFM believes that the public has a vital interest in ensuring that physicians keep up to date. We support a CME requirement and are committed to collaborating with the AAFP and other educational partners as they work to increase the effectiveness of the opportunities they provide for learning.

Self-Assessment Is Not Enough

Existing literature shows that learners have some ability to self-assess strengths and weaknesses, but considerable systematic biases in self-appraisal exist. The predominance of the evidence suggests that in medicine, as in other professions, experts do not know what they do not know. In addition, across multiple disciplines–social science, formal logic, humor, English grammar, face recognition and particularly in medicine–those with less expertise consistently overestimate what they know—they are “unskilled and unaware.” This is called the Dunning-Kruger effect and has been documented for more than 30 years.⁷ Importantly, lack of understanding of knowledge gaps extends to choice of topics for CME. Physicians often choose CME topics not through a rigorous and iterative assessment of their personal knowledge gaps but instead for reasons of convenience, personal interest, or even because of the belief that they are already strong in an area and less effort will be necessary to “check the box”.⁸ Thus, individual learning needs may be missed.

These findings underscore the importance of periodic independent assessment of overall cognitive expertise, such as ABFM provides in the 1-day examination or the new Family Medicine Certification Longitudinal Assessment (FMCLA). In addition, on a more regular basis, ABFM offers focused assessment on individual topics such as hypertension or behavioral health or care of specific populations of patients, such as children or women through Knowledge Self Assessments (KSAs) and the ABFM National Journal Club. The quarterly online Continuous Knowledge Self-Assessment (CKSA) questions address a variety of conditions seen in family medicine practice. There is modest evidence that providing options for learners increases learning: ABFM's goal is to improve learning and potentially to help physicians develop better skills of self-assessment.

Testing Enhances Learning and Retention

Testing powerfully improves learning and retention, both through ongoing practice with retrieval of information and through enhancing motivation. The difficulty of the test is also important: too easy a test does not force retrieval of much information, and too difficult a test can be demotivating. There is a sweet spot! In addition, immediate feedback, both about why a question is right and why the other answers are wrong, is key for driving learning. A significant literature also suggests the importance of independent assessment of procedural performance, with progress toward mastery.⁹ This has clear implications for residency training and for ongoing improvement of performance of specific procedures in practice.

Studying for, and taking, the 1-day family medicine certification examination is a powerful motivator that enhances learning.¹⁰ But the feedback provided by ABFM in this method of testing is not specific and is limited to the organ system categories that frame our current examination blueprint. Some examinees also experience a level of anxiety that may not be conducive to learning. By contrast, FMCLA provides the opportunity to learn in a low anxiety environment where the examinee controls the time, location, and pace of completing.¹¹ It provides immediate and specific feedback for each question, with information about why the right answer is right and the wrong answers are wrong, along with evidence-based critiques and references for further review. Of participants, 95% report accessing these resources while answering the questions, 85% report looking information up about a clinical topic after they complete questions, and 84% report changing their practice as a result of FMCLA. Combined with confirmation that FMCLA is functioning well in assessing physicians' overall knowledge, the evidence is compelling that FMCLA is both summative—providing assessment of learning–and formative—providing assessment for learning. For these reasons, ABFM plans to put FMCLA at the center of its overall design of continuous certification.

Goals and Consequences Motivate

Intrinsic motivation matters. When we first take personal responsibility for patients as medical students and residents, the speed and depth of our learning increases, as it does in the first year out of residency and when physicians change their practices. Desire to improve care is a key component of professionalism. The implication is that the profession and the specialty need to be explicit in “supporting” professionalism and intrinsic motivation across the careers of family physicians through medical school and residency requirements and selection, and in shaping the environments of clinical care and continuing education family physicians work in over their careers. For ABFM, this means supporting intrinsic motivation rather than a mind set of “checking boxes”—providing enough options to allow Diplomates real choice and increase engagement. It also means working for changes in the clinical practice environment that make it easier to be professional, a key goal of ABFM's Center for Professionalism & Value in Health Care.^12,13

Participating in continuous board certification itself also has the potential to promote learning. ABFM data suggest that testing with higher stakes is associated with better performance. CKSA and FMCLA have similar questions presented in similar ways, but FMCLA scoring reveals about 10% more questions answered correctly. We are currently experimenting with more specific feedback—giving Diplomates the testing points of questions they answered incorrectly and highlighting the questions they got wrong when they indicated that the topic was relevant to their practice or were confident in their answer. We are also assessing the effectiveness of spaced-repetition to determine if we can influence the forgetting curve and promote transfer of learning from 1 clinical setting to similar settings. We will continue to align our efforts with what we learn from the cognitive sciences.

Going forward, ABFM will seek to optimize learning from its knowledge assessments. Our new blueprint is organized around clinical activities, which we believe will facilitate learning, and we envision the possibility of adjusting the questions received by an individual Diplomate based on that Diplomates demonstrated gaps of knowledge. The American Board of Anesthesiology is already doing this, and artificial intelligence has the potential to augment this effort. We are also beginning to collaborate with the AAFP to improve the effectiveness of the handoff between identification of knowledge gaps in our assessments and effective CME. Our goal is to use the independent assessment we provide to help guide Diplomate self-education. We believe that supporting Diplomate learning is part of ABFM's responsibility to improving the health of the public.

As always, we welcome your feedback.