Longitudinal Assessment: Where We Are and Why It Is Important

On January 4, 2019, the American Board of Family Medicine (ABFM) launched the Family Medicine Certification Longitudinal Assessment (FMCLA) pilot project as an alternative to the 1-day examination. For those eligible—Diplomates in good standing, due for the examination in 2019—FMCLA provides 25 questions online every quarter, which can be completed anywhere, and at any time that is convenient. Four years will be allowed to complete a total of 300 questions. It would be possible to complete the requirement in only 3 years, but flexibility is built in to allow delaying questions, or even an entire quarter, so that personal or professional needs can take priority if necessary. Family Medicine Certification will remain active during this time as long as Diplomates are meaningfully participating in FMCLA and keeping up their regular stage requirements.

FMCLA allows access to reference materials, with up to 5 minutes to complete each question. In addition to increasing convenience and reducing the cost of assessing cognitive expertise, we believe that the format will also support Diplomates' learning. After answering a question, the correct answer is provided, along with a critique explaining why the correct answer is the best answer, and why the other choices are wrong. References are also provided for additional learning opportunities. After the first 100 questions are completed, participants will receive a preliminary performance report that includes their likelihood of passing, as well as feedback on specific gaps in their knowledge, to assist in their selection of further study or continuing medical education opportunities.

What has been the early experience from the FMCLA pilot? There is clearly great interest in the new format. As of this writing, over 90% of Diplomates choosing FMCLA indicated that convenience, the ability to monitor progress, and helping stay up to date were key issues. In addition, nearly two thirds indicated that the expense of board review courses was relevant to their choice. A total of 8418 Diplomates were eligible for the pilot, representing 9% of all Board-Certified family physicians. As of this writing, 73% of eligible Diplomates have chosen FMCLA, while 13% have applied for the April examination. The door to the FMCLA pilot closed in March. Our current intent is to offer both FMCLA and the 1-day examination for the foreseeable future, so that Diplomates can choose their preferred method. While FMCLA seems to be attractive to many, we believe that there are valid personal reasons or preferences why some may choose to take the 1-day examination.

The FMCLA IT platform seems to be performing quite well. Our staff at ABFM is monitoring performance daily and reviewing the feedback comments provided by participants every week. When suggestions for improvement or concerns are identified, these are addressed in real time. Feedback on the questions themselves has also been valuable, leading us to conclude in a few cases that we should not use a particular question for standard setting. Of over 2644 Diplomates who have completed the questions for the first quarter as of this writing, the average time spent per question has been 2 minutes and 20 seconds. Finally, although complete data are forthcoming, many have reported improved relevance and better learning. As 1 American Academy of Family Physicians Board member who is in the FMCLA pilot commented:

The biggest pro for using this format is that it is forcing me to learn something new each week. If I get something wrong, I know immediately and get the critique, which informs my ability to use the information in real life. The other pro is not having to schedule time off to take the examination. And I love the flexibility of choosing when and where to answer questions.¹

Over the longer term, we will be getting additional feedback from Diplomates about how we can improve FMCLA and better support learning. We anticipate that Diplomates themselves will become more familiar with the process—an early lesson, from the American Board of Anesthesia, was from Diplomates who reported that it did not work well to answer questions in the grocery line!—and how to use references selectively in real time, ideally in a way similar to practice. It is important for Diplomates to understand the test is challenging, as befits the knowledge needed to set the standard for cognitive expertise in the specialty. Each question has been written by practicing family physicians and pretested on initial certifying physicians and Diplomates, so that the calibrated difficulty of each question is generalizable to the population of over 90,000 family physicians. Answering only about two thirds of the questions correctly is necessary to pass the standard for Board Certification.

A major part of our evaluation will be assessing how well the longitudinal test performs psychometrically to identify the cognitive expertise necessary for Board Certification compared with our standard 1-day examination. The evidence is longstanding, extensive and clear that physicians' ability to assess their own knowledge is limited.^2⇓⇓–5 The value of the examination to both Diplomates and to the public is that it provides an independent assessment of cognitive expertise. The first question of our evaluation of the pilot will be how specific questions perform in an “open-book” environment. Given the large number of participants, we will know the answer by the time this article is published. In a little over 2 years, we will know how our standard setting performs at the level of both the individual and the population of test takers, as well as how stable cognitive expertise remains over time. A final major issue will be evaluating how well our emphasis on professionalism and the security steps we have taken have been able to safeguard the identity of individual participants and the integrity of the specific multiple-choice questions. We are committed to maintaining the value of the Certificate for all Diplomates.

The other major question for us is the degree to which FMCLA can improve learning. In keeping with contemporary thinking about how physicians keep up to date⁶ and the experience of other American Board of Medical Specialties Boards,⁷ we believe that assessments play a critical role in driving learning. As noted above, we are providing answers, a critique and an article after each question. After the 100th question, each Diplomate will receive an estimated scaled score, representing our best prediction of the likelihood of ultimately passing the examination. The goal is to help Diplomates know where they are—and signal whether they need to do anything differently over the next several years to pass the examination. As we do for the 1-day examination, we will also provide a summary of individual performance by organ system. In addition, we will be experimenting with different types of feedback, geared toward relevance to the Diplomate's practice and their relative confidence in the answers they provide. Evaluation at the end of the first year of the pilot will focus on optimization of helping Diplomates learn. Over the longer term, our goal is to provide a guide for self-directed learning for each Diplomate, while supporting the American Academy of Family Physicians and our other partners as they develop tools for family physicians to keep up to date.

By the end of the summer, we will be able to determine whether we can extend the pilot to those who are due to take the examination in 2020. We are also beginning to plan with other Boards about how to extend longitudinal assessment to Certificates of Added Qualifications, such Sports Medicine, Geriatrics, etc. We will explore how to extend the opportunity for interested Diplomates to begin FMCLA earlier than their 10-year required assessment. Finally, in the spring of 2021, we are scheduled to report formally to American Board of Medical Specialties about the results of the pilot. It is at that time we will share a plan for extending longitudinal assessment to all Diplomates.

Over the long term, FMCLA represents a first expression of ABFM's renewed commitment to seek a new relationship with board-certified family physicians as they serve their patients and communities. We want to develop new programs with your input and with a goal of supporting you in your demonstration of lifelong learning, cognitive expertise, professionalism, and commitment to improving practice. The coming years will see much more in addition to FMCLA.