Abstract
Introduction: Maintenance of Certification (MOC) was implemented to help physicians remain current with evolving medical standards, but has been criticized for being irrelevant to practice. We assessed family physicians' (FPs') opinions about the content of American Board of Family Medicine (ABFM) self-assessment modules (SAMs).
Methods: We used ABFM administrative data from feedback surveys completed after each of the 16 SAMs from 2006 to 2016. FPs rated agreement with 2 statements—1) “Content is appropriate for my practice,” and 2) “Content was presented at an appropriate level”—on a scale of 1 (strongly disagree) to 6 (strongly agree). We calculated mean ratings of each statement by year and stratified by Knowledge Assessment (KA) and Clinical Simulation (CS) portions of the SAM. We plotted mean ratings by FPs' age at their first SAM completion and the total number of SAMs completed.
Results: SAMs were completed (n = 633,198) from 2006 to 2016 with 448,408 (71%) feedback surveys completed. The annual mean ratings of both statements varied little (less than 0.5) and were above 4.5 for all SAMs. CS ratings were consistently lower than KA ratings. FPs of all ages at first SAM provided similar ratings and agreement with content appropriateness increased with repeated exposure to SAMs.
Conclusion: Over 11 years, the content of ABFM SAMs was regarded by FPs as appropriate for practice and presented at an appropriate level. Continued monitoring of feedback is necessary to keep the content of MOC programs relevant for physicians' practice.
The American Board of Medical Specialties (ABMS) approved a Maintenance of Certification (MOC) program in 2000 to provide a platform to continuously assess physicians' performance and currency with evolving medical evidence. ABMS set standards for MOC that allowed each member board to design a program to meet the needs of their Diplomates within a 4-part framework: professional standing (Part I), lifelong learning and self assessment (Part II), medical knowledge and skills (Part III), and improvement in practice (Part IV).1 However, these standards have sometimes triggered tensions between certifying boards and their Diplomates.2 Despite growing evidence linking MOC participation with improvement in patient care,3⇓⇓⇓⇓–8 opinions about MOC have been controversial and its value challenged.9⇓–11 Physician experiences with MOC have demonstrated both satisfaction,12⇓⇓–15 and dissatisfaction.16,17
The ABMS MOC Part II activities are intended to be relevant, easy-to-use, cost-effective, and meaningful for physicians.1 However, there is little evidence demonstrating whether these goals have been met. The American Board of Family Medicine (ABFM), as an ABMS member board, is able to offer insights regarding physician opinions about its MOC for Family Physicians (MC-FP) Part II activities, as a result of its long-standing business requirement of collecting feedback from Diplomates who complete a self-assessment module (SAM), which comprised a knowledge assessment (KA) and a clinical simulation (CS). Soon after their rollout in 2004, an early analysis of the ABFM feedback surveys indicated that most family physicians (FPs) had a favorable experience with SAMs,13 and a more recent analysis of over 320,000 feedback surveys of FPs who completed a SAM between 2004 and 2013 found that a large majority of participants rated their experience with SAMs as highly positive.18
As the potential scope of practice of FPs is very broad, it is important to determine whether the SAMs, which are focused on a specific clinical area, are providing knowledge that is appropriate for practice. Scope of practice often changes as FPs age and an internal ABFM analysis of 6 months of data found that older FPs rated KA more positively than CS compared with younger FPs.19 As a result, we hypothesized that older Diplomates may have found the CS difficult and been less satisfied than younger Diplomates. This study sought to evaluate FPs' opinions about ABFM SAMs and also to determine whether opinions differed between younger and older FPs.
Methods
Data
This study analyzed data from ABFM SAM feedback surveys completed between January 2006 and June 2016 before ABFM decoupled SAMs into independent Knowledge Self-Assessment (KSA) and Clinical Self-Assessment (CSA) activities.20 Until 2009, physicians were required to complete the surveys to receive Continuing Medical Education (CME) credits. After 2009, survey completion was optional.
During the period of the study, participants in MC-FP needed to complete 2 SAMs in a 3-year period to satisfy the lifelong learning requirement. Each SAM included 2 components: a 60-question Knowledge Assessment (KA) that focused on medical knowledge, and a Clinical Simulation (CS) to simulate a patient clinical encounter. To keep the SAMs current, ABFM adopted multiple strategies to update the content regularly.21 Diplomates had to answer 80% of the KA questions correctly before proceeding to the CS. After completing both SAM components, FPs were asked to evaluate the activities. For each KA and each CS, FPs were asked to rate how much they agreed with the following statements: 1) “Content is appropriate for my practice” and 2) “Content was presented at an appropriate level.” The number of SAM topics from which Diplomates could choose increased from 2 (Diabetes and Hypertension), at launch in 2004 to a selection of 16 topics by 2016. The 2 questions about content appropriateness were first asked in 2006 and remained unchanged in all 16 SAMs through 2016.
Measures
Statements of content appropriateness were rated on a numeric scale of 1 to 6 (from 1, strongly disagree to 6, strongly agree) for most SAMs. For the 4 SAMs (Pain Management, Well Child Care, Health Behavior, Maternity Care) that employed a rating scale of 1 to 4, we averaged the ratings using the raw scores and then weighted the mean ratings by a factor of 1.5 to align them with the 6-point scale. Mean ratings of the statements were used to measure FPs' opinions about content appropriateness of individual SAM KA/CS. Overall opinions about KA/CS were measured by aggregating mean ratings across all SAMs to KA/CS level. In addition, age at first SAM completion was used to establish cohorts of FPs aged under 40 years, 40 to 49 years, 50 to 59 years, and 60 years and older. Total number of SAMs completed during the study period was counted for each physician.
Analysis
We calculated the percentage of feedback surveys completed by SAM topic and by year. To test survey response bias, we examined whether respondents and nonrespondents differed in age, gender, medical degree (MD vs DO), and International Medical Graduate (IMG) status using data from 2015, the most recent year with a full year's data. We compared the differences within each SAM rather than across SAMs within in a calendar year because it became difficult to differentiate respondents from nonrespondents when a physician could complete multiple SAMs in a year but responded to any or none of the surveys. We examined mean ratings of content appropriateness for each SAM KA/CS topic over the years of its offering. Lastly, we investigated differences in overall opinions about content appropriateness of KA/CS by plotting the aggregated mean ratings against the total number of SAMs completed and by physician age cohort. The population of all SAM feedback surveys completed between 2006 and 2016 were available for analysis, mitigating the need for inferential statistical testing. Analyses were conducted using SAS 9.4 (SAS Institute Inc., Cary, NC) and Microsoft Excel. This study was approved by the American Academy of Family Physicians Institutional Review Board.
Results
Over the 11-year period, 448,408 feedback surveys (from 633,198 SAMs) were completed for a response rate of 71%. The number of SAMs completed each year by topic can be found in the Appendices. The response rates before 2009 were above 85% for the 10 SAMs that were available and, expectedly, declined after the requirement for CME credits was removed (Table 1). After reaching the lowest point in 2012 when only 56% of SAM feedback surveys were completed, the response rates started to increase. Of all the SAMs, the mean response rate was highest (77%) with the Asthma SAM and lowest with the Preventive Care SAM (59%) during the study period.
Response Rates to American Board of Family Medicine (ABFM) Self-Assessment Module (SAM) Feedback Surveys 2006 to 2016
Compared with nonrespondents, feedback survey respondents in 2015 were not significantly different in gender, medical degree, or IMG status for 10, 13, and 9 of the 16 SAMs respectively (see complete results in the Appendices). In contrast, the mean age was higher among respondents for all the 16 SAMs except for Diabetes (45.9 vs 45.4, P = .09) and Preventive Care SAMs (44.5 vs 45.1, P = .06). The largest mean age difference was 3.1 years (Mental Health, 43.2 vs 46.3, P < .0001).
Within individual KA modules, changes in ratings over the years were minimal (eg, the largest mean difference was 0.3 between any 2 years) and all the KAs were rated higher than 4.5 (of 6) (Tables 2 and 3). The mean ratings for “Content is appropriate for my practice” and “Content was presented at an appropriate level” were nearly identical. Across modules, some KAs (eg, Coronary Artery Disease and Cerebrovascular Disease) were rated slightly lower than others (eg, Diabetes or Hypertension). Nevertheless, the largest mean rating difference in any given year was less than 0.5. The overall pattern of the ratings within and across modules held in CSs (Tables 4 and 5). Compared with KAs of the same topic area, most CSs were rated lower but only by a magnitude of 0.1 to 0.2, with the exception of the Maternity Care and Childhood Illness CS's, whose content appropriateness was rated approximately one half point lower than their KA counterparts.
Ratings of Content Appropriateness of American Board of Family Medicine (ABFM) Self-Assessment Module (SAM) Knowledge Assessment (KA) from 2006 to 2016
Ratings of Content Appropriateness of American Board of Family Medicine (ABFM) Self-Assessment Module (SAM) Knowledge Assessment (KA) from 2006 to 2016
Ratings of Content Appropriateness of American Board of Family Medicine (ABFM) Self-Assessment Module (SAM) Clinical Simulation (CS) from 2006 to 2016
Ratings of Content Appropriateness of American Board of Family Medicine (ABFM) Self-Assessment Module (SAM) Clinical Simulation (CS) from 2006 to 2016
Figure 1 illustrates differences in FPs' opinions about overall content appropriateness by charting mean ratings on a continuum of increasing number of KAs completed, by age at first KA. Regardless of the topic area, we observed that the ratings were lowest among FPs who had completed 1 KA but increased as they completed more KAs over time (2006 to 2016). This trend applied to both younger and older FPs. Nevertheless, the ratings appeared to be higher among older FPs and lower among younger FPs. The corresponding trends of CSs closely mirrored those of KA (Figure 2). Nonetheless, ratings of content appropriateness of CS were lower than those of the KAs.
Overall ratings of American Board of Family Medicine (ABFM) Self-Assessment Module (SAM) Knowledge Assessment (KA) content appropriateness by increasing numbers of modules completed and by age at first module completion.
Overall ratings of American Board of Family Medicine (ABFM) Self-Assessment Module (SAM) Clinical Simulation (CS) content appropriateness by increasing numbers of modules completed and by age at first module completion.
Discussion
Building on a decade's worth of Diplomate feedback, this study provides evidence that FPs generally value ABFM SAMs. Contrary to claims that MOC activities are irrelevant and unnecessary,9⇓–11 the content of all 16 SAM topics was regarded by most FPs to be appropriate for practice and presented at an appropriate level. The fairly stable ratings over the life cycle of each individual KA and CS further suggested that the ABFM's process for updating both components kept the content relevant to FPs' practice.21 As we found no evidence of large differences between responders and nonresponders and nearly 70% of completed SAMs had feedback surveys, these findings are representative of a majority of ABFM Diplomates.
Compared with FPs who completed fewer KAs or CSs, those who completed more activities tended to rate the content higher. That is, FPs' opinions about the content became more favorable as they engaged with more activities. Ensuring that the content continues to be relevant for practice is crucial for the success of MOC Part II activities, which are intended to serve as a lifelong learning platform for physicians. The consistently high ratings of content appropriateness over the past decade suggested that ABFM SAMs served their purpose well.
The lowest ratings reflected the opinions of those who completed fewer SAMs and were likely FPs who were newly introduced to MOC. This may imply that younger cohorts of FPs had different initial experiences with or expectations of SAMs specifically, and perhaps MOC in general. Therefore, listening to and addressing the concerns of physicians who just began their MOC journey may help boost participation and satisfaction over the long term. This finding also highlights the importance of continuing to monitor Diplomates' feedback. As a result of an analysis of these feedback survey data showing that most critiques of the SAM were associated with technical issues in the CS,18,22 in July 2016, the ABFM split the SAM into 2 independent activities: the KSA and the CSA. The function of soliciting FP feedback remains and will continue to assist the ABFM in evaluating the performance of its new KSA and CSA modules.19
Limitations
This study has several limitations. First, the rating scales were not uniform across all SAM topics. Although we aligned different scales to be comparable, the weighted mean ratings may have skewed physicians' opinions about SAM content that was originally rated on a scale of 1 to 4. Nevertheless, the rating patterns of these SAMs agreed with those SAMs on a scale of 1 to 6. Second, survey respondents tended to be older than nonrespondents which suggests a potential overestimation of the results as older FPs viewed SAMs more favorably than younger FPs. However, the age differences were small despite being statistically significant and would be unlikely to affect the main results in a significant way.
Conclusion
This study provides substantial evidence supporting the perceived value of ABFM SAMs to Diplomates. Over the past decade, Diplomates rated the content of ABFM SAMs as appropriate and presented at an appropriate level across 16 diverse clinical areas. Importantly, FPs who completed more SAMs tended to have more favorable opinions of the content than those who completed fewer. Continued monitoring of feedback will play a critical role in keeping the content of MOC programs relevant for physicians' practice.
Appendix
Numbers of American Board of Family Medicine (ABFM) Self-Assessment Modules (SAM) completed from 2006 to 2016
Demographic Differences between Self-Assessment Module (SAM) Feedback Survey Respondents and Non-Respondents in 2015
Notes
This article was externally peer reviewed.
Funding: none.
Conflict of interest: The authors are employees of the American Board of Family Medicine.
To see this article online, please go to: http://jabfm.org/content/32/1/79.full.
- Received for publication December 22, 2017.
- Revision received June 19, 2018.
- Accepted for publication June 20, 2018.
