Abstract
Background: Medical certifying boards' core mission is assuring the public that Diplomates have the requisite knowledge, skills, and professional character to provide high-quality medical care. By understanding their Diplomates' workforce and practice environments, Boards ensure that certification is relevant to the profession and accountable to the public. Current and reliable data are key to meeting this function. The objective of this article was to describe American Board of Family Medicine (ABFM) data collection procedures and demonstrate the capacity to compare cohorts by examination year.
Methods: We used data from ABFM examination application practice demographic questionnaires from 2013 to 2016. Descriptive and bivariate statistics assessed variation in Diplomate and certification candidate characteristics across examination cohorts.
Results: From 2013 to 2016, 55,532 family physicians applied for either initial certification (n = 15,388) or to continue their certification (n = 40,144). Diplomate characteristics varied slightly from year to year with more International Medical Graduates and fewer men in later cohorts but, these differences were not large between cohorts. Initial certification candidates were more likely to be women, and racial or ethnic minorities than Diplomates seeking to continue their certification, and each year's cohort was characterized by increasing numbers of female and US medical graduates.
Discussion: Data collected from Diplomates as part of examination registration have proved invaluable to serving the mission of the ABFM and advancing knowledge about the specialty of family medicine. Continued refinement of data collection to enhance data reliability and usefulness, while reducing collection burden, will continue.
The stated purposes of the American Board of Family Medicine (ABFM) include improving the quality of medical care, establishing and maintaining standards of excellence in Family Medicine, improving the standards of medical education in Family Medicine, and determining the fitness of specialists in Family Medicine to hold ABFM certification. For decades, the ABFM has used data to help guide these purposes. Examples include the use of examination performance data to help the Accreditation Council in Graduate Medical Education (ACGME) accredit family medicine residencies; the use of demographic data obtained during the examination application process to determine the readiness of Diplomates to use the internet to begin participation in Maintenance of Certification for Family Physicians (MC-FP) program in 2003; and the collection of race and ethnicity data to test the fairness of examination questions.1
Data collected by the ABFM are largely a byproduct of its business operations. Examination scores and results are obvious examples, but certification activity, residency, and fellowship training, feedback surveys on certification activities, and licensure and disciplinary actions are also collected. For over 30 years, the ABFM has required a practice demographic questionnaire to be completed by Diplomates when they apply for the recertification examination. These data were collected on “bubble sheets” (Online Figure) until the advent of MC-FP in 2003, when the ABFM began collecting these data online. Collection of these data are an important business requirement as the Board of Directors use the data to guide their policy decisions.
Beginning in 2008, the ABFM sought to better understand the practice characteristics and scope of practice of its Diplomates by partnering with the Robert Graham Center to conduct secondary research using the data from the practice demographic questionnaire and other administrative data. This collaboration has produced over 50 peer-reviewed publications and has documented a decline in scope of practice of family physicians,2⇓⇓–5 high participation rates in MC-FP,6⇓–8 and growing use of electronic health records.9,10 In 2012, the ABFM launched its own research department to better understand the effectiveness of11,12 and satisfaction with13,14 its certification programs and the practice and person characteristics associated with family physicians delivering high-quality care.15⇓⇓–18
The interval between recertification examinations has changed from 7 to 10 years due to changes in MC-FP requirements and a change to a Continuous Certification program but well-defined cohorts of physicians should apply for the examination at known intervals. In effect, the examination application practice demographic questionnaire constitutes a census of recertifying family physicians for that year. Therefore, the objective of our study was to compare the demographic characteristics of recertifying family physicians over the last few years and to describe ABFM data collection.
Methods
ABFM Examination
The ABFM has offered its examination twice per year since 2003. In 2012 the summer examination was moved to April and is now considered the spring administration of the examination. Registration for the examination, of which the examination application practice demographic questionnaire is a mandatory component, occurs 3 to 4 months before the actual examination.
Examination Application Practice Demographic Questionnaire
The current questionnaire asks about practice content and scope of practice, practice organization and structure, presence of other health care professionals, and use of electronic health records. All Diplomates seeking recertification receive the same core questionnaire. In 2014, the ABFM added 4 rotating question sets to sample physicians on topics relating to meaningful use of electronic health records, patient centered medical home features, types of payment their practices accept, and procedures performed. In 2016 a fifth question set on physician wellness and burnout was added. The sampling strategy is a round-robin style, whereby the first registrant receives the first question set, the second the second, the third the third, the fourth the fourth, the fifth, then the order starts again with the sixth person receiving the first question set. This permits additional data collection while reducing burden.
Beginning in 2014, residents in their final year of residency were required to complete a practice demographic questionnaire when registering for the examination. For spring examinees, these data are collected in December to February of their third year. Hence these questions mirror those on the questionnaire for practicing physicians but ask about intentions to perform procedures and clinical activities rather than actual provision of activities.
Data
We obtained examination registration questionnaire data from 2013 to 2016 from ABFM databases. Diplomate characteristics were obtained from ABFM administrative databases and linked to examination registration and result.
Analytic Strategy
We used descriptive statistics to characterize each examination cohort by year and administration. Bivariate statistics, χ2, and t-tests assessed variation in Diplomate characteristics by examination cohort. To test whether the rotating question sets were producing representative samples of all recertifying physicians, we tested for differences in characteristics of examinees by question set versus all examinees from 2014 to 2016 using bivariate statistics. All analyses were performed using SAS version 9.3 (Cary, NC). The American Academy of Family Physicians Institutional Review Board approved this study.
Results
The number of recertification examinees from 2013 to 2016 varied from 9,000 to 11,000 and the number of certification examinees varied from 3700 to 3900 (Table 1). From 2013 to 2016, the differences in Diplomate characteristics were negligible with average board exam scores between groups only differing by 13 points (reported on a 200 to 800 scale), average age varying less than 1 year, and racial and ethnic categories varying by less than 3%. While the mean age difference of initial certification candidates was statistically significant, the difference was only 1.7 years between cohorts and has little meaningful difference. Noticeable trends in residency graduates included lower percentages MDs versus DOs and International Medical Graduates. Residency graduates did not vary much by year in the percentages in racial and ethnic categories but the percentage of white race varied dramatically from that of all recertifying Diplomates (mid 60s vs high 70s). Residency graduates were also more likely to be Asian compared with recertifying family physicians.
Demographic Characteristics and American Board of Family Medicine Certification Examination Score and Result of all Examinees by Year, 2013 to 2016 (n = 55,532)
Diplomate characteristics between the 5 rotating question sets were only significantly different for mean age, percent passing, gender, and race. However, these differences were negligible and likely not meaningful (Table 2). For example, age varied by less than 0.5 years between the question sets. The statistical significance of racial differences is likely due to the introduction of the “other” category in 2016, which was associated with a reduction in the percentage self identifying as white. The lower numbers in the first question set about use of electronic health records was due to Diplomates being selected for this question set but then indicating they did not use an electronic health record on the questionnaire.
Demographic Characteristics of American Board of Family Medicine Certification Examination Recertification Examinees by Rotating Question Sets 2014 through 2016 (n = 27,325)
Discussion
Using 4 years of ABFM data, we found small variations in characteristics of examination candidates seeking to continue their ABFM certification. Females, DOs, and US medical graduates are becoming more frequent in the graduating resident cohorts applying for ABFM certification. These cohorts of family physicians are largely representative and reliable samples of the overall population of family physicians.
With programmatic changes, Diplomates may take the test every 10 years, which will lengthen the time between data points. Further, the ABFM is exploring examination alternatives that will also impact the numbers of physicians in the examination registration cohorts. The ABFM intends to collect data from Diplomates apart from examination registration, with the first iteration of this data collection strategy being the joint Association of Family Medicine Residency Directors/ABFM National Graduate Survey.19 The Graduate Survey is not mandatory but it is designed to help the ABFM better understand physicians early in their careers as well as inform residencies about adequacy of training. Residency level reports with national comparisons are provided to residencies each year. This partnership also shows how certifying boards can use the data they collect from their Diplomates to inform the ACGME and residencies on the outcomes of training.20
Using these practice demographic data collected from its Diplomates and analyses of feedback data from its quality improvement activities,13 the ABFM knew that Diplomates were struggling to obtain accurate whole-panel quality data and meet increasing reporting requirements. In response, the ABFM developed the PRIME registry to help Diplomates turn their data into information that will help with population management, identifying quality gaps, and to report their data where needed.21 These data can be combined with examination registration data and examination performance to better understand what practice arrangements and tools are associated with the physician being able to deliver high-quality care.
In recognition that the ABFM has a limited capacity to use all the data it collects to better understand the specialty, we created an external collaboration process by which other researchers can access ABFM data. Such data sharing is only done after vetting by the ABFM research team and executives, ethical approval at the investigator's institution, and executing a data sharing agreement. Only deidentified data are shared with external investigators. This process has extended the utility of the ABFM data and lead to deeper understanding of residency characteristics associated with graduates providing maternity and prenatal care,22 and buprenorphine prescribing.23 Other collaborations have informed practices' involvement with patient engagement24 and whether ability to deal with social determinants is associated with physician burnout.25 The ABFM has also partnered with the Council of Academic Family Medicine Educational Research Alliance to conduct a unique prospective medical education cohort study that will merge data from a special residency program director survey with future graduate survey data.
The major limitation of our study is the use of self-reported practice and demographic data. To validate ABFM data, state-level estimates of electronic health record use by family physicians was compared with estimates from the National Ambulatory Medical Care Survey. The estimates were highly comparable, which supports the validity of these data.9 Future risks to data collection include potential changes to the certification program that would further space out or even remove examination requirements for recertification, making reliable estimates of Diplomate characteristics difficult to obtain. To mitigate this risk, the ABFM has already begun collecting data more frequently apart from examination registration.
In conclusion, the ABFM is increasingly more strategic about its data collection with a primary goal of helping the organization accomplish its mission. In particular, the ABFM uses these data to enhance and modify its continuous certification paradigm. Finally, these data are proving invaluable in understanding the changing landscape of the specialty and creating options for the ABFM and its sister organizations to study and support ways to reduce Diplomate burden. As the specialty continues to evolve, the ABFM will use these data to further refine the assessment tools used to continuously certify family physicians.
Appendix
Notes
This article was externally peer reviewed.
Funding: none.
Conflict of interest: All authors are (or were) employees of the American Board of Family Medicine.
To see this article online, please go to: http://jabfm.org/content/32/1/89.full.
- Received for publication May 4, 2018.
- Revision received July 17, 2018.
- Accepted for publication July 19, 2018.
References
In this issue
Jump to section
Related Articles
Cited By...
- Regional Variation in Scope of Practice by Family Physicians
- Rural Family Physicians Are More Likely to Collaborate with Multisector Community Organizations
- A Few Doctors Will See Some of You: The Critical Role of Underrepresented in Medicine (URiM) Family Physicians in the Care of Medicaid Beneficiaries
- Machine Learning to Identify Clusters in Family Medicine Diplomate Motivations and Their Relationship to Continuing Certification Exam Outcomes: Findings and Potential Future Implications
- Small Independent Primary Care Practices Serving Socially Vulnerable Urban Populations
- Variation in Practice Patterns of Early- and Later-Career Family Physicians
- Early-Career Compensation Trends Among Family Physicians
- Maternity Access in Rural America: The Role of Family Physicians in Providing Access to Cesarean Sections
- Racial/Ethnic Minority Identifying Family Physicians Are More Likely to Work in Solo Practices
- Characteristics of Family Physicians Practicing Collaboratively With Behavioral Health Professionals
- The Association of Practice Type and the Scope of Care of Family Physicians
- Pandemic Disruption in Residency Did Not Alter Trends in Intended Scope of Practice
- Ready (or Not)--Here They Come
- Ready (or Not)--Here They Come
- Pandemic Disruption in Residency Did Not Alter Trends in Intended Scope of Practice
- Family Physician Income Disparities by Race and Gender
- Can Family Medicine's Counterculture History Help Shape an Anti-Racist Future?
- Family Physicians Increasingly Deliver Care in Diverse Languages
- One-Third of Family Physicians Remain in Independently Owned Practice, 2017-2019
- Practice Patterns of Family Physicians With and Without Addiction Medicine Board Certification
- Debt of Family Medicine Residents Continues to Grow
- Factors Associated with Time Spent Practicing Sports Medicine by Those with a Certificate of Added Qualification
- The Essential Role of Family Physicians in Providing Cesarean Sections in Rural Communities
- A Cross-Sectional Study of Factors Associated With Pediatric Scope of Care in Family Medicine
- Proportion of Family Physicians Caring for Children is Declining
- The American Board of Family Medicine: Celebrating 50 Years of Continuing Transformation
- Team Configurations, Efficiency, and Family Physician Burnout
- Burnout Among Family Physicians by Gender and Age
- Practice Predictors of Buprenorphine Prescribing by Family Physicians
- Early-Career and Graduating Physicians More Likely to Prescribe Buprenorphine
- Prevalence and Factors Associated with Family Physicians Providing E-Visits
- Associations Between Burnout and Practice Organization in Family Physicians
- Declining Endoscopic Care by Family Physicians in Both Rural and Urban Areas
- Family Physicians' Contributions to Rural Emergency Care and Urban Urgent Care
- Conversations, Communication and Counseling Are Key Family Medicine Tactics to Improve Patient Health