Abstract
Purpose: The American Board of Family Medicine (ABFM) approved the use of a new blueprint for the Family Medicine Certification Examination, the In-Training Examination, Family Medicine Certification Longitudinal Assessment, and the Continuous Knowledge Self-Assessment. It will go into effect in January 2025. The blueprint defines the content domains for the questions on the examination and the percentage of questions in each domain. This article describes the process used to establish the clinical activities that comprise family medicine and group them into content domains.
Methods: A list of family medicine clinical activities was generated from a list of family medicine entrustable professional activities, diagnostic codes from National Ambulatory Medical Care Survey data, PRIME registry data, a publication on skills and knowledge needed at the completion of residency, content topics from the National Board of Medical Examiners, previous ABFM item classification categories, and an in depth analysis from the International Classification of Primary Care diagnosis. The distilled list of clinical activities were grouped into content domains. Finally, the activities were reviewed by stake-holders to establish if the activities and their grouping were representative of the practice of family medicine.
Results: A 5-category content domain structure was adopted: Acute Care and Diagnosis, Chronic Care Management, Emergent and Urgent Care, Preventive Care, and Foundations of Care.
Conclusions: The clinical activities and their grouping into content domains were accepted as being representative of the practice of family medicine. Meeting this criteria made these activities and grouping an appropriate basis for a practice analysis survey.
Introduction
Creation of a foundational content outline for clinical knowledge assessment is an important step in defining the scope of a specialty. The American Board of Family Medicine (ABFM) examination blueprint was last updated in the early 2000s, and the dramatic changes in family medicine practice over the past 20 years require a review of the validity of the construct. A second goal is that a new blueprint should better support learning. The current American Board of Medical Specialties (ABMS) standards for continuing certification1 require that Diplomate feedback2 be instructive for identifying knowledge gaps. In adopting longitudinal assessment, ABFM has embraced both summative assessment of the cognitive expertise and formative assessment to facilitate Diplomate learning. The new blueprint will provide both more statistical precision within each of its content domains, and the domains will be more clinically relevant. Finally, the new blueprint will be based on a more robust practice analysis and includes an adjustment for the risk of patient harm. This article is the 1 of 3 articles related to the new blueprint. This article describes the development of the domains of care and the creation of a list of family medicine clinical activities. The second article describes the practice analysis and the creation of the final blueprint,3 and the third article will describe how ABFM will use the new blueprint.
Background
When creating a professional medical knowledge assessment examination, one uses a test specifications document which is often referred to as the examination blueprint. This blueprint, a top-level content structure, considers the content domains that reflect the scope of practice for that medical specialty. Next, weights or percentages must be defined for each of these content domains which specify the percentage of the test questions in each content domain. In family medicine, the blueprint specifications provide the macro-structure for crafting all examination forms that use the Family Medicine Certification – Scale (FMC-S). These examinations include the one-day Family Medicine Certification Examination (FMCE), the Family Medicine Certification Longitudinal Assessment (FMCLA), the In-Training Examination (ITE) for residents, and the Continuous Knowledge Self-Assessment (CKSA). All examinations have been constructed such that when an examination is built, the proportions of the content domains will be representative of the national practice of family medicine.
Since the founding of ABFM in 1969, Diplomates have been required to retake and pass an examination regularly to remain ABFM-certified. The examination has always been intended to measure family medicine-related, decision making ability and the requisite knowledge needed to make those decisions; however, the nomenclature and methods for organizing the content and for assigning the percentage of questions to each content domain have varied somewhat over the years.
In the 1980s, ABFM used a 3-dimensional approach (body system, etiology, and stage of disease) to create the certification examination.4 An additional structure using clinical categories for classifying questions was added later to provide another way for diplomates to connect their weaknesses to available educational materials. The clinical categories were comprised of: Internal Medicine, Pediatrics, Psychiatry, Behavioral Sciences, Obstetrics, Gynecology, Surgery, Community Medicine, and Gerontology.5
In 2006, the ABFM changed the examination blueprint domains from clinical categories to body system categories,6 reflecting the Case Western Reserve body-based curriculum design that had become dominant in medical school curricula. To create this blueprint, ABFM conducted a study using National Ambulatory Medical Care Survey (NAMCS) data and surveys of practice content to assign percentages to the body system content domains. This blueprint was composed of 14 body system domains and 2 additional ancillary content domains. The proportions for these 16 domains ranged from 1% to 13% of the examination. Consequently, some of the content domains had too few questions to serve as precise and useful feedback for examinees. This blueprint was used from 2006 to 2024. Although validity studies indicated that the ABFM 2006 body system-based blueprint represented the practice of family medicine,7,8 there were some International Classification of Diseases (ICD-9) diagnoses seen in the practice of family medicine which could not readily be connected to specific body system content domains in the blueprint.9–11 In these studies, ABFM also piloted a risk of harm methodology, in which practicing family physicians assessed the risk of harm if a particular diagnosis was not made correctly or if the clinical management was inappropriate. These studies demonstrated that assessment of risk of harm was feasible and scalable, providing the opportunity to include risk of harm in the new blueprint.
Described here is the process ABFM used to (1) create the 5 Domains of Care that form the backbone of the blueprint, (2) create a list of the fundamental clinical activities that family physicians most commonly perform (Family Medicine Clinical Activities list, FMCA), (3) group these clinical activities into the content domains, (4) get input from external groups regarding whether the domains of care and the clinical activities adequately reflect the scope of family medicine, and (5) devise survey rating scales to assess the frequency with which physicians perform the activities and to assess the risk of patient harm if the clinical activity is not performed correctly.
The FMCA list is intended to address what is included in the scope of family medicine blueprint. A separate article will address the percent of questions (how much) in the test are to be assigned to each content domain.3
Creating Domains and Activities
Step 1. Creating the Domains of Care
Inclusion of Entrustable Professional Activities (EPA) as Domains. The 2025 ABFM blueprint uses the Family Medicine EPAs12–14 from the Association of Family Medicine Residency Directors’ (AFMRD) as the foundation. We built on this Family Medicine EPA13 structure which was defined in the Family Medicine for America’s Health project. From this list of 20 EPA statements, we selected the 4 EPA statements that describe the broad types of care provided by family physicians and we used these EPAs as the 4 fundamental clinical content domains in the 2025 blueprint. These descriptions represent the types of clinical work that family physicians perform routinely in their offices and other settings such as hospitals, urgent care centers, or skilled nursing facilities. Other EPA statements were included in the fifth domain Foundations of Care (FOC) or were integrated into the FMCA list (both described below). Some EPAs were not appropriate for knowledge assessments as they are value statements, not observable activities.
The 4 Clinical Foundational Domains with brief definitions are:
Acute Care & Diagnosis-
the diagnosis and initial treatment of clinical scenarios commonly seen in ambulatory practice settings. We defined acute problems as those lasting <12 weeks.
Chronic Care Management-
the management of ongoing chronic disease and multimorbidity.
Emergent and Urgent Care-
the expedient diagnosis, stabilization, and treatment of clinical scenarios where time is of the essence and there is significant more risk of harm. We defined emergent problems as those that need diagnosis and management within 24 hours without respect to where the site of initial evaluation occurs and those clinical scenarios that are seen in the inpatient hospital setting.
Preventative Care-
the provision of primary prevention and health counseling which prevents disease and morbidity across a lifetime. We emphasized primary and secondary prevention and not tertiary prevention.
The fifth domain, FOC, was created to cover all other aspects of Family Medicine knowledge that are needed in the provision of care to patients and the community such as health law, statistics, health equity, and quality improvement science, etc. Because FOC topics cover aspects of knowledge that rarely contain uniquely observable clinical activities, they could not be assessed with a survey of frequency. Similarly, there are often no identifiable ICD diagnoses or symptoms associated with topics in this domain.
Several EPA statements were integrated as topics under FOC. They were:
Use data to optimize the care of individuals, families and populations.
In the context of culture and health beliefs of patients and families, use the best science to set mutual health goals and provide services most likely to benefit health.
Advocate for patients, families and communities to optimize health care equity and minimize health outcome disparities.
Provide leadership within inter-professional health care teams.
Step 2. Creating the FMCA List
The second step was to develop a list of clinical activities that describes family physicians’ clinical activities across the various settings in which they could see patients. We considered a very broad list of conditions, diseases and clinical situations from multiple sources that family physicians see and do in their practices. These activities were intended to represent the broad scope of family medicine. These clinical activities were then grouped into the 4 clinical content domains.
Inclusion of Some EPAs as Activities
Some statements from the Family Medicine EPAs13 described more specific aspects of care and as such were included as a clinical activity within the 4 clinical content domains of the blueprint. These EPA statements include:
Manage prenatal, labor, delivery and post-partum care.
Manage end-of-life and palliative care.
Diagnose and manage mental health conditions.
Other Sources of Clinical Activities
To capture clinical activities performed by family physicians in practice, we carefully reviewed and considered many different sources. This exhaustive list included 2 separate national datasets of recent National Ambulatory Medical Care Survey (NAMCS) data,9,10 PRIME15 registry data, a publication on skills and knowledge needed at the completion of residency that listed 76 core competencies,14 outlines of the National Family Medicine Curriculum16 published by the Society of Teachers in Family Medicine (STFM) and the Content Outline from United States Medical Licensing Examination17 (USMLE), a listing of examination content topics used internally by ABFM examination content development staff, and lastly an analysis of the International Classification of Primary Care18 (ICPC-3) diagnosis listing.
As the first step, a listing of several thousand ICD-9 and ICD-10 codes reported by primary care and family physicians from 2016 and 2018 NAMCS datasets, respectively, were analyzed and then grouped into common thematic areas and curated through an iterative process into a condensed list of approximately 180 topics or diagnoses. This formed our initial Family Medicine Clinical Activities list or the “working list of clinical activities.”
Next, PRIME15 registry data, using a listing of the 500 most frequent diagnoses encountered in primary care practices were analyzed, compared, and contrasted to this working list of diagnoses. These large data sets provided evidence of what family physicians were actually doing in their practices. New elements that were not already on the working list were identified and added. Next a seminal publication by Shaughnessy, Gravel et al.14 listing 76 ambulatory core competencies for family medicine residency completion and the National Clerkship Curriculum from the Society of Teachers of Family Medicine (STFM)16 and United States Medical Licensing Examination (USMLE)17 content topics were compared against the list of diagnoses and again any gaps were noted and added or refined to the working list. Next a listing of approximately 100 clinical topics used internally by ABFM editorial staff in previous examination question writing and examination construction were compared with the working list of activities. Again, new elements were identified and added to the working list. Finally, we analyzed the International Classification of Primary Care version 318 (ICPC-3) and again cross-referenced and made any needed modifications to the working list. The ICPC-3 review provided another check that the content of primary care was included in the final FMCA list.
Each clinical activity of this penultimate FMCA list was then assigned to the clinical content domain where it would most commonly occur. For example: the clinical activity noted as “provide care for Acute Chest Pain” was placed in Emergent and Urgent Care Domain. The nomenclature of each clinical activity was again reviewed for clarity and uniqueness. Each activity on the list was also modified to be expressed as an activity under each content domain category, such as “Perform or Provide Urgent and Emergent care for: Acute Chest Pain,” or “Perform or Provide Chronic care management of Heart Failure.”
Step 3. Validating the List of Clinical Activities
The next process was to validate the working list of clinical activities to assure it was representative of the broad scope of family medicine. For this, we conducted several focus groups. The intent was to establish that the coverage of the 4 clinical content domains had been adequately described by their activities. We conducted 2 teleconference meetings, each with 6 family physicians from a broad range of practices and geographic areas. Each participant was given the listing of clinical activities 2 weeks ahead of the meeting with the instructions that asked the following questions: (1) is the current activities list representative of the content domains and of family medicine in general?, (2) should any activities on the working list be deleted or merged into a single more general activity?, and (3) should any activities be added to the list? Each teleconference meeting lasted about 2 hours. From this information, the working clinical activity list was again refined to a list of 202 clinical activities grouped into the 4 clinical domains.
A blue-ribbon panel with representatives nominated by the American Academy of Family Physicians, the American Board of Family Medicine, the American College of Osteopathic Family Physicians, the Association of Departments of Family Medicine, the Association of Family Medicine Residency Directors, NAPCRG, and the Society of Teachers of Family Medicine was convened. The blue-ribbon panel met a total of 5 times and provided general input on the new blueprint and specific input reviewing the process for creating the 5 content domains of the blueprint and the process for defining the family medicine clinical activities list. Likewise, the ABFM Board of Directors provided input into the process and creation of this framework.
Step 4. Assessing Frequency and Risk of Harm
As an improvement on previous ABFM blueprints, we sought to inform the blueprint weights in a way that not only considered the frequency with which those activities were performed by practicing family physicians, but also the risk of harm to the patient if the activity was not properly performed, such as a missed diagnosis or if performed incorrectly. This is the first ABFM blueprint to consider this concept in how we weight the content domains of the examination. We used a 5-point, Likert19 scale (1-daily, 2-weekly, 3-monthly, 4-Few times per year, 5-Rarely) to create a frequency Index (FI) for each activity. Similarly, we used a 4-point Likert19 scale (1-Minimal, 2-Moderate, 3-Considerable, 4-Extreme) to create an Index of Harm (IoH) for each activity. The IoH8 had been used previously with ABFM blueprint validity studies. This data will allow the final realigned blueprint to more accurately include the importance of both FI and IoH in those activities.
Step 5. Drafting the Family Medicine Activities Survey (FMAS)
We then drafted the FMAS. The main body of the survey contained the 202 clinical activities with each activity being assigned to only one of the clinical content domains. For each activity, the raters were instructed to rate it for 1) the frequency of occurrence and 2) the risk of patient harm. There was also a brief demographics section. Because the survey would be administered via the internet, the survey was reviewed by staff for ease of use in an online environment.
Step 6. Pilot Study
We then conducted a pilot study of the FMAS with 67 family physicians from a broad range of practices in 2 states in small groups. We used the state AAFP chapters in Ohio and Florida to help recruit practicing family physicians. Demographic questions were asked of the participants to ensure we had a broad range of practices. After completing the FMAS, these diplomates participated in a 90-minute Zoom discussion to provide feedback about how well clinical activities represented the scope of family medicine and the user-friendliness of the survey. No additional changes were needed to the listing of the clinical activities, but minor adjustments to the format of the FMAS were made. The data collected from the pilot survey regarding the FI and the IoH scales was analyzed and reviewed to assure that the rating scale produced sensible results. The process and the results of the final FMAS are reported in detail in a separate article3.
Discussion
We used the Family Medicine entrustable professional activity statements from Family Medicine for America’s Health and the AFMRD to provide the foundation of the 2025 ABFM blueprint. Data from 2016 to 2018 NAMCS survey and the PRIME Registry identified the most common clinical activities of family physicians. We then cross checked this preliminary list of clinical activities with an extensive set of resources listed above in Other Sources of Clinical Activities. Many focus groups, chosen for breadth of practice type, reviewed the clinical activities and the overall process. The process and the results it produced were also reviewed and approved by a blue-ribbon advisory panel and the ABFM Board of Director’s Knowledge Assessment Committee. These Domains of Care will form the basis of our examination feedback and will also provide more precise diplomate feedback than the smaller categories of the previous blueprint.
In future blueprint updates, we can repeat this process to further refine our list of clinical activities and to update our frequency and risk of harm adjustments. Benefits of our process are as follows:
Significant and extensive external review to ensure a comprehensive process to create the most accurate content coverage in the FMCA and the blueprint.
For the first time we considered both frequency of occurrence and risk of harm that will lead to more clinically relevant weighting of the exam content areas.
The specific clinical activities within each of the 4 clinical content domains can be updated periodically as additional activities become evident in future practice analysis.
Provides a strong connection between what family physicians are doing and what will be on the examination.
Limitations
Inevitably, there is a subjective component to the definition of clinical activities. At many points in the process however, outside groups reviewed and added to the content. In addition, the NAMCS sample includes some nonfamily physicians and has limited numbers of practitioners; the PRIME registry includes the records for many more physicians and is a representative but nonrandom national sample. Using both contributes to validity of the list of clinical activities. Both NAMCS and PRIME data use ICD codes, which do not cover all the conditions family physicians see; nor do family physicians give diagnosis codes for all they do. Moreover, neither data set focuses on diagnoses, emergency rooms, inpatient medical wards or labor and delivery units, potentially skewing the list of clinical activities. However, using both data sets as well as other sources of activities and multiple focus groups for validation minimizes this bias. We initially planned to have several in-person discussions at medical conferences and engage with diplomates. This opportunity was not available due to the COVID-19 pandemic. However, we still were able to obtain input from diplomates across a wide geography and type of clinical practice by using technology and virtual meeting types. Some finetuning of the clinical categories may also be possible after we have a few years of experience using this blueprint.
This article describes the first stage of the development for the new ABFM blueprint. It identifies what we are including in the construct of family medicine. The second article describes how empirical support was generated for the percentage of questions that would be in each domain on future examination forms. This second article is about how much.
Notes
This article was externally peer reviewed.
Funding: None.
Conflict of interest: None.
- Received for publication December 30, 2024.
- Revision received February 27, 2025.
- Accepted for publication March 5, 2025.
