Abstract
Introduction: Approximately 5% of the US adult population has Attention Deficient Hyperactivity Disorder (ADHD) that can negatively impact quality of life. Health care professionals report a need to increase their knowledge of and confidence in treating adult ADHD. The American Academy of Family Physicians National Research Network (AAFP NRN) collaborated with a panel of experts to create a web-based AAFP Adult ADHD Toolkit composed of resources to aid in the diagnosis, management, and treatment of adults with ADHD.
Objectives: Assess the impact of using an AAFP Adult ADHD Toolkit in a practice setting.
Methods: Ninety-seven primary and behavioral health care professionals from AAFP NRN practices (n=6) used the Toolkit for 17 weeks. Data on Toolkit use, usefulness, implementation, impact, and changes in knowledge and confidence were collected via pre-post and weekly surveys. Mixed methods, regression analyses, t-tests, and mixed ANOVA were used to assess change over time.
Results: Use of the Toolkit improved health care providers' knowledge by midpoint relative to baseline in areas related to treatment effects, side effects, and outcomes (3.6 vs 3.0; P = .004); existing ADHD resources (3.3 vs 2.9; P = .03); and management of ADHD in patients with comorbid conditions (3.2 vs 2.7; P = .01). By the end of the study, Toolkit use was associated with increased confidence in mental health and life history interview techniques (3.5 vs 3.0; P = .03); treatment options for ADHD with comorbid mental health disorders (3.2 vs 2.3; P ≤ .001); and treatment options for ADHD with coexisting substance use disorders (3.0 vs 2.3; P = .003). By the end of the study, most participants (n=47, 87%) reported the Toolkit addressed most of their needs related to diagnosis, treatment, and management of adult ADHD.
Conclusion: Availability and adoption of the Toolkit into the routine care of adults with ADHD measurably increased health care professionals' knowledge especially in those providers who regularly see adult patients with ADHD.
- Adult ADHD
- Attention Deficit Disorder
- Evaluation
- Family Physicians
- Mental Health
- Practice-based Research
- Primary Care
- Primary Health Care
- Surveys and Questionnaires
- Toolkit
Introduction
Attention deficit hyperactivity disorder (ADHD) is a prevalent condition in both children and adults, affecting about 5% of the adult population. Primary Care Practitioners (PCPs) are usually the first clinicians to be in contact with adults affected by ADHD. PCPs receive little training for diagnosing and treating adult ADHD and know little about the comparative effectiveness and cost-effectiveness of the diagnostic and treatment options for this disorder. Previous studies reported a low level of competence in diagnosing ADHD in adults, poor familiarity with existing diagnostic criteria, low awareness of existing provider and patient resources, and deficiencies in evidence-based medication treatment decisions for adults with ADHD among PCPs.1⇓⇓–4 Many family physicians reported they prescribe stimulants based on their own success, comfort level, and clinical judgment rather than evidence-based guidelines.5 This is not surprising, as there are no evidence-based primary care-focused treatment algorithms and practice guidelines for adult ADHD in the United States. While adult ADHD resources for physicians and patients exist, there is no comprehensive, accurate, and easily accessible “one-stop shop” resource on adult ADHD. PCPs report a need to increase their knowledge of the disorder and confidence in diagnosing and managing ADHD in adults through education, tools, and resources.
To address this need, we sought to use the consensus-based participatory approach to develop and evaluate a toolkit for facilitating comprehensive care for adults with ADHD. The evaluation was designed to assess the implementation of the toolkit in routine clinical care with minimal outside intervention. The objectives of this article are to report on (a) the methods used in toolkit development, and (b) the results of implementation and evaluation on provider-level outcomes (knowledge, attitudes, and practice/behavior), use, and perceived value of the toolkit.
Methods
This is a single-arm repeated measures intervention study conducted in 6 practices with 97 providers from November 1, 2019, through March 31, 2020. The study evaluated the newly developed toolkit for adult ADHD detection and management in primary care. The AAFP Institutional Review Board approved this study. All project participants provided informed consent at the onset of their participation.
Toolkit Development Process
The toolkit content was created through a comprehensive research review in conjunction with expert panel input. The project team reviewed prior studies and published literature, identified existing best practice guidelines, and performed a comprehensive audit of publicly available tools and resources. Simultaneously, a panel of adult ADHD experts was convened to provide individual and group input. Five family medicine physicians, 1 pediatrician, 1 psychiatric clinical pharmacist, and 1 adult patient with ADHD comprised the panel. The panel members were identified through the AAFP National Research Network and selected based on relevant expertise among individuals who volunteered. Panel members' input consisted of individual toolkit content review, discussions in 2 1.5-hour moderated virtual group meetings and a 1-day in-person meeting. For this project, panelists reviewed, ranked, selected, and organized the materials in an iterative review and development process. Themes and content from individually completed worksheets and documented panel discussions informed the toolkit's resources, materials, structure, assembly, and format. The resulting web-based AAFP Adult ADHD Toolkit (i.e., Toolkit) includes the contents shown in Figure 1. The full Toolkit is available free of charge to any user at https://www.aafp.org/patient-care/public-health/adhd-toolkit.html.
AAFP Adult ADHD Toolkit Content Overview: Sections and Sub-sections.
Toolkit Evaluation in Real-World Primary Care Setting
We evaluated the Toolkit in a practice-based setting using data from provider surveys and web analytics. The AAFP NRN recruited primary care practices that fit the following criteria: use the Toolkit for at least 17 weeks and complete a series of surveys developed by the research team. A purposeful sampling method was used to include participating practices for an equal distribution of small private practices, residency practices, and large health care systems. All providers from these practices were invited to participate. All participating providers gave informed consent and completed a baseline survey. The baseline survey assessed participant demographics, knowledge, confidence, and needs around various aspects of adult ADHD care. Practices received one-on-one training in the use of the Toolkit. We followed the practices for 17 weeks to monitor key aspects of Toolkit use, provider knowledge, confidence, and perceived value of the Toolkit. The research team distributed weekly provider surveys to monitor self-reported use of the Toolkit and collected web analytics data. At the midpoint (week 9) and the end of the implementation period, providers were assessed on when, how, and why they used the Toolkit materials and what parts they found most/least useful. Lastly, provider knowledge and competence were assessed using predefined domains.
Data Collection
The project team developed and administered online surveys which generated Toolkit evaluation data. The project team's IT department provided web analytics data. All surveys were distributed electronically through e-mail using Qualtrics (Provo, Utah).
Web analytics data were generated through an Adobe Analytics report for the stated evaluation period. The report included data for Visits, Unique Visitors, Page Views, and Downloads (where applicable) for each page and subpage of the Toolkit. In the context of this project, a visit is 1 individual visitor who viewed the Toolkit website and proceeded to browse. Total visits are reported as counts of all visits to the specified page, no matter how many times the same visitor may have been to the specified page during the specified time frame. Unique Visitors is the count of the number of unique individuals that visited a particular page during the specified time frame regardless of how many times they visited. Page Views is defined as the total number of views or visits to a particular page for the specified time period. Downloads indicate the number of times a specified document was opened in the non-html version and subsequently downloaded by a visitor. Only the project participants and the project staff had access to the web-based Toolkit during the evaluation period. The internal IP addresses from the project team's organization were excluded from Visits, Page Views, and Downloads data. None of these web analytics data were otherwise identifiable.
Data Analysis
Descriptive statistics (counts, percentages, and means with standard deviations) were used to report demographic, survey, and web analytics Toolkit use data as appropriate. Before the analyses, all participants were assigned to 2 subgroups: those who self-reported use of the Toolkit (“Users”) and those who self-reported they did not use the Toolkit (“Non-Users”) using cumulative survey data. For examining the characteristics of those who used versus did not use the Toolkit, we used binary logistic regressions. For the above-mentioned analyses, missing data were excluded on a case-by-case basis. For the assessment of the change over time in the knowledge and confidence between those who used the Toolkit and those who did not use the Toolkit, we used Mixed ANOVA. Data only from the participants who completed baseline, midpoint, and end of study surveys were included in the ANOVA analyses. For the knowledge and confidence assessments, average group scores on 1 to 5 point scale for each predefined domain were compared across 3 time points (baseline, midpoint, end of study) between the subgroups (used the Toolkit vs did not use the Toolkit). An α of 0.05 was used throughout the analysis. All analyses were conducted with SPSS 25 (IBM Corp. Released 2017. IBM SPSS Statistics for Windows, Version 25.0. Armonk, NY: IBM Corp).
Results
Sample Overview
Ninety-seven providers from 6 practices participated in the Toolkit evaluation and completed the baseline assessment. Five practices were described as large organizations with more than 15 practicing physicians in the clinic. Of these 5, 4 were affiliated with a University and 1 with a large state-wide health care organization. One practice was a small solo practice. All participating practices were located in urban areas in one of the following states: Georgia, Florida, New York, Indiana, Ohio, and Oregon, and geographically distributed in Southeast (n=2), Midwest (n=2), Northeast (n=1), and Pacific Northwest (n=1). Participating providers included 51 family medicine or general internal medicine physicians, 34 physician residents, 6 nonphysician clinicians, 5 mental and behavioral health specialists, and 1 pharmacist. Provider characteristics are presented in Table 1. All 97 participants completed the baseline survey, with 86 (87%) and 81 (84%) participants completing midpoint and end of study surveys respectively.
Baseline Demographic Characteristics of 97 Participating Providers
Care Patterns in Adult ADHD
Before beginning the study, participants completed a baseline survey where they reported that on average, they spent more than half of their time in direct clinical care (n=74, 87%). At the baseline, most respondents (n=64, 66%) estimated on average seeing approximately 1 to 6 adult patients with ADHD in a typical week. However, when asked to report weekly, a smaller number of providers reported seeing adult patients with ADHD (n=36, 37%) throughout the 17-week study period, averaging about 1 to 3 adult patients with ADHD in a week (range, 1 to 6). More than half of the study participants (n=61, 63%) reported that they did not see any patients with adult ADHD while in the study. The most frequent problems patients with ADHD had throughout the project as reported by the participants were related to prescription medication for ADHD and/or refills (39%), concentration/attention problems (9%), seeking a formal diagnosis of ADHD (8%), and other problems (19%).
Toolkit Use and Utilization
Web analytics reports indicated that the Toolkit and all subpages received 394 page views, of which Toolkit Home Page (47%) and Assessment & Diagnosis (18%) were the most frequently visited sections. The printable/downloadable handouts were downloaded 158 times, with the top 3 being on diagnosis and assessment and patient self-management (Table 2). Most of the total Toolkit page views (66%) and total resource downloads (54%) occurred in the first 4 weeks of the evaluation. After the first month, the Toolkit visits remained stable averaging 8% of all visits over the remaining months.
Adult Attention Deficit Hyperactivity Disorder (ADHD) Toolkit Web Usage Data
An analysis of survey data indicated that 52 providers self-reported use of the Toolkit (“Users”) and 45 self-reported they did not use the Toolkit (“Non-Users”). Postbaseline survey data were available for 45 Users and 26 Non-Users. On average, Toolkit Users reported they used it 1 to 9 times a week. Similar to the web analytics use, self-reported Toolkit use was the highest closer to the start of the study: 15 Users in the first week and 16 Users in the second week. The most frequent reason for using the Toolkit was self-education (29, 64%) and seeking guidance on diagnostic assessment tools and strategies (23, 51%). Having no patients with ADHD was listed as the top reason for not using the Toolkit.
Next, we examined the chartership of Users versus Non-Users. We hypothesized that seeing/having adult patients with ADHD, the number of patients with ADHD seen in a week at the baseline, and nonresident status in practice would be associated with the use of the Toolkit. The only significant variable associated with the use of the Toolkit was whether participants saw patients with ADHD over the evaluation period; if they did, they were 49.5 times more likely to use the Toolkit than participants who did not see patients with ADHD. The number of patients with ADHD seen in a week and the nonresident status did not contribute to the use of the Toolkit (Table 3).
Logistic Regression Results of an Association of Residency Status, Number of Patients with Adult Attention Deficit Hyperactivity Disorder (ADHD), and Having Patients with ADHD on the Use of the AAFP Adult ADHD Toolkit.
Impact of Toolkit on Knowledge and Confidence
In both Toolkit Users and Non-Users, improvements in knowledge levels were observed over time (Table 4). When comparing Users versus Non-User, significant improvements in knowledge were observed in the following categories: “Monitoring for treatment effects, side effects and outcomes” (midpoint 3.6 vs baseline 3.0; P = .004); “Existing resources for patients and clinicians about adult ADHD” (midpoint 3.3 vs baseline 2.9; P = .03); and “Management of ADHD in patients with comorbid conditions” (midpoint 3.2 versus baseline 2.7; P = .01), and at the end of study (3.4 vs 2.9; P = .02). There were no differences between Users and Non-Users in any of these categories.
Impact of Toolkit on Knowledge and Confidence among Toolkit Users and Non-Users
Improvements were observed over time in participants' confidence levels in all areas (Table 4). When comparing the end of study data to baseline data, the Toolkit Users reported significantly higher confidence levels than the Toolkit Non-Users in the following categories: “Mental health interview techniques and life history interview” (3.5 vs 3.0; P = .03); “Choosing treatment options for ADHD with comorbid mental health disorders” (3.2 vs 2.3; P < .001); and “Choosing treatment options for ADHD with coexisting substance use disorder” (3.0 vs 2.3; P = .003).
Perceived Value of the Toolkit in Addressing Provider Needs
When asked about the most pressing needs related to adult ADHD (Table 5), respondents indicated it is “better understanding for nonmedication management options, effectiveness and side effects” (n=42, 52%), followed by “better understanding diagnostic criteria and diagnostic process for adults” (n=33, 41%).
Needs Related to Adult Attention Deficit Hyperactivity Disorder (ADHD) Reported by Study Participants.
The Toolkit Users reported that in most cases the Toolkit addressed “some to most of their needs” related to adult ADHD (n=40, 78% at midpoint and n=47, 87% at end of study). Fewer respondents indicated that the Toolkit addressed “very few to none of their needs” (n=11, 22% at midpoint and n=7, 13% at end of study). At the end of the study, the majority of participants indicated they would benefit from additional adult ADHD resources easily accessible at the point of care and education related to adult ADHD (n=62, 87% and n=63, 86%, respectively).
Discussion
This article describes the development and evaluation of the AAFP Adult ADHD Toolkit, as well as implementation and evaluation results regarding provider outcomes on use, knowledge, confidence, and perceived value of the Toolkit. Recent studies highlight the importance of stakeholder engagement in the research process6, including resource development.6⇓⇓–9 Stakeholder input is more likely to produce materials that are pertinent and tailored to intended audiences versus those that do not incorporate such perspectives.7,8,10,11 For this study, we employed an eclectic expert panel (family physicians, a pediatrician, a psychiatric clinical pharmacist, and a patient) to develop the Toolkit. All members held a unique understanding of the topic, and they contributed to clinically relevant detail and/or the appropriateness of the material for those adults who have ADHD. Our stakeholder engagement approach ensured the creation of a comprehensive adult ADHD resource by using a thorough and iterative process of content material creation and review via individual and group input.
The results of our study indicated that the Toolkit shows promise in improving PCPs' knowledge and competence related to adult ADHD even over a relatively short duration. This is of particular importance as many studies have underlined the lack of adequate education and training PCPs' receive in adult ADHD.12⇓–14 This insufficient education and training have yielded lower PCP awareness, understanding and knowledge, and lack of confidence in diagnosing, treating, and managing adult ADHD.15⇓–17 The improvements observed in several other challenging areas, particularly in domains related to managing ADHD with coexisting conditions including substance use disorders, are especially encouraging. The knowledge and competence improvements are not surprising given that those who used the Toolkit, accessed it most in the first month post-introduction. In addition, the majority used it for self-education and seeking guidance on best practices. Interestingly, knowledge and confidence improved in both groups, including those who reported they did not use the Toolkit in their practice. Training on and access to the Toolkit were provided to all, potentially boosting the practitioners' confidence. Web analytics data also indicated that the participants may have visited the Toolkit initially without actually using it in practice. In assessing knowledge levels, we relied on self-reports and cannot confirm to what extent these levels would correlate with any objective measures such as test scores in either Users or Non-Users. It is unknown, however, whether once incorporated into care, the Toolkit results in optimal adult ADHD patient care. The “knowledge to practice” translation of this resource will need to be assessed in future studies.
Specific factors related to uptake of practice and provider resources are not fully understood. In general, the uptake of practice and provider resources has been slow, due mainly to the absence of available time, which has caused issues with being able to effectively treat a patient.18,19 Uptake can be enhanced when combining multiple strategies to grab the user's attention.20 More recent studies have pointed to the effectiveness of providing an online resource, such as a toolkit, to provide the necessary resources for providers, but also note that users are looking for specific materials and any online resource has to be usable for a wide audience.21,22 One of the hypotheses in this study was related to the resident status and the difference in how they might use the Toolkit. In total, 34 residents participated in this study which contributes to the relatively young mean age of our study group. The residents, however, did not use the Toolkit any differently than their more experienced nonresident colleagues. Our study shows that the use of the Toolkit was associated only with whether the providers were seeing patients with ADHD during the evaluation period. Having patients with ADHD may have determined the need for additional resources such as this toolkit. PCPs on average reported seeing about 1 to 3 patients a week. Interestingly, the number of patients with adult ADHD seen in a week was not associated with the use of the Toolkit and the use did not increase with having more patients. On one hand, this indicates that toolkits, such as this one, may be relevant to those providers who see and treat patients with a target condition and may not be universally utilized by all PCPs. On the other hand, the evaluation period was short, and it may be coincidental that some providers did not encounter patients with ADHD over several weeks of this project's duration. About two-thirds of participants reported seeing no patients with adult ADHD over the course the study. In addition to a short evaluation period, adult ADHD is often not detected and misdiagnosed in primary care. Low competence in recognizing and screening for ADHD in adults among the study participants could also potentially lead to underreporting of encountering patients with adult ADHD in some PCPs. Future studies of longer duration including patient outcomes related to screening and diagnostic assessment are needed to confirm our findings.
The uncertainty around the effectiveness of toolkits concerning improved clinical outcomes has been a question of debate in the literature. Evaluated toolkits have generally been found to have high user satisfaction23 and effective knowledge translation24 for clinical users, and similar studies that evaluate self-reported provider knowledge and confidence have found toolkits to have a positive effect.24 In addition, toolkits are a simple, flexible, and expedient knowledge transfer intervention making them a pragmatic tool for implementation with many organizations continuously investing resources in their development and dissemination.25 Our study showed that the AAFP Adult ADHD Toolkit addressed most of the providers' needs they stated, increase levels of knowledge and competence, and uncovered a need for additional education, training, and resources potentially beneficial for PCPs. Future studies should assess whether the Toolkit or its components are positively associated with clinical, safety, and care quality outcomes.
The implementation process shows that PCP participants had a sustained interest in this topic as an overwhelming majority completed the midpoint and end of study surveys. Though Toolkit use fell drastically after the first month, the portion of study participants continued visiting the Toolkit and downloading its resources through the end of the evaluation period. These findings suggest the content received in the first month of the study satisfied PCPs' most pressing needs for diagnosis and treatment. Even though we have not explored the actual reasons for the use patterns we observed, the lower use after the initial first month of the study may be related to a decreased need to visit the Toolkit after downloading the relevant information. Yet, it may prove beneficial to release Toolkit information at various times or add an interactive portion of the Toolkit to foster continued use.
Results of our study need to be interpreted with caution as they may not be generalizable to all practice types and contexts. This toolkit implementation and evaluation were conducted in only 6 practices. These 6 practices, although representing typical primary care practices, include several large academic practices that are likely to be more interested in adult ADHD. Further work is needed to formally test the implementation of the toolkit in a variety of practices. The Toolkit development may have also not fully considered the perspectives of internal medicine or clinical psychology as these specialties were not represented on the expert panel. These specialists however participated in the Toolkit evaluation, so their input is included in this work. This evaluation did not assess effectiveness for improving patient, clinical, and safety outcomes or quantify cost and resources for implementation. Due to the short implementation period, it was not feasible to assess the long-term effects or sustainability of Toolkit use. We did not explore the reasons why many providers reported seeing no patients with adult ADHD over the study period or why their weekly reports on patient encounters during the study did not align with the average numbers of adult patients with ADHD they reported before the study. We did not compare the Toolkit to other quality improvement methods. Furthermore, observations from the development phase identified evidence gaps, including resources on women, older individuals, breastfeeding, and medication management, for example, medication tapering, chronic use of stimulants, and medication holidays, among others. Therefore, our analysis does not include these particular topics due to a lack of evidence-based data. These information gaps would serve as additional points of information for Toolkit enhancements and upgrades. As far as the next steps, we plan to make the Toolkit available for PCPs and primary care practices so content materials can be distributed widely and improve on adult ADHD diagnosis and patient care needs.
Conclusions
We developed a multi-component comprehensive Toolkit to support clinical care of adult patients with ADHD for primary care and other providers. The Toolkit measurably increased PCPs' knowledge especially in those providers seeing adult patients with ADHD. The web-based Toolkit is an effective strategy to deliver much-needed information to practicing clinicians and their patients. The PCPs and their patients will also benefit from additional relevant education and an array of resources easily available at the point of care.
Acknowledgments
We are extremely grateful to our Expert Panel members whose hard work and dedication made this project possible. We thank the AAFP NRN for providing essential support and expertise.
Notes
Funding: This project was supported in part by an Independent Investigator Research Grant from Shire Human Genetic Therapies, Inc. Shire was not involved in the study design, data analysis, and interpretation or reporting of this work.
Conflict of interests: None.
To see this article online, please go to: http://jabfm.org/content/34/4/741.full.
- Received for publication November 20, 2020.
- Revision received February 3, 2021.
- Accepted for publication February 10, 2021.
