Abstract
Background: Continuing growth of the research spectrum of practice-based research networks (PBRNs) creates a need (1) for new approaches to training clinicians in research protocols and (2) to standardize clinician data collection. Each existing training method has shortcomings when used in geographically dispersed PBRNs. We describe here the use and costs of Internet-based training in support of a research protocol across a PBRN consortium.
Methods: Clinicians in 4 PBRNs in the PRIME Net consortium participated in training for a study of acanthosis nigricans (AN). We compared results of pre- and posttraining assessments of knowledge and ability to correctly diagnosis AN. We also calculated costs for placement of the training on the Internet.
Results: Among 103 participating clinicians, statistically significant increases in knowledge acquisition were demonstrated for all but 2 of the individual topics and in total scores on the assessments. AN diagnostic sensitivity increased from 52% to 99% to 96% to 100%, whereas specificity increased from 70% to 96% to 74% to 97%. Total costs for the web aspects of the training were $3732.
Conclusion: The Internet can be an effective and feasible alternative method for training clinicians in support of PBRN research.
The last decade has seen an explosion of interest in practice-based research networks (PBRNs) as a mechanism for primary care, translational, and effectiveness research. The number of networks has grown from 1 to 2 dozen PBRNs in the 1990s to more than 100, based on recent estimates.1
Together with the growth in numbers of PBRNs and interest in their potential contributions to the research endeavor has come a parallel expansion in the scope of their research and research methodology. For many years, PBRNs have conducted primarily descriptive research with the aim of increasing understanding about the dynamics of primary care. More recently, designs featuring more complex descriptive and mixed-method research, clinical interventions, and clinical trials have been proposed and conducted.
Increasingly sophisticated research designs carry with them a greater need for training of research participants to assure fidelity to the research protocol. For example, there is a need to assure standardization of data collection among participating clinicians; consistency of clinicians’ subject classification and data collection across sites is an important element of internal validity in PBRN research. Without consistent subject classification across clinicians, there would be a risk of classification bias in study results.
In the past PBRNs have relied on written manuals, in-practice research coordinators, and/or central research assistants for the training of clinician network members. In a few cases where funding has existed, networks have brought clinicians to a central location to provide training. Each of these approaches to protocol training can be problematic, however. Written manuals provide consistency in content and are inexpensive, but may be limited in their ability to convey complex material or assess trainee comprehension without in-person training as well. Using in-practice research coordinators (practice staff designated to handle study coordination) can facilitate study acceptance but may inject inconsistency in the presented material, and a training method would still be required for the scattered in-practice coordinators. Central research coordinators can enhance consistency of training and provide interpersonal interaction to address questions, but this could become expensive in geographically dispersed networks. The option of bringing clinicians to a central location for training is not practical, with rare exception, because of the expense and the time commitment needed from clinicians.
A potentially attractive alternative is interactive web-based training for PBRN study protocols. This option would offer the advantages of its relatively low cost, consistency in training, increased ability to address complex material through interactive formats, capability to reach widely dispersed clinicians, and capacity to assess clinician comprehension. With iterative training/testing loops, a clinician can review and retest in areas of uncertainty. Web-based training would not provide interpersonal interaction, which can be helpful in identifying potential areas of confusion or in identifying future problems with the protocol, but this method would allow the clinician to take part in the training at their convenience.
We have been unable to identify any descriptions of web-based training for PBRN research. For the reasons outlined above, information regarding web-based training could be helpful in planning future PBRN research. Therefore we examined the application of web-based clinician training for a study conducted in a national consortium of PBRNs, the primary aim of which was to examine the process of conduct and assessment in the training.
Methods
Study Design
We conducted a descriptive study of the effectiveness of web-based training for a PBRN study, examining (1) knowledge acquisition by clinician network members and (2) cost and feasibility of this approach to clinician training. The particular training studied focused on the problem of subject classification and eligibility because it was a critical component of the research process for the parent study (described further below). This step in the research process also required clinical judgment, raising the possibility of classification bias if it was not standardized. The study protocol was approved by institutional review boards at each of the 4 networks’ sponsoring institutions.
Study Setting
The study was conducted in PRIME Net, a national consortium of 8 PBRNs that focuses on research addressing the health and health care of medically underserved populations.2 The 8 networks include Research Involving Outpatients Settings Network (RIOS Net, New Mexico); Colorado Research Network (CaReNet); Southeast Regional Clinicians Network (SERCN, 11 Southeastern states); Southern Primary Care Urban Network (SPUR-Net, Houston); Collaborative Research Network (CRN, Northern California); Southwestern Ohio Area Research Network (SOAR-Net); Metro-Net (Detroit); and LA-Net (Los Angeles). The clinician members of these networks predominantly serve minority communities and practice in the public sector. Four of the networks (RIOS Net, CaReNet, SPUR-Net, and the SERCN) participated in this study, which was part of a larger study of acanthosis nigricans (AN) and clinician behavior change.
Sample and Intervention
As part of the overall study, participating clinicians from each of the 4 networks were asked to report on attributes of patients seen in their practices over a 1 to 2 week data collection period. Each participating clinician was provided CME credit for reviewing the module, and consented to participating in the study. The presence or absence of AN, a skin condition associated in many cases with hyperinsulinemia, was of particular interest for assessment and notation. To prepare clinicians to recognize and diagnose AN, a web-based training module was developed by the PBRNs. The primary purpose of the module was to enable and establish participating clinicians’ ability to diagnose AN. The module included background information about AN (appearance, classification, usual anatomic locations of the lesions, association with metabolic parameters, possible management strategies after a diagnosis of AN) and a number of photographs of AN. After the didactic portion of the module, clinicians were asked to complete an assessment of their understanding of AN, including correctly diagnosing several photographs as either showing AN or not showing AN (the complete training and assessment module can be viewed online.)3 Because the validity of clinician diagnosis of AN was going to be an important element of the study, each clinician scoring less than 100% on the assessment was asked to review errors and retake the assessment. We planned to use a sample of 110 clinicians in this study. For other elements of this component of the overall study (overall protocol, data collection and management, etc) we used more traditional clinician training processes, including written materials/manuals and central research coordinators.
Data Collection
Each participating clinician was assigned a log-in identity that allowed them to access the training website. We then collected the following data elements from clinician interactions with the website: time spent reviewing materials, overall score per assessment attempt, and numbers of assessment attempts required to achieve 100% score. For individual items on the assessment, we collected data on the percent of correct responses on the first attempt and the number of attempts required for all clinicians to achieve a correct response. Finally, through the use of a pretest, we gathered information about each clinician's familiarity and previous practice with AN. The latter step allowed us to compare clinician understanding before and after completing the module and to examine the acquisition of content knowledge. To assess the feasibility of this approach to clinician training we collected data related to the development and conduct of the web-based training, including costs of software development and fielding. We did not gather data related to the costs of the module's content development because these costs would be “fixed” regardless of the method of training.
Data Analysis
We used standard descriptive statistics to analyze the results of the assessments. In comparing pre- and posttest scores we used the exact McNemar test for individual questions and the nonparametric Wilcoxon signed rank test for total scores.
Results
Sample
Of a planned sample of 110 clinicians, 103 completed the training, including 21 from RIOS Net, 22 from CaReNet, 31 from SPUR-Net, and 29 from SECRN. The principal reason for nonparticipation was the overall timing of data collection in the parent study. Clinicians spent a median time of slightly more than 17 minutes on the training and assessment module, including pre- and posttests, as determined from website log-in times.
Pretest Findings
Ninety-one percent of the clinicians reported some familiarity with AN before the training (a result that may have been influenced by general information received about the parent study before beginning the training). As shown in Table 1, most clinicians were aware of the association of AN with hyperglycemia, hyperinsulinemia, and risk for diabetes. In contrast, only a minority of the clinicians were aware of the association of insulin levels with the appearance of AN.
Comparison of Percent of Clinicians Correctly Answering Questions about Aacanthosis Nigricans (AN) Before and After Web-based Training Module
Clinicians viewed 10 photographs of AN or other skin conditions with similar appearance as part of the pretest; the percent of correct identification of individual photographs as AN/not AN varied from 52% to 99% (Table 2). Clinicians who responded on the pretest that they were familiar with AN scored no better than those who reported no familiarity with AN (P = .89). Lower percentages of clinicians correctly diagnosed AN when it was atypical in appearance or present without pigmentation in a patient with light-colored skin (photographs 6 and 7). Mistaken diagnoses of a non-AN condition as AN were most likely when the condition presented with areas of hyperpigmentation on the anterior neck.
Percent of Clinicians Correctly Identifying Photographs of Dermatologic Conditions as Acanthosis Nigricans (AN) or Not AN
Posttest Findings
After the training, the clinicians took the same set of knowledge questions regarding AN, together with a new set of photographs of AN/non-AN skin conditions (no feedback was provided to clinicians about their responses on the pretest). The principal changes noted from the pretest to each clinician's first attempt on the posttest were in the higher percent of clinicians aware of the relationship between levels of insulin and grade of AN (34% vs 63%; P < .001) and the relationship of decrease in size/grade of AN with decreased insulin levels (53% vs 88%; P < .001). A substantially larger percent of clinicians were also aware that AN could be treated pharmaceutically after the training than before the training (27% vs 68%; P < .001). On the other hand, there was a slight decrease in the percent correctly responding that AN could appear as a velvety skin patch, a dirty neck, or an area of hyperpigmentation on the neck (89% vs 82%; P = .06) and in the percent who responded that AN could be associated with both hyperinsulinemia and hyperglycemia (65% vs 48%; P = .003).
The percent of correct responses to the AN/non-AN photographs on the clinicians’ initial taking of the posttest varied from 74% to 100%. The AN pictures were correctly diagnosed 96% to 100% (sensitivity), whereas non-AN conditions were correctly identified 74% to 97% (specificity). The 2 non-AN rashes most commonly misdiagnosed as AN were both conditions with hyperpigmentation of the anterior neck. As compared with the pretest, AN in a person with light-colored skin was more often correctly identified by the clinicians (52% vs 97%; photograph 8).
Overall Scores (Comparison of Pretest and Posttest)
Out of a maximum possible score of 20, the median score on the pretest was 15 (range, 10–20) and the median score on the initial take of the posttest was 18 (range, 11–20). Although only 15% of the clinicians achieved a 100% score on their initial posttest, the median number of times clinicians needed to take the posttest to achieve a perfect score was 2 (range, 1–9; median, 1 retake). There was a statistically significant improvement in the paired pretest/initial posttest scores (Wilcoxon signed rank test; P < .001).
Costs
Programming for the web module, including both the training and assessment portions, required an estimated 62 hours. Data management for the assessment results required an additional 32 hours. Applying average hourly salary and benefits costs for these tasks ($39.70 at our institution), the total cost for implementation of this web-based clinician training was $3732, or $36.23 per clinician trained, based on participation of 103 clinicians.
Discussion
This study of web-based training of clinician members of 4 PBRNs provides evidence that this approach to training can significantly enhance short-term knowledge acquisition that is key to successful research completion. Comparing pre- and posttest assessments of clinicians’ understanding of AN′s relationships and providers’ ability to distinguish AN from other skin conditions with a similar appearance, we found significant increases in knowledge after the web-based training. These increases came despite many clinicians’ self-reported familiarity with AN and despite relatively high levels of correct responses to individual questions on the pretest. Perhaps most importantly, considering the subject classification goals of the parent study, after the training the sensitivity of clinicians in diagnosing AN was 96% to 100% and the specificity of their diagnoses was 74% to 97%. These figures are comparable to the previously published sensitivity and specificity of a similar group of clinicians’ diagnosis of AN after in-person training (96% and 94%, respectively).4
Two items on the assessment demonstrated a counterintuitive decrease in percent of correct responses from pre- to posttest. In examining the possible causes for this, we determined that the training module had been unclear about the material on which each question was based (association of AN with elevated glucose and insulin—the module emphasized relationship to elevated insulin but did not clearly mention an association with elevated glucose; appearance of AN as a velvety patch, dirty neck, and/or patch of hyperpigmentation—the module mentioned only the velvety appearance and specifically mentioned that pigmentation may be absent in fair-skinned persons). These oversights in the training module probably explain the drop in percent of correct responses on these 2 items.
In addition to the findings related to short-term knowledge acquisition, we found the costs of the web-development portion of the training to be relatively low. Compared with the costs of alternative methods of training, these costs of web-based training would probably seem lower than those associated with developing in-practice research coordinators as trainers, with central research coordinators conducting the training in the practices, or with clinicians traveling to a central location for training. Considering the challenges that PBRNs often have in supporting infrastructure costs of operation, these lower costs could be quite important and deserve further study.
Compared with using a written manual as the sole training device, the costs for web-based training are probably higher. Actually, some aspects of web-based training, as conducted in our example, are very similar to manual-based training with its emphasis on didactic presentation of study material. However, the interactive potential of web-based training offers substantial advantages over manual-based training, as exemplified by our iterative assessment/feedback/review cycles.
Implications
As PBRNs expand in number and scope of research there will be an increasing need for training of geographically dispersed clinicians who are members of the networks. Cost considerations as well as complexity of research designs mandate that new models of training clinicians be developed to complement those used previously. Innovative uses of web technology for training busy clinicians have the potential to address some of the difficulties associated with on-site, in-person training, conducted either in the clinician's office or at a central location.
Comparison with Previously Published Literature
We were unable to identify any prior systematic analysis of the effectiveness of training methods for clinician members of PBRNs. There are, however, a number of studies of the use of the Internet for presenting continuing medical education, an area that is both growing in its use and, in its concepts, comparable to training clinicians for research.5 Fordis et al6 conducted a randomized controlled trial of the efficacy of Internet-based continuing medical education compared with live, interactive continuing education workshops and with a control group receiving no training. Targeting cholesterol management education, this research demonstrated “comparable or superior” changes in knowledge and clinical practice measured at 12 weeks after the training for those trained online versus those trained in-person. Several studies have compared online continuing medical education to no intervention, demonstrating increased knowledge, attitudes, confidence, and/or satisfaction with the online training.7–19) A meta-analysis of Internet-based health professions education (including, but not limited to, practicing clinicians) demonstrated comparable effectiveness in knowledge, skills, and behaviors compared with traditional instructional methods.20
Limitations
It is important to bear in mind a number of possible limitations of this study. First, we report the results of a single example of web-based training for PBRN research focused on one aspect of a complex study. It is possible that, under different circumstances, our finding of effectiveness in knowledge acquisition and cost feasibility may not be confirmed. However, the fact that this training was conducted in 4 networks enhances the potential generalizability. Second, we tested short-term knowledge acquisition and did not assess longer-term retention of knowledge gained, nor did we assess skills or behavior change resulting from the training (ie, independent validation of clinician diagnoses of AN in subsequent patient encounters). Third, we did not perform a controlled trial of web-based training with more traditional training methods for clinicians participating in PBRN studies. It is possible that one of these more traditional training methods would have had greater impact on knowledge acquisition; however, this seems unlikely considering the pattern and level of correct responses on our posttest. Finally, we chose a topic for web-based training that the clinicians were relatively familiar with, so that basic constructs in the topic area did not have to be established in the clinicians’ minds. It may be that topics that are less familiar or more complex would not have had similar outcomes from the training.
Future Research
The results of this study, together with the limitations noted above, suggest directions for future research in this area. Examination of web-based training for PBRN research using more extensive or complex topics or in a wider variety of PBRN settings would be helpful. Other aspects of the PBRN research process, such as consenting procedures, data collection and management, blinding procedures, and standardized interactions with patients may all be areas in which web-based training could be studied. Validation that knowledge and skills acquired during training are correctly applied to data collection in PBRN research will also be an important next step. Furthermore, controlled comparison of the outcomes of web-based training with alternative modes of clinician training would increase our understanding of the role of this approach in PBRNs.
Conclusions
The results of this study of the use of web-based clinician training in support of clinical research conducted in a consortium of primary care PBRNs show that, as measured by short-term knowledge acquisition and cost feasibility, this is a useful alternative to traditional methods of training. We found significant increases in clinicians’ ability to differentiate findings that determined research subject eligibility while demonstrating increased understanding of the clinical condition under study. Considering the critical role that clinicians have in PBRNs in study group assignment, this study shows the utility of web-based training at a time of rapid expansion of the numbers and expectations of PBRNs. Further research is warranted to test other web-based research training applications for PBRNs.
Acknowledgments
We wish to acknowledge and thank each of the RIOS Net, CaReNet, SPUR-Net, and SERCN clinicians who voluntarily took time to participate in the PRIME Net acanthosis nigricans study, including the training and assessments reported here, as part of their efforts to identify new ways to provide the best care to their patients. We also acknowledge Garth Colasurdo for programming the training and assessment modules; Robert Rhyne and Michael Crouch for assisting with developing the training and assessment materials; Gina Urias-Sandoval and Gina Cardinali for their assistance with the study; and Anthony Adams for information systems support.
Notes
This article was externally peer reviewed.
Funding: This project has been funded in whole or in part with Federal funds from the National Institutes of Health under contract no. HHSN268200425211C, “Re-Engineering the Clinical Research Enterprise,” and from the Health Resources and Services Administration, under grant no. D54HP00032.
Conflict of interest: none declared.
See Related Commentary on Page 348.
- Received for publication January 30, 2009.
- Revision received March 27, 2009.
- Accepted for publication March 30, 2009.
References
In this issue
Jump to section
Related Articles
Cited By...
- Cultivating a Cycle of Trust With Diverse Communities in Practice-Based Research: A Report From PRIME Net
- Practice-based Research Networks (PBRNs): Meeting the Challenges of the Future
- Reports of Persistent Change in the Clinical Encounter Following Research Participation: A Report From the Primary Care Multiethnic Network (PRIME Net)
- No Longer Simply a Practice-based Research Network (PBRN): Health Improvement Networks
- Acanthosis Nigricans: High Prevalence and Association with Diabetes in a Practice-based Research Network Consortium--A PRImary care Multi-Ethnic Network (PRIME Net) Study
- Increasing the Role of Practice Networks in Medical Research
- Fourth Journal of the American Board of Family Medicine Practice-based Research Theme Issue