Abstract
Background: Melanoma incidence and mortality is a growing concern. Better recognition and management of skin cancer by primary care providers (PCPs) could help, but studies suggest they would benefit from additional education. Effective educational programs are needed.
Methods: We developed and conducted a voluntary before-and-after evaluation of a 1- to 2-hour interactive, web-based course in skin cancer detection for practicing, board-certified PCPs (http://www.skinsight.com/info/for_professionals/dermatology-education-resources). Voluntary participants' ability to diagnose and manage skin cancer was assessed using pretests, immediate tests, and 6-month posttests. The effect on actual practice patterns was assessed using participants' patient panels: referrals or visits to dermatology and skin biopsies during the 6 months after the course were compared with those during the same period before the course.
Results: The mean age of the 54 participants was 50.5 years (standard deviation, 11.1); 54% were women and 52% were Asian. The mean score for appropriate diagnosis and management increased from 36.1% to 46.7% (odds ratio, 1.6; 95% confidence interval, 1.4–1.9), with greatest improvement in benign lesions, from 32.1% to 46.3% (odds ratio, 1.9; 95% confidence interval, 1.6–2.4). Dermatology referrals for suspicious lesions or new visits by participants' patients decreased at both sites after the course (from 630 to 607 and from 726 to 266, respectively).
Conclusions: This course improved skills in practicing PCPs. Improvement was greatest in the diagnosis and appropriate management of benign lesions and dermatology utilization decreased.
Melanoma incidence has been increasing over several decades, and in men the melanoma mortality rate continues to increase.1⇓–3 Because it is possible to visually examine the skin, screening for melanoma may decrease mortality through earlier detection at a stage when surgical removal would be curative.4,5
It has been suggested that primary care providers (PCPs) could diagnose skin cancer, especially if their skills and confidence were improved.6,7 PCPs have encounters with more patients than any other group of physicians. The dermatology workforce is limited, and studies have found that PCPs can diagnose and appropriately refer melanoma at the same level as dermatologists if they receive targeted education.8,9 Skin cancer training programs, both in person and web-based, have been shown to improve the knowledge and skills of PCPs, which is important because as few as 30% of primary care residents report receiving formal training in performing a skin examination.6,8,10⇓–12
Few educational programs for skin cancer detection have been developed, and even fewer for practicing PCPs in particular.13⇓⇓–16 Web-based curricula could potentially reach a large proportion of the 370,000 practicing PCPs in the United States for whom in-person training is more difficult to provide. A systematic review identified evaluations of 2 web-based skin cancer education programs for PCPs; no study has assessed the effect of these programs on PCP practice.12,17
The aim of this study was to develop an interactive web-based course (http://www.skinsight.com/info/for_professionals/skin-cancer-detection-informed/skin-cancer-education) to improve the skills of practicing PCPs in skin cancer detection, emphasizing melanoma.18 The INFORMED (Internet Curriculum for Melanoma Early Detection) program was developed to provide a web-based early detection training program available for widespread use and includes a large image database. This course was evaluated for its affect on the abilities of PCPs to diagnose and manage skin lesions suspicious for melanoma and nonmelanoma skin cancers. To assess the potential effect on PCP practices, PCP practice patterns were measured before and after receiving the training.
Methods
A before-and-after design was used to evaluate the effectiveness of a self-paced, individually delivered, web-based skin cancer course. The course was developed by a team of dermatologists, primary care clinicians, and medical educators over a 9-month period and was refined during an iterative process using clinicians' feedback. We recruited 54 practicing PCPs from 9 practices in 2 health maintenance organizations to take the course in June 2011. Participants' ability to diagnose and appropriately manage skin cancer was assessed in a pretest, immediate posttest, and 6-month posttest. The effect of the course on practice patterns was assessed by tracking the participants' patient panels for dermatology referrals or visits and skin biopsies during the 6 months following the course compared with the same period 1 year earlier. The study was approved by the institutional review boards of all investigators, and written informed consent was obtained from all participants.
Course Development and Description
The course covered the 3 most common skin cancers: melanoma, basal cell carcinoma (BCC), and squamous cell carcinoma (SCC), as well as benign lesions important in the differential diagnosis of skin cancers, such as seborrheic keratoses.18 The approximately 450 clinical images of lesions used in the course and for testing were vetted by study dermatologists, and malignancies were verified pathologically. Participants chose 1 of 2 different learning formats: a traditional textbook format or a case-based format, which had 9 case vignettes illustrating teaching points with interactive self-assessment quizzes and immediate feedback. Both formats contained the same 9 topic areas: (1) melanoma “ABCD-E”; (2) “ugly duckling” sign for identifying suspicious lesions; (3) benign lesions including seborrheic keratoses; (4) nodular subtype of melanoma; (5) all other melanoma subtypes; (6) melanoma risk factors; (7) BCC; (8) SCC; and (9) office-based policies for integrating skin examination into practice. The course was approved for 2 hours of American Medical Association Physician Recognition Award category I continuing medical education credit.
Course Delivery
The study was conducted with PCPs at 2 integrated health care delivery systems (site A and site B). The 2 delivery systems collectively provide comprehensive inpatient and outpatient care to more than 3.5 million health plan members and employ approximately 3200 PCPs who care for adults at more than 80 local practices. Each delivery system has its own electronic medical record and database to track medical encounters and diagnoses.
Site investigators met with clinical and/or administrative leaders of several local practices to explain the program and arrange for 4 to 7 eligible clinician volunteers from each practice. PCPs were eligible if they spent at least 50% of their clinical time in primary care (internal medicine, geriatrics, or family medicine) and if they had practiced at the site for at least 12 months. To encourage participation, volunteers were offered an honorarium, continuing medical education credit, and a meal.
Small-group educational sessions were held during June 2011 at 4 practices for 25 participants at site A and at 5 practices for 29 participants at site B. Each session was held on site at the participants' practice location after clinical hours with individual laptop computers. The session lasted approximately 3 hours; in addition to the course, the session included a meal, consent, the pretest and immediate posttest, and directed group feedback/discussion on recommended course improvements. At the end of the session, each participant was provided a unique log-in code and encouraged to revisit the course at any time in the future.
Course Evaluation
The course was evaluated according to the approach proposed by Moore,19 with emphasis on the highest levels: competence (ability to diagnose and manage appropriately) and performance (practice patterns). Changes in these attributes, along with changes in attitudes and confidence and participants' assessment of the course (assessed on a 5-point Likert scale, with 5 representing high agreement), were assessed with a pretest, an immediate posttest, and a posttest 6 months after the educational session. The pretest showed images of 25 representative skin lesions; for each lesion, participants chose a diagnosis and management plan (reassure or refer). The immediate posttest had the same 25 images but in a different, randomly determined sequence. Six months later the same test was repeated to assess longer-term effect of the course on competence. For diagnostic questions, there were 6 possible answer choices for each lesion.
The possible answer choices for the identification of each lesion included any 6 of the following: superficial spreading melanoma, nodular melanoma; nodular BCC, superficial BCC, pigmented BCC; SCC; seborrheic keratosis; typical nevus (mole); lentigo, hemangioma; dermatofibroma; blue nevus; actinic keratosis; atypical (dysplastic) nevus; or scar. For management questions, there were 2 possible answer choices for each lesion: “refer or biopsy” or “reassure.”
To determine the effect of the course on clinical performance, the patient panel of each participant was tracked electronically for 6 months after the course (late 2011) and compared with the same 6-month period 1 year before the course (late 2010). At both sites, numbers of patient skin biopsies and skin cancer diagnoses were ascertained. Because of differences in clinical practices and electronic information at the 2 sites, dermatology referrals (and reasons for the referrals) were tracked at one site and new and established dermatology patient visits were tracked at the other.
Analysis
Descriptive analyses were used to describe the participants, using means and standard deviations (SDs) for continuous measures and counts and percentages for categorical measures. Crude proportions of making a correct diagnosis and referral decisions during the pretest and posttests were determined for all participating PCPs by image type and by provider characteristics. To test whether the educational program had a significant statistical effect on PCP competence, generalized linear mixed effects models were used to account for within-clinician and within-image clustering and correlation. Logit link was used since all considered outcomes are dichotomous and logit link guarantees that the probability of having an event is in the meaningful range of [0,1]. The effect measure for a logit link, odds ratio (OR), was calculated with the 95% confidence interval. All analyses were completed using SAS 9.2 (SAS, Inc., Cary, NC).
Results
The mean age of the 54 participants was 50.5 years (SD, 11.1 years) and ranged from 30 to 68 years (Table 1); 54% were female, 52% were Asian, and 39% were white. Among the participants, 80% had trained in internal medicine and 45% completed their medical education in the United States. While 59% reported skin cancer education during residency, only 15% reported it since beginning practice. Participants had sex, race, and training background characteristics similar to their nonparticipant colleagues.
Fourteen participants chose the traditional textbook format and 38 chose the case-based format. Both groups spent about 1 hour on the program (63 and 69 minutes, respectively). They graded the program highly (mean, 4.6 and 4.4 on a 5-point scale, respectively). Only one participant returned to the website in the 6 months after the educational session.
Table 2 shows the effect of the course on participants' abilities in 3 major categories: average overall scores (choosing both the correct diagnosis and correct management plan for lesions); average scores for correct diagnosis of lesions; and average scores for correct management (refer or reassure) of lesions. For all 3 major categories, immediate posttest scores were significantly higher than pretest scores. The overall score on the pretest overall score was 36.1% (9 of 25 lesions), rising approximately 10 percentage points on the immediate posttest to 46.7% (odds ratio, 1.6; 95% confidence interval, 1.4–1.9). The pretest management score (72.9%) was higher than that for diagnosis (41.5%), especially for malignant lesions. In both cases, immediate posttest scores rose significantly. Improvement was greatest in management scores for correctly reassuring, rising almost 20%, from 52.3% to 71.9%. However, management scores for correctly referring suspicious lesions dropped slightly, from 86.6% to 82.8%. Scores for the 6-month posttest declined from those of the immediate posttest but in general remained higher than pretest scores.
Pretest scores predicted posttest scores, but the degree of improvement in posttest scores tended to be inversely correlated with pretest scores (Figure 1). Those scoring lowest on the pretest improved most on the immediate posttest, while the mean score of the highest quartile actually dropped slightly. Scores for the 6-month posttest maintained the improvement for most quartiles but decayed further in the highest quartile.
Overall mean (± standard deviation) test scores for pretest, immediate posttest, and 6-month posttest among participants by initial quartile of performance.
It was investigated whether participant characteristics were associated with improvement in the immediate posttest scores (Table 3). The most striking finding was that the overall scores of participants who reported no previous skin cancer training during their medical education improved 17.4 percentage points (from 33.3% to 50.7%), compared with <12 points for those who had more training. Participants' attitudes and confidence improved modestly after the course (Table 4): <1 point on a 5-point Likert scale.
Among participants' patients at site A there was a slight decrease in referrals for possible skin cancer to dermatology (from 630 to 607) in the 6 months after the course compared with the same period 1 year earlier (Figure 2). At site B, the number of new patient visits seen in dermatology from participants' panels decreased substantially (from 727 to 266), despite an increase in new visits to dermatology system wide (from 8202 to 9624). Skin biopsy rate and skin cancer diagnoses were comparable in 2010 and 2011 at both sites. Therefore, there was no increase in dermatology utilization for suspicious lesions after the skin cancer training.
Dermatology referrals/new visits of participants' patients during the last 6 months of 2010 (before the course) and 2011 (after the course) with skin biopsies and cancers at sites A and B. NMSC, nonmelanoma skin cancer.
Discussion
We developed and evaluated a web-based skin cancer course for practicing PCPs, with interactive feedback and opportunities for self-directed learning. The course improved PCPs' overall ability to diagnose and manage skin lesions almost 30% (about 10 percentage points). Improvement was greater with benign skin lesions, with immediate posttest scores improving approximately 12 percentage points for diagnosis and 20 points for management (Table 2). The course did not increase dermatology utilization and specialty referrals, likely because of the improvement in the diagnosis and management of benign lesions. The course was particularly effective among participants who reported no previous skin cancer education and those whose pretest scores were in the lower quartile. Improvement was still evident 6 months after taking the course.
Taken together, the scores suggest that before taking the course, participants had most difficulty in distinguishing benign from malignant skin lesions and that the course improved this ability. Improvement in confidence scores is consistent with this interpretation. Comparing dermatology referrals or visits by the participants' patients before and after the course also is consistent with improved skills for diagnosing and managing benign lesions. The number of dermatology referrals for skin cancer–related reasons decreased slightly among participants' patients at site A, while there was a substantial decline in new dermatology patients at site B. Numbers of skin biopsies also decreased slightly at both sites. Thus, there is no evidence that the course led to an increase in false-positive referrals of patients because of heightened concern by PCP participants after taking the course; on the contrary, the course was associated with dramatically decreased dermatology visits at one of the sites, consistent with improved management of benign lesions. Referred or new patients who ultimately were diagnosed with skin cancer were stable over time. It is important to note that the improved management of benign lesions by and the diagnostic accuracy of trained PCPs and stable referral to dermatology specialty care suggests the course did not lead to overdiagnosis and overtreatment. With appropriate education in skin cancer and skin examination, improved diagnosis—as opposed to overdiagnosis—may actually translate into a reduction in unnecessary referrals and treatments while in theory expediting skin cancer diagnoses.
The course had a strong effect on participants without previous training (Table 3). The trend among those who scored poorly on the pretest (Figure 1) is consistent, and improvement seems to be durable for at least 6 months. Our course and tests concentrated on skill competency (ability to diagnose and manage skin lesions correctly) rather than on general knowledge about skin cancer (eg, risk factor awareness). PCPs have limited time in which to take lengthy training courses, highlighting the appeal of a shorter 1- to 2-hour web-based design, as used in this course. Hence it is important to emphasize diagnosis and management skills, which require integration of specific, rather than general, knowledge and maximize efficiency.19
In our review of skin cancer education programs, 2 previously published web-based programs for practicing PCPs were identified.12 Harris and colleagues13,14 developed a 6-hour interactive, problem-based program covering melanoma recognition, differentiation of benign lesions from melanoma, risk factors, and prevention. Similar to our results, differences between pretest and immediate posttest scores in both studies demonstrated large improvements in correct management (reassurance) for benign lesions. The authors did not evaluate whether the improvements lasted over time. Despite the differences in participants, tests, and analysis methods, the similar results in our study indicate that a shorter course might be as effective as a course of 6 hours. Gerbert et al20 investigated the effects of an approximately 3-hour Internet-based skin cancer tutorial that enrolled 71 of 2844 members of the Society of General Medicine after randomization; 27 in the intervention group and 19 in the control group completed the program. In the intervention group, the mean overall diagnosis score improved 3% (from 68% to 71%) in the immediate posttest compared with a 5% decline in the control group. The mean overall management score improved 3% (from 75% to 78%) in the intervention group with an 8% decline in the control group. Improvements were not maintained at 8 weeks. Most physicians in the study were academic internists, which may have accounted for high pretest scores and less room for improvement on posttest scores.
There are several notable strengths to this study. To our knowledge, our study is the first to evaluate whether a web-based skin cancer course changed practice patterns.12 As Moore19 points out, practice improvements are ultimately the most important (but least studied) outcome of medical educational interventions. Our results demonstrate an important negative result, namely a decrease in dermatology referrals because of PCPs' suspicion of skin cancer at site A and a dramatic decrease in the number of primary care patients with a new dermatology visit at site B, findings consistent with PCPs' improved ability to diagnose and manage benign lesions. In addition, the number of biopsies and skin cancers were stable at both sites. As expected, the numbers of melanoma cases were too small to determine whether the course had any diagnostic or survival effect.
Because our study was not a randomized trial and depended on volunteers, it is possible that the 54 participants are not representative of their practices. To address that possibility, we collected available information on all other PCPs in the practices (Table 1) and found no substantial demographic differences. Participants who volunteered for this study may have been more interested in learning about skin cancer than their peers, including those with lower confidence who wanted to improve their skills/confidence and others very confident in the area who were striving for refinement of their expertise. In these closed health systems, we are presuming that the general practice preferences and management patterns of PCPs remained stable over time (including referring to dermatology all suspicious lesions and deferring to the dermatologist regarding the need for confirmatory biopsy). Improvements on posttest comparisons may have been due to using the same skin images in both the pre- and posttest. However, images were presented in a different random order, and participants were given no feedback on the test. Hundreds of other images were presented in the intervening course, and it is unlikely they recalled the specific images in the posttest. Furthermore, improvement was sustained in the 6-month posttest and translated into actual clinic practice, supporting true improvement after the course.
The health maintenance organization setting, with an emphasis on using prevention to maintain health, may be an ideal practice setting for this intervention, and our results may not be generalizable to private practice settings. However, because all participants were practicing PCPs our results may be more generalizable than those of Harris et al,14 whose study included 33% non-PCPs, or Gerbert et al,20 most of whose intervention subjects were in academia. Participants in our study donated their time after completing full days in the clinic, and fatigue may have adversely affected the impact of the course. In addition, because of the time needed for meals, administrative details, consent forms, minor computer problems, and debriefing at the end of the session, participants spent less time on the course than we had anticipated, and many said that they felt rushed to complete the course. The results of our study may have underestimated the effect of the intervention because of these factors. Finally, while we were able to measure actual practice patterns, we had different measures of dermatology utilization at the 2 sites.
Conclusion
Improving practicing PCPs' skills at diagnosing and managing skin lesions is an important way to improve patient care because patients frequently come to their physicians with skin complaints or questions. In addition, some groups have advocated skin cancer screening.7 Population-wide screening would have to involve PCPs because of the small number of dermatologists available. Evidence that skin screening may be effective for decreasing melanoma mortality continues to accumulate, but to date no randomized trial has demonstrated decreased mortality.4,5,21 Regardless of whether skin cancer screening is subjected to a rigorous randomized trial, it is important to ensure PCP competence in diagnosing and managing skin lesions. We found that a brief, web-based skin cancer course improved the diagnostic and management skills of practicing PCPs, with improvement still seen at 6 months, without negative effects on dermatology referrals or visits or skin cancer diagnoses.
Acknowledgments
This study was developed by the INFORMED (Internet Course for Melanoma Early Detection) Group, which, in addition to authors listed, includes Jacqueline M. Goulart, MD, Michelle Groesbeck, BS, Richard Krajenta, MS, Shoshana Landow, MD, Kimberly Marcolivio, M Ed, Monica Sokil, RD, and E. Margaret Warton, MPH.
Notes
This article was externally peer reviewed.
Funding: This study was funded by a Team Science Award from the Melanoma Research Alliance, in cooperation with the National Cancer Institute–supported HMO Cancer Research Network (U19 CA 79689).
Conflict of interest: none declared.
- Received for publication March 29, 2013.
- Revision received June 18, 2013.
- Accepted for publication June 20, 2013.
