Abstract
Purpose: Clinical evidence shows minimal benefit to vitamin D screening and subsequent treatment in the general population. This study aims to assess the effectiveness of 2 light-touch interventions on reducing vitamin D test orders.
Methods: The outcomes were weekly average vitamin D rates, computed from adult primary care encounters (preventive or nonpreventive) with a family medicine (FM) or internal medicine (IM) provider from June 14, 2018 through December 12, 2018. We conducted an interrupted time series analysis and estimated the cost impact of the interventions. The interventions consisted of an educational memo (August 9, 2018) distributed to providers and removal of the vitamin D test (FM: August 15, 2018; IM: October 17, 2018) from the providers’ quick order screen in the electronic health record. Change in order rates were analyzed among physicians (MDs and DOs), physician assistants (PAs), and nurse practitioners (NPs).
Results: There were 587,506 primary care encounters (FM = 367,947; IM = 219,559). Vitamin D order rates decreased from 6.9% (FM = 5.1%; IM = 9.9%) to 5.2% (FM = 4% [P < .01], IM = 7.9% [P < .01]). For FM, the vitamin D test order rate continued to fall at a 0.08% per week rate after the interventions (end of study: 2.73%). The education intervention showed a relative decrease in each provider type (FM-physician = 16% [P < .01], FM-PA = 47% [P < .01], FM-NP = 20% [P = .01], IM-physician = 14% [P = .02], IM-PA = 52% [P < .01], IM-NP = 34% [P = .04]). Annualized savings was approximately 1 million dollars.
Conclusions: Emailed evidence-based provider education may be an effective tool for modifying providers’ vitamin D test ordering behavior. The lack of the effectiveness of the vitamin D test removal from the quick order screen found for IM highlights the challenges facing simple electronic health record interventions when multiple alternate ordering pathways exist.
- Electronic Mail
- Health Policy
- Interrupted Time Series Analysis
- Population Health
- Primary Health Care
- Vitamin D
Introduction
The Washington Health Alliance based on the Choosing Wisely campaign highlighted that more than 35% of vitamin D screening is wasteful.1 While serum vitamin D concentrations are increasingly measured in clinical practice to evaluate vitamin D deficiency2 the utility of vitamin D screening had not been established in the general population. In prior studies, adults treated for vitamin D deficiency were no less likely to die or to develop cancer, diabetes, or bone fractures.3 Further studies have demonstrated no association between vitamin D supplementation and reductions in fatigue, depression, chronic pain, mortality, or osteoarthritis.4⇓⇓⇓⇓–9 The United States Preventive Services Task Force concluded that there is insufficient evidence to make a recommendation on population-based screening in asymptomatic adults.10⇓–12 The Endocrine Society and American Society for Clinical Pathology, in association with the Choosing Wisely Campaign, have issued recommendations against routine screening for vitamin D deficiency.4 Recently, dominant insurance payors started to deny claims for most vitamin D screening orders.13
Despite the lack of evidence, presence of society guidelines, and payor pressures to decrease vitamin D screening, it remains a prevalent practice. When gaps between evidence and practice exist, health systems lack strategies proven to change provider behavior to reduce unneeded care.14 In previous studies, various combinations of behavioral modification tools including provider education, electronic health record (EHR) based interventions (eg, clinical decision support, alerting), choice architecture modifications (eg, removal from ordering lists), and peer comparison reports have each been applied to help modify providers’ ordering behaviors.15⇓⇓⇓⇓–20 In prepost cross-sectional descriptive analyses, Felcher et al16 found that vitamin D screening rates decreased nearly 70% after implementation of a bundle of 3 clinical decision support tools: educating providers about new screening guidelines, a hard-stop alert (ie, order prevented, justification required to override), and laboratory order list modifications. The authors, however, could not isolate the effect of the education intervention from EHR components. Petrilli et al15 compared vitamin D test order rates before, during, and after the deployment of an EHR alert showing order rates decreased both during the development period, when education about the alert and clinical guidelines was disseminated, and after the alert implementation. The authors also estimated the separate effect of education during the development of the EHR alert, but the concept of education in their study was wide ranging and encompassed informational meetings and memos over 12 months.15 None of these prior studies explored differences in response to interventions between family medicine (FM) and internal medicine (IM) providers, or between physicians—medical doctors (MDs) and Doctor of Osteopathic Medicine (DOs) and other advanced practice providers, such as physician assistants (PAs) and nurse practitioners (NPs).
In this study, we analyzed effectiveness of a 1-time educational e-mail sent to FM and IM primary care providers, with and without concurrent modifications to electronic ordering options for vitamin D testing. We hypothesized that an education memo and EHR intervention, both individually and jointly, would modify provider behavior, leading to decreased vitamin D test order rates, with a larger effect when the interventions were combined.
Methods
Study Design
We used an interrupted time series (ITS) design to evaluate the association between intervention rollout and vitamin D test order rates across primary care practices within a large clinically integrated network in the Southeastern United States. For analyses, MDs and DOs were grouped as physicians (for simplicity referred to throughout as MDs). PAs and NPs were grouped separately because workflows and patient responsibilities are distinct from MDs. The study was approved by the Institutional Review Board at Atrium Health (#10-18-08E) with a waiver of consent and waiver of authorization.
Data and Sample Selection
We included office visits for adults (≥18 years) with a FM or IM primary care provider between June 14, 2018 and December 12, 2018. We used Current Procedural Terminology codes21 for Evaluation and Management Services from billing data (Epic) to select office visits that were classified either as nonpreventative or preventive. The vitamin D test orders, patient, and provider characteristics were sourced from the health system’s EHR data (Cerner).
Outcome Measures
The primary outcome was the weekly vitamin D test order rate. Secondary outcomes included weekly vitamin D test order rate in each specialty (FM or IM) separately. We created 3 measures: 1) weekly rate for FM and IM providers, 2) weekly rate for FM providers only, and 3) weekly rate for IM providers only. A vitamin D test was defined as an order for serum 25-hydroxyvitamin D or 1,25-dihydroxyvitamin D. Vitamin D test orders were attributed to the visit if the lab was ordered within 14 days before or 7 days after the visit, allowing for providers to order labs before or after visits. The vitamin D test order rate was computed as the number of eligible primary care office visits during the week (ie, 7-day period) with an associated vitamin D test order.
Interventions
Two distinct interventions were implemented. Intervention 1 (Education Memo) was an emailed memorandum outlining the clinical evidence and guideline recommendations against vitamin D screening in the general population. The memo informed providers that some payors were no longer covering the screening and of the planned dates when the vitamin D test would be removed from the EHR’s “quick order” screen (described in next paragraph-Intervention 2). The Education Memo was created by physician leaders with support by local medical group leadership and emailed to all primary care physicians on August 9, 2018.
Intervention 2 (Quick Order Screen Removal) was the removal of the vitamin D test from the laboratory orders list on the EHR’s “quick order” screen. The “quick order” screen lists frequently ordered labs, allowing the provider to order with a single click rather than using a search function. This intervention was implemented using a staggered approach, on August 15, 2018 for FM and on October 17, 2018 for IM. Timing of the EHR removal for FM was intended to be simultaneous with the Education Memo, but due to logistics this occurred within the same week. The delayed timing of the EHR removal for IM was by design to allow the evaluation to isolate the educational memo effect from the EHR effect.
Power Analysis
Power was based on the planned ITS design where event rates were analyzed across consecutive time points.22⇓⇓–25 Based on the health system’s historic data, vitamin D screening order rates ranged from 5% to 6% in 2017 for FM among approximately 65,000 encounters per month. Similarly, the vitamin D screening order rate for IM ranged from 8% to 10% among 48,000 encounters per month. A hypothesized decrease in post intervention vitamin D test rate of 1.5% point (5% significance level) yielded a power of over 80% for both primary care groups (FM: from 5% to 3.5%; power = 99.9%; IM: from 8% to 6.5%; power = 82.7%).
Statistical Analysis
Patient-level descriptive statistics were reported as means with standard deviation for continuous variables and frequencies with percentages for categorical variables. The effects of the Education Memo with and without concurrent Quick Order Screen Removal on vitamin D test order rates were estimated using a segmented regression analysis with ordinary least squares estimation, controlling for temporal trends. The unit of analysis was 1 week (time interval), and 26 weekly intervals were included in the sample: 8 weeks before the interventions (week 1 to 8), 10 weeks during the staggered intervention implementation (week 9 to 18), and 8 weeks after the last intervention, Quick Order Screen removal for IM (week 19 to 26).
We constructed 3 model specifications. In Model 1, we assessed change in the weekly vitamin D test order rate, after all primary care providers had been exposed to both interventions (weeks 19 to 26) compared with the preintervention rate (weeks 1 to 8). We excluded weeks 9 through 18 (washout) during which the interventions were rolled out, which allowed us to assess the total effect of the interventions. From Model 1, we estimated the time trend in the weekly vitamin D test order rate over weeks 1 to 8 and 19 to 26, the effect of the intervention dummy variable, and the change in slope in the vitamin D test order rate after the interventions.
In Models 2 and 3, we assessed the effect of the interventions separately for FM and IM subgroups. In Model 2 (FM only), we used the same model equation specifications as described for Model 1 to assess the joint effects of the interventions. The preintervention period was comprised of weeks 1 to 8 and the postintervention period of weeks 10 to 26. We excluded week 9 (washout) from the analysis due to the proximity of the rollout of the Education Memo and the Quick Order Screen Removal interventions for FM, which also prevented us from isolating the separate effects of these 2 interventions.
In Model 3 (IM only), we leveraged the staggered, sequential roll out of the 2 interventions to IM providers (ie, Education Memo in week 9 and the Quick Order Screen Removal in week 18), to isolate the individual intervention effects in the IM subgroup. We used the entire sample of 26 weeks to estimate the time trend in the weekly order rate over the study period. We evaluated separate effects of each intervention from the corresponding changes in the postintervention slope following each intervention.
To compare temporal ordering patterns without the influence of the study interventions, we conducted a sensitivity analysis by estimating Model 1 with data from the same weeks in the previous year (ie, June 14 to December 12, 2017). In a post-hoc analysis, we explored differential intervention effects separately for FM (Model 2) and IM (Model 3) groups, stratified by provider educational degree (MDs vs PAs vs NPs), patient age (<65 vs ≥65 years), the reason for the visit (Nonpreventive vs Preventive), patient race (white vs African American) and patient gender (male vs female). The emailed educational memo only went to MDs (disseminated to advanced practice providers by physician leaders). We computed the annualized cost impact of the interventions from the payor’s perspective by multiplying the number of visits in 2018 by the estimated difference in the vitamin D test order rates with and without the interventions, and by the Medicare payment rate for vitamin D tests. All statistical analyses were performed using SAS Enterprise Guide v7.15 (Cary, NC). Statistical tests were 2-tailed, and P-values <0.05 were considered statistically significant.
Results
Population Characteristics
We calculated weekly vitamin D test order rates from 587,506 primary care encounters including 367,778 visits with FM providers (62.6%), and 219,728 visits with IM providers (37.4%). There were 72,263 preventive visits (12.3%) and 515,243 nonpreventive visits. MDs treated the majority of patients (66%). On average, patients were 55 years old, 60% were female, 69% white, and 20% were African American. Characteristics of the study population were consistent across the preintervention, intervention, and postintervention periods (Table 1).
Population Characteristics
Interrupted Time Series Regression Results
In Model 1, we found that the 2 interventions were collectively associated with a 1.74%-point decrease (25% relative decrease) in the weekly vitamin D test order rate (6.9% to 5.2%; absolute slope change, −1.7%; 95% CI, −2.3 to −1.2; P < .01; Table 2 and Figure 1) in primary care. The postintervention change in slope was not statistically significant (absolute slope change, −0.06; 95% CI, −0.18 to 0.06; P = .31).
Results: Effect of Interventions to Weekly Vitamin D Test Order Rates
Weekly vitamin D test order rates for primary care, comprised of family and internal medicine (Model 1). Abbreviations: EHR, Electronic health record.
In Model 2, we found a 1.12%-point decrease in the weekly vitamin D order rate (5.13% to 4.01%; absolute slope change, 1.12%; 95% CI, −1.44 to −0.81; P < .001) for only FM providers (Table 2; Figure 2). In addition, for this subgroup there was an additional 0.08%-point decrease in the postintervention rate per week (absolute slope change, −0.08; 95% CI, −0.14 to −0.02; P = .01), showing the cumulative effect of the interventions increased over time. Using the decreased slope for the FM group, we estimated the relative difference of the vitamin D test order rate fell 21.8% immediately after the interventions (week 10), and 46.75% by the end of the study (week 26), from 5.14% to 4.01% to 2.73%, respectively.
Weekly vitamin D test order rate for family medicine (Model 2). Abbreviation: EHR, Electronic health record.
In Model 3, we analyzed the effect of the 2 interventions on the vitamin D test order rate for IM providers only (Table 2; Figure 3). We found that the Education Memo was associated with a 1.9%-point decrease (9.88% to 7.98%; absolute slope change, −1.9%; 95% CI, −3.0 to −0.8; P = .002) in the vitamin D test order rate, with a relative decrease of 19.23%. The “Quick Order” screen removal did not have a statistically significant effect on the weekly order rates (absolute slope change, 0.07; 95% CI, −1.05 to 1.21; P = .89). In the IM subgroup, there was no statistically significant change in the postintervention slope for both the interventions (Educational Memo: absolute slope change, −0.04; 95% CI, −0.26 to 0.18; P = .72; Quick Order: absolute slope change, −0.07; 95% CI, −0.29 to 0.15; P = .53).
Weekly vitamin D test order rate for internal medicine (Model 3). Abbreviation: EHR, Electronic health record.
Sensitivity Analysis and Stratified Analysis
In assessing data from the same months in the previous year, we observed a 7.8% vitamin D test order rate during the equivalent preintervention time period across primary care and a statistically insignificant increase of 0.5% (P = .16) in the order rate after the equivalent postintervention time frame. In the subsample analyses, we addressed the effect of the interventions on vitamin D test order rates for specific subgroups using specifications from Model 2 for FM and Model 3 for IM (Table 3). We found that preintervention vitamin D test order rates and the magnitude of the interventions’ effects varied across the subgroups. For both FM and IM, vitamin D test order rates were higher for preventive than established visits, but they decreased more for preventive visits than for nonpreventive visits following intervention rollout; for established visits in FM, the order rates continued to decrease every week after the interventions. For both FM and IM specialties, vitamin D test order rates decreased more among PAs (FM, 47%, IM, 52%), NPs (FM, 20%, IM, 34%), than MDs (FM, 16%, IM, 14%) following the interventions. Vitamin D test order rates were lower and decreased more for male patients than female patients, and the intervention response by patient age and race differed between IM and FM.
Sub-Group Analysis by Specialty
Cost Impact from the Payor’s Perspective
In 2018, the total number of nonpreventative and preventive primary care office visits was 1,353,591, out of which 778,410 were in FM and 575,181 in IM (Table 4). In Models 2 and 3 we estimated that the preintervention order rates were 5.13% for FM and 9.88% for IM, and that by the end of the study period (week 26) the postintervention rates were 2.73% for FM and 7.98% for IM. Based on these estimates, the annualized effect of the interventions in 2018 would have been 29,610 prevented vitamin D test orders (18,682 [FM] and 10,928 [IM]). In 2018, the Medicare payment per vitamin D order to the hospital system was approximately $35.82. Using the Medicare payment rate, from the payor perspective the annualized saving associated with the interventions was $1,060,640.
Cost Impact Analysis (Payor Perspective, Medicare Rate)
Discussion
This study found significant associations between 2 interventions that aimed to modify providers’ test ordering behavior and reductions in vitamin D test order rates. Specifically, when combined the interventions were associated with a decrease in the vitamin D test order rate for primary care (Model 1). This same association was demonstrated in our sub analyses of the combined interventions in FM. Unique to this study, we also had the opportunity to unbundle the interventions in the subgroup analyses for IM. Here we again saw a decrease after the education memo, but not after the quick order removal. The ITS design allowed us to examine time trends in the vitamin D order rate not only before and during the intervention, while also enabling the evaluation of rates over time during a sustainability period.15
Our findings raise the possibility that a carefully designed education memo alone is a powerful tool for changing provider behavior. To our knowledge, this is the first study to isolate the effect of a 1-time emailed educational memo on provider lab ordering behavior. Other studies have shown education may contribute to lowering vitamin D screening order rates, but either were not able to isolate this effect from other intervention components, or the intervention education was more intensive consisting of multiple efforts over time.15,16,26 In our case, the 1-time distribution may have contributed to the memo’s effectiveness because providers were not inundated with alerts or messaging, avoiding the alert fatigue phenomenon.27 Both the content and author/sender of the memo in this study were likely also key to the intervention’s success.28 In this case, the emailed memo was signed/sent by the CEO of the physicians’ clinically integrated network and endorsed by the leadership of the health system’s primary care group. Indeed, behavioral literature suggests that authority of the writer gives the message itself authority and credibility.29 Further, the education content included evidence-based recommendations and information on some insurers decision to deny claims for vitamin D screening for the general population, with the latter implying a financial impact on patients.
We did not find evidence of an incremental benefit to the removal of the vitamin D test from the EHR quick order screen for IM. While our finding contradicts other studies that found that EHR interventions modified provider behavior, this may be explained by some differences specific to our intervention.15⇓–17 It is likely that the EHR intervention here was not robust enough. Instead of ordering vitamin D test using the quick order screen, providers still had the option to order the test using a customized “favorites” screen, “powerplan” screens designed for specific conditions, or using the search function. None of these alternatives were changed as part of the intervention and would not require much additional effort on a provider’s part to use. Physicians may either have been using them before the intervention and simply continued or changed their behavior to using these alternate ordering pathways after the intervention. Lastly, our EHR intervention did not embed point of care alert education, which in other EHR interventions may serve as an ongoing reminder to providers of the why behind not ordering a vitamin D test.
Direct comparison of the magnitude of the vitamin D test rates and intervention effects across published studies is difficult because of heterogeneity in inclusion criteria, vitamin D order definitions, and methods employed to reduce orders. In general, our FM vitamin D test order rates (5.13% preintervention, 4.01% at week 10, and 2.73% at the end of the study, week 26) are comparable to Petrilli15 (8% and 3%) and Felcher16 (7.4% and 2.4%) while our IM pre- and postintervention rates are higher (9.88% and 7.98% posteducation memo).
We estimated that the annualized number of vitamin D test orders prevented due to the interventions would have been 29,610 in 2018 with a corresponding annualized saving of $1,060,640 from the payor’s perspective. This saving highlights the potential for a larger impact that health systems could have on lowering health care costs through strategically employing low-cost educational nudges across a variety of potentially wasteful procedures, including other preventive lab screenings.
The study has several limitations and results should be interpreted within the context of the study design. While we were not able to randomly allocate the interventions, our ITS study design accommodated a rigorous evaluation within the timeline of a quality improvement intervention to reduce vitamin D test order rates. Similar to prior studies without a contemporary control group,16 we conducted a sensitivity analysis to adjust for potential effects of seasonality, with findings supporting our interpretation that the observed effects were related to the interventions rather than annual or seasonal factors. To our knowledge, no other concurrent system changes occurred during the study period that potentially influenced the vitamin D test ordering behavior, though the potential for unmeasured confounders such as hearing about payor policy changes through other channels still exist.
Another limitation is that the interventions for FM took place nearly simultaneously, which did not allow for separating the analysis to look at the contribution of individual components within the FM group. The educational memo was only emailed to MDs; however, in our cohort, one third were ordered by non-MDs, which may have limited the effectiveness of the education intervention. Despite this, we saw decreases across ordering rates for PAs/NPs, suggesting that the education may have been disseminated more broadly by MDs. Lastly, these results reflect the practice patterns of providers within a single large clinically integrated network in the southeastern US. While the study population is geographically diverse (urban and rural), the generalizability may be limited as other factors that influence provider and patient behavior may vary in different settings.
Conclusion
A low-resource intervention that included an educational e-mail with or without concurrent modifications to EHR ordering was associated with significant reductions in vitamin D test ordering in primary care. This finding affirms for health care systems seeking to drive value and evidence-based care delivery that brief provider education through a memo can be a powerful, cost-effective component of an overall strategy. While EHR interventions are likely also a part of any such strategy, this study highlights the need for a more comprehensive EHR approach—simply changing 1 order pathway when many exist may not impart the desired changed. Future research should further explore differences in the potential drivers of ordering behavior between IM and FM specialties as well as behavior change across physicians and advanced practice providers.
Notes
This article was externally peer reviewed.
To see this article online, please go to: http://jabfm.org/content/33/2/252.full.
Conflicts of interest: AM reported being cofounder of iEnroll, LLC. No other disclosures were reported.
Funding: This project was internally funded by Atrium Health.
- Received for publication September 13, 2019.
- Revision received December 5, 2019.
- Accepted for publication December 6, 2019.
