Abstract
Background: Re-Engineering Systems for Primary Care Treatment of Depression (RESPECT-D) sought to improve patient outcomes by disseminating the 3-component model of depression management. The purpose of this study was to determine whether an integrated model of depression management continued to be used by primary care clinicians after the end of a randomized controlled trial (RCT).
Methods: A descriptive evaluation was conducted at 2 time points. First, during a 12-month period after the end of the RESPECT-D RCT when referrals to care management were determined for each of the 5 participating health care organizations. Second, 3 years after the RCT ended, when clinicians were surveyed about use of the 3-component model.
Results: Three organizations continued to support the model with minimal modification. One made a major modification to it and one did not continue to support it. In the 12 months after the RCT, 1039 care management referrals were made. Seventy-one percent of RCT clinicians (n = 92) completed the follow-up survey. Of these, 87% reported using the Patient Health Questionnaire-9; 58.9% reported availability of care management and 45.1% reported availability of informal psychiatry consultation.
Conclusion: Practical clinical interventions can be sustained in primary care practice after the completion of an RCT. Additional resources may be needed to sustain and spread the program.
Re-Engineering Systems for Primary Care Treatment of Depression (RESPECT-D) sought to improve patient outcomes by disseminating the 3-component model (TCM) of depression management in primary care.1 The TCM is an evidence-based model implemented through existing quality improvement programs based in health plans and medical groups. The project included a multisite randomized controlled trial (RCT).2
The model includes: (1) preparation of primary care practices and clinicians to provide systematic depression care; (2) centrally based care managers to provide telephone support to patients and communicate with their primary care clinicians; and (3) supervision of care managers by psychiatrists who are also available to provide informal advice to primary care physicians. Within the model, the Patient Health Questionnaire-9 (PHQ-9)3,4 is administered by the primary care clinician as part of the initial diagnostic assessment, either when depression is suspected or when depression treatment is being modified. It is then used to monitor treatment response and guide treatment changes.3 The TCM does not promote the use of the PHQ-9 as a broad screening tool, but as a means of systematizing diagnosis and monitoring across the 3 components of the model. Results of the RESPECT-D RCT2 demonstrated that a significantly greater proportion of intervention patients experienced improvement in depressive symptoms and achieved remission at 6 months when compared with those receiving standard care.
These findings are consistent with a growing body of literature that demonstrates that integrating aspects of mental health care within primary care settings improves outcomes for patients with depression.5–13 Because many interventions aimed at improving systems of depression treatment in primary care have relied on grant-funded research, their sustainability beyond the financial support of the research project has been limited.14
A few studies have begun to explore issues of the sustainability of various programs and clinical models within primary care.15,16 Goodson et al15 explored the degree to which a toolkit, used to facilitate provision of preventive services, was institutionalized within 5 primary care practices after the conclusion of their state's Department of Health project funding. They found that factors that contributed to program institutionalization included organizational stability, integration of the toolkit within existing services, toolkit visibility, planning for the end of grant funding, and the presence of a program champion. Blasinsky et al17 used qualitative methods to identify barriers and facilitators to sustaining Project IMPACT, a collaborative care intervention for older adults with major depression or dysthymia. Positive patient outcomes were identified as the most important factor for model sustainability. Organizational support of the model, the presence of trained staff, and funding options were also important facilitators of sustainability. An exploration of sustainability of primary care interventions in Australia identified 3 important themes, including the importance of local and national champions, the impact of political and financial forces, and the skills and motivation of personnel within the organizations.18
To our knowledge, the current study is among the first to use quantitative methods to assess the sustainability of a depression care model within primary care practice in the United States. To advance the current knowledge of this topic, we explored 2 questions after the end of the RESPECT-D RCT1: (1) Did primary care clinicians continue to utilize care management, as evidenced by patient referrals to this service? and (2) Did primary care clinicians report continued use of the key elements of TCM in the management of depressed patients?
Methods
Specifics of the methodology used in the RESPECT-D RCT are described in detail elsewhere19 and summarized here. Five health care organizations (3 medical groups and 2 health plans) served as collaborating institutions. These health care organizations were recruited because of their interest in improving depression care and to reflect the diversity of organizations across the United States. To participate in the RCT, each health care organization needed to be affiliated with a minimum of 10 primary care practices, have an existing quality improvement program, and commit to sustaining and disseminating the clinical model if it proved to be beneficial.
These 5 organizations engaged 60 practices and 188 clinicians in the RCT. Practices were paired based on several factors, including clinician specialty, number of clinicians, the presence of on-site mental health services, and the distance from the health care organization's central office. Within pairs, practices were randomly assigned to the intervention or usual care arm. Nine hundred eighty-seven patients were referred for eligibility evaluations. Of these, 405 were eligible and completed baseline interviews.2
Two additional items are relevant to this evaluation of sustainability. First, to our knowledge, near the completion of the RCT at least 3 of the 5 participating organizations applied for and received additional funding for depression quality improvement work. Second, to fulfill a promise that was part of clinician recruitment, all clinicians assigned to the usual care group were offered the full intervention after RCT data collection was complete, including both training in the model and the ability to refer patients to depression care management. The full intervention was subsequently offered to clinicians within these organizations who did not participate in the RCT. This spread of the model was done by the organizations themselves, with initial support from project funds for ongoing evaluation activities with RCT staff, continuing medical education meetings to introduce the intervention to additional clinicians, visits to practices to assist in implementing the model, and training for additional care managers and supporting psychiatrists.
We report here the evaluation information from 2 distinct time points. First, we report the number of referrals to care management during a 1-year period after the end of the RCT. Second, we describe clinician-reported depression management practices approximately 3 years after the end of the RCT.
One-Year Follow-Up
For 1 year after the end of the RCT, the 5 health care organizations were asked to report the number of patients referred to care management. Three of the organizations tracked referrals to care management via an electronic patient registry. Another of the organizations, which had a single care manager, reported referral numbers based on a spreadsheet maintained by the care manager. The fifth organization relied on an outside care management group and was unable to provide any referral information. The number of referrals was reported for clinicians who had been in the RCT intervention group, clinicians who had been in the RCT usual care group, and clinicians who had not participated in the RCT but were subsequently trained in the TCM.
Three-Year Follow-Up
Between December 2005 and April 2006, all clinicians who participated in the RESPECT-D RCT (n = 188) were invited to complete a 1-page self-report survey regarding their current use of the TCM. Clinicians were asked about their depression management activity in 3 areas: (1) their use of the PHQ-9 for diagnosis and monitoring of depressive symptoms; (2) the availability of a care manager, the most recent point at which this service was used, and the number of referrals made in the previous 3 months; and (3) the availability of informal support from a psychiatrist within their organization, the most recent time at which they conferred with the psychiatrist, and the number of times they used this resource within the last year.
Clinicians who completed the survey received a $20 gift certificate to an online retailer. All analyses were conducted using SPSS for Windows 13.0 (SPSS, Inc., Chicago, IL).
Results
One-Year Follow-Up
Three of the 5 health care organizations (sites A, B, and C; all medical groups) continued to provide the TCM without modification and supplied data regarding the number of patients referred to care management. Clinicians within these 3 health care organizations made 1039 referrals to care management over the first year after the RCT concluded. The proportion of clinicians at sites A through C who made at least one care management referral during the 1-year follow-up period and the mean number of patients referred are shown in Figure 1. It is of note that a higher proportion of usual care clinicians made at least one referral during this period. In addition, on average, usual care clinicians made more referrals (mean, 8.79; SD, 15.0) than TCM clinicians (mean, 2.73; SD, 6.87). This represents a statistically significant difference (t103 = 2.79; P < .001).
Referrals to care management during the 1-year follow-up evaluation period. RCT, randomized controlled trial; TCM, three-component model; UC, usual care.
To provide a context for understanding referrals to care management, the number of referrals made by TCM clinicians during and after the RCT were compared. Of the 66 TCM clinicians at sites A, B, and C, 61 remained during the 1-year follow-up period. During the 12 month RCT, these 61 clinicians made a total of 417 referrals to care management (mean, 6.9; SD, 5.8). During the 1-year follow-up period they referred a total of 168 patients (mean, 2.7; SD, 6.9).
During this time, the 3 sites also reported activities to spread the TCM within their organizations. Site A had trained an additional 106 clinicians in the TCM. Of these, 27.4% made at least one care management referral (mean, 2.9; SD, 8.9). Four additional clinicians had been trained at site B, all of whom made at least one referral to care management (mean, 9.7; SD, 7.1). Fifty-four clinicians had been trained in the clinical model at site C. Of these, 37.0% made at least one referral to care management (mean, 1.1; SD, 2.0). Patterns of referrals over the 1-year evaluation period showed that, in general, during each quarter the 3 health care organizations maintained or increased their total number of referrals to care management.
At site D (a health plan), the process of patient referral to care management was transformed after the end of the RCT. Clinicians continued to have the option to refer individual patients for care management, but the primary mechanism for referral was through a centralized telephone screening procedure for patients identified through administrative data as having a high risk of depression. Site D therefore reported the total number of patients screened for depression (N = 979) and those identified as depressed and eligible for care management services (N = 361). Site D also reported spreading the TCM to additional clinicians but was unable to give precise numbers of clinicians who had been trained.
Care management at site E (a health plan) was provided by an external disease management company; information regarding the number of referrals was not available. Site E did not undertake any activities aimed at spreading the TCM to additional clinicians.
Three-Year Follow-Up
At the time of the 3-year clinician survey, 58 of the original 188 clinicians (30.8%) had either left their practice or the entire practice was no longer associated with the health care organization. The remaining 130 clinicians (69.1%) received the follow-up survey. Of these, 92 (70.8%) completed the survey.
The percentages of clinicians responding from each health care organization ranged from 53% (site D) to 81% (site B). Forty-eight (52.2%) had been assigned to the usual care condition and 44 (47.8%) had been in the intervention arm. Using clinician characteristics measured at the time of the RCT baseline evaluation, clinicians who completed the follow-up survey were compared with those who did not, and no significant differences were found between responders and nonresponders (Table 1).
Characteristics of Clinicians Responding or Not Responding to 3-Year Follow-Up Survey
Table 2 summarizes clinicians’ self-reported depression management activities. The majority of clinicians (87%) responded affirmatively to the question, “Are you using the PHQ-9 as a tool to help assess depression diagnosis and severity?” Though not shown in the table, use of the PHQ-9 by clinicians at sites A, B, and C (medical groups) differed significantly from that reported by clinicians at sites D and E (health plans); χ2 (1, n = 92) = 35.66; P < .0005.
Depression Management Activities Reported by Clinicians in 3-Year Follow-Up Survey
RCT study assignment (either usual care or intervention) was also associated with current PHQ-9 use. Ninety-eight percent of clinicians from the intervention arm indicated PHQ-9 use whereas 77.1% of usual care clinicians reported current PHQ-9 use; χ2 (1, n = 92) = 8.63; P < .005.
The 5 health care organizations continued to make care management available, but only 58.9% of clinicians surveyed indicated that their health care organization currently made care management available for depressed patients (Table 2). Of respondents who indicated that care management was not available to them, 40.5% were TCM clinicians; the great majority of these (78.6%) had made at least one referral to care management during the RCT. Of clinicians who reported making a care management referral, 43.4% indicated that they had referred a patient for care management within the previous 3 months.
Four of the 5 HCOs indicated that a psychiatrist was available for informal consultation regarding the management of depressed patients. However, in response to the question, “Does your health care organization make available to you a psychiatrist whom you can call to obtain advice about depression management?”, only 45% of clinicians indicated that their health care organization made this service available (Table 2). There was no quantitative data available regarding the number of clinicians who used informal psychiatry consultation during the RCT; thus a comparison of knowledge of the service during the RCT versus after the RCT was not possible. However, informal reports suggest it was seldom used. Of clinicians who reported the availability of psychiatry consultation, 41.5% indicated that they had communicated with the psychiatrist within the previous 3 months. Nearly 27% of clinicians reported communicating with the psychiatrist 3 to 5 times in the previous year.
Discussion
Overall, the primary care clinicians that participated in the RESPECT-D RCT continued to incorporate core elements of the TCM in their current depression management. Three organizations continued to support the model with minimal modification whereas one made a major modification to it and one did not continue to support it. Clinicians continued to make referrals to care management for 1 year after the end of the RCT. The use of key elements in the model persisted 3 years after RCT completion, including referral to care management, informal psychiatry consultation, and use of the PHQ-9 in diagnosis and monitoring of depression status.
The spread of the TCM was also evident because 4 of the 5 sites reported training additional clinicians who subsequently made referrals to care management. The fifth organization went through a series of leadership and operational staff changes during and after completion of the RCT. Although the impact of these changes on the potential for dissemination within that organization is unknown, these changes may have had an adverse impact.
Although the numbers of respondents from each organization were too limited to make assertions regarding site differences with confidence, the 3-year survey results suggest that medical group health care organizations (sites A, B, and C) were more successful than health plans (sites D and E) in sustaining use of the PHQ-9. Differences in the continued use of the PHQ-9 were also seen based on RCT group assignment. Significantly fewer usual care clinicians reported current PHQ-9 use as compared with those who had participated in the intervention. It is possible that clinicians who received RESPECT-D training within the structure of the RCT had a greater buy-in to this aspect of the TCM whereas clinicians trained later were more focused on new services seen as directly benefiting their patients, such as referral to care management and accessing informal consultation with the psychiatrist.
Some clinicians working in organizations that sustained the model were unaware that resources such as care management and informal psychiatry consultation were still available. The majority of TCM clinicians who indicated in the 3-year follow-up survey that care management was not currently available to them had made at least one referral to the service during the RCT. This suggests that marketing of the TCM after completion of the RCT may need more attention.
The current analysis does not allow us to give specific explanations for differences in sustainability between medical group and health plan health care organizations. We can, however, hypothesize about aspects of the organizations that might impact the capacity to sustain the clinical model. First, health plans may have been less clinically involved with the associated primary care practices than their medical group counterparts. Second, although a physician's complete patient panel is of interest to a medical group, health plans only cover a portion of a physician's patients. Patients who are seen by clinicians in these health care organizations but who have alternative or no insurance coverage would not be eligible for referral to care management services. Therefore clinicians may be reluctant to offer a valued service to only some of their patients simply based on insurance coverage or ability to pay.
Why was it that during the 1-year follow-up period usual care clinicians were, on average, making more referrals to care management than TCM clinicians? Despite being only speculation, there are 2 distinct possibilities. First, TCM clinicians may simply have referred most of their depressed patients to care management during the RCT. Second, over the RCT period, TCM clinicians may have developed their own approaches to discriminate patients who they think would benefit from care management from those who would not. Thus, during the follow-up period they were making fewer but more appropriate referrals.
We acknowledge several limitations to this analysis. Most importantly, all information regarding on-going implementation of the TCM was gathered via self-report, either from clinicians or organizations with no independent validation available. In addition, because the organizations and clinicians were recruited because of their initial interest in enhancing depression management, it is unknown whether those less motivated would adopt and maintain the model in a similar fashion.
Because of the cross sectional nature of the follow-up clinician survey, we are only able to report a “snapshot” of clinician-reported behavior at one point in time. This methodology does not allow for a broader analysis of the degree to which the model was adopted and sustained. We are also unable to explore practice effects, ie, whether clinicians in the same practice sustained the model in a similar fashion.
We have noted that near the end of the RCT several of the organizations applied for and received funding to support further innovations in depression care. Although these funds were not available until the trial was complete and were not used to support dissemination and sustainability in RCT practices, this support may well have better enabled the organizations receiving them to support the depression management model over time.
We also noted that organizations were provided with support after the RCT to enable their ongoing cooperation with evaluation and to aid in further spread of the TCM. The promise of this financial support helped in recruitment. That promise also helped maintain the clinicians’ ongoing cooperation with the rigorous requirements of the RCT and in their retention. Without that promise, participants would have been less representative, more dropouts would have occurred, and there would have been less complete data in the RCT.
The financial support of the RCT in follow-up impacted sustainability results in several other ways. Without transitional support, we think that more compromises would have been made in maintaining the tested depression management model in the organizations that chose to continue with it. In a recent article20 we report that fidelity matters in terms of outcome. To meet our commitment to clinicians assigned to the usual care arm, we wanted to be sure they received the model of care that showed benefit for the intervention group. In addition, we believe that costs associated with implementing a new model in a practice differ and are higher than the costs of sustaining the model once established. Our focus in this work was on whether the model would be sustained once it was implemented.
The report provides some evidence that key aspects of a practical depression management model can be sustained after the completion of an RCT. However, costs and challenges involved should not be underestimated. Financial and other incentives to provide evidence-based depression care have been weak.14 The Quality and Outcomes Framework21 in the United Kingdom encourages use of depression measures and may help better align incentives with quality depression care. Research on sustainability, including the impact of incentives and of practice and clinician factors, deserves increased attention from investigators, health care providers, and policy makers.
Notes
This article was externally peer reviewed.
Funding: This research was funded by a grant from the John D. and Catherine T. MacArthur Foundation through its Initiative on Depression and Primary Care.
Conflict of interest: Thomas Oxman and Allen Dietrich are partners in a mental health quality company, 3CM LLC, for which SLB and JWW have served as consultants.
- Received for publication February 9, 2007.
- Revision received April 20, 2007.
- Accepted for publication April 26, 2007.
