Treatment Differences in Primary and Specialty Settings in Veterans with Major Depression
=========================================================================================

* Victor Puac-Polanco
* Lucinda B. Leung
* Robert M. Bossarte
* Corey Bryant
* Janelle N. Keusch
* Howard Liu
* Hannah N. Ziobrowski
* Wilfred R. Pigeon
* David W. Oslin
* Edward P. Post
* Ronald C. Kessler

## Abstract

*Introduction:* The Veterans Health Administration (VHA) supports the nation's largest primary care–mental health integration (PC-MHI) collaborative care model to increase treatment of mild to moderate common mental disorders in primary care (PC) and refer more severe-complex cases to specialty mental health (SMH) settings. It is unclear how this treatment assignment works in practice.

*Methods:* Patients (n = 2610) who sought incident episode VHA treatment for depression completed a baseline self-report questionnaire about depression severity-complexity. Administrative data were used to determine settings and types of treatment during the next 30 days.

*Results:* Thirty-four percent (34.2%) of depressed patients received treatment in PC settings, 65.8% in SMH settings. PC patients had less severe and fewer comorbid depressive episodes. Patients with lowest severity and/or complexity were most likely to receive PC antidepressant medication treatment; those with highest severity and/or complexity were most likely to receive combined treatment in SMH settings. Assignment of patients across settings and types of treatment was stronger than found in previous civilian studies but less pronounced than expected (cross-validated AUC = 0.50-0.68).

*Discussion:* By expanding access to evidence-based treatments, VHA's PC-MHI increases consistency of treatment assignment. Reasons for assignment being less pronounced than expected and implications for treatment response will require continued study.

*   Comorbidity
*   Depression
*   Integrated Health Care Systems
*   Mental Health Services
*   Military Medicine
*   Primary Health Care
*   Psychotherapy
*   Self-Report
*   Veterans Health

## Introduction

Depressive disorders are more prevalent among US veterans1⇓–3 than civilians.4⇓–6 The Veterans Health Administration (VHA) has initiated a system of primary care–mental health integration (PC-MHI) to address this high prevalence and that of other common mental disorders by including psychologists, psychiatrists, nurses, and social workers on primary care (PC) teams to collaborate in evaluation and treatment.7 PC-MHI is the country's largest implementation of a collaborative care model for treatment of common mental disorders and consequently represents a unique opportunity to study implications of team-based treatment. The model has proven effective and efficient in treating mild and moderate depression7⇓⇓⇓–11 while referring more severe and refractory cases to specialty care11 based on VHA clinical practice guidelines.12 However, setting and type of treatment may differ from guidelines because of differences in patient preferences and experiences, differences in comfort levels of PC clinicians in treating depression, and geographic differences in access to services. Whether these factors influence treatment decisions regarding setting and type of treatment, in turn, might have implications for treatment quality and outcomes.13⇓⇓⇓⇓⇓⇓–20

Previous research in civilian samples comparing patients in PC versus specialty mental health (SMH) settings has found mixed evidence for differences in depression severity and complexity.21⇓–23 We would expect assignment to be more distinct in VHA given the existence of PC-MHI and VHA treatment guidelines calling for less complex cases to be treated in integrated PC and more complex cases to be referred to SMH. However, it is unknown whether this is the case. In addition, unknown is what other factors may affect assignment, including patient factors (eg, preferences, comorbidities, sociodemographics, treatment adherence), provider factors (eg, preferences, willingness to treat, time constraints), and system factors (eg, referral resources, incentives). Evidence suggests that prescriber specialty and place of treatment are important factors in determining outcomes.24,25 As a result, understanding the drivers of patient assignment to a given setting and treatment can help improve care quality, predict successful treatment, and potentially lower health care costs. The current report's goal is to present national data on these issues as part of an observational study of baseline predictors of differential treatment assignment across VHA settings and treatment types among patients with new diagnoses of depression.

## Methods

### Sample

Patients were recruited in weekly samples between December 2018 and June 2020. Eligible patients were defined as those identified from VHA electronic medical records (EMRs) as making an outpatient visit at either a PC or SMH clinic for treatment of major depression in the prior week and either receiving a prescription for antidepressant medication (ADM) or referral to psychotherapy. Patients were recruited regardless of whether depression was the primary complaint. As we were interested in analyzing patients with a new diagnosis, past 365 days' exclusions included any VHA visit with a diagnosis of major depression or any ADM prescription. We also excluded patients with a suicide plan in the past 2 weeks or lifetime severe mental disorders (ie, any VHA visit with a diagnosis of bipolar disorder, psychosis, dementia, intellectual disabilities, autism, Tourette's disorder, stereotyped movement disorders, borderline intellectual functioning, or prescription of either antimanic or antipsychotic medication; see Appendix 1 for ICD-9/10-CM codes). In addition, excluded after completing the baseline survey were patients who did not report in the survey that depression was a primary or secondary visit reason.

Recruitment began with a weekly mailing of a letter to a probability sample of eligible patients from VHA records in the conterminous United States who had an initial outpatient visit in the past week, inviting them to participate in a study of depression treatment that would require completing a self-report web or phone-based baseline questionnaire averaging 45 minutes with a $50 incentive and a 3-month self-report follow-up averaging 20 minutes with a $25 incentive. Given the substantial proportion of VHA depressed patients treated with ADM only, we undersampled patients having a record indicating a PC-MHI contact with ADM but not psychotherapy. This allowed a larger proportion of patients treated with psychotherapy to be included in the sample for purposes of comparing psychotherapy between primary and specialty settings. The recruit letter included an 800 number for questions or to opt out. We then made up to 3 recruitment calls at different times over the next week. Cases not reached within the 3 calls were closed out. We focus in the current report on baseline results of the 2610 respondents who passed all study inclusion and exclusion criteria. The Institutional Review Board of Syracuse VA Medical Center, Syracuse, New York, approved these procedures.

### Measures

#### Administrative Variables Comparing the Analysis Sample with the Population

Information was abstracted from the VHA EMR for patients to whom we mailed invitations (n = 55,106) about sociodemographics (age, sex, race/ethnicity, marital status) and location of home address, whether the incident visit was at a community-based or hospital-based clinic, if depression was the primary or secondary diagnosis and, if secondary, whether the primary diagnosis was another mental disorder or a physical disorder; and if the patient was seen on the day of initial treatment by a primary care clinician (PCP), was prescribed ADM, was referred to psychotherapy, or received a code indicating a PC-MHI contact. Prior mental health history was also abstracted from EMR.

#### Treatment Setting and Type

Administrative data from the initial visit and following 30 days were used to distinguish patients whose treatment occurred exclusively in PC versus SMH. Patients who began treatment in PC and then moved to SMH were coded as SMH. Treatment type was coded as psychotherapy (patients who were referred to psychotherapy), ADM (patients who received an ADM prescription), or combined (referral to psychotherapy and an ADM prescription). Patients who only had initial visit data were included in the analysis.

#### Depression and Psychiatric Comorbidity

Depression symptom severity in the 2 weeks before seeking treatment was assessed in the baseline survey with the 16-item Quick Inventory of Depressive Symptomatology Self-Report Scale (QIDS-SR).26 Total scores were transformed into Hamilton Rating Scale for Depression (HRSD) severity levels of none, mild, moderate, severe, and very severe using published transformation guidelines.27 Additional questions from diverse instruments were used to enrich the assessment of depressive features to search for dimensions that might distinguish patients across settings and predict treatment response, all using the same 2-week recall period. Depression persistence was defined using questions from the Composite International Diagnostic Interview28 to obtain retrospective assessments of depression age of onset, number of years with depression, and length of current depressive episode. Patients were also asked about other presenting mental health problems, asked which were primary versus secondary, and were administered brief dimensional screening scales for comorbid disorders of special interest: post-traumatic stress disorder (PTSD), alcohol/substance disorder, and somatic symptoms disorder. (See Appendix 2 for an overview.)

### Analysis Procedures

A comparison of administrative variables between baseline survey respondents and nonrespondents in the sample of 55,106 was conducted using logistic regression. The R program sbw29 was then used to implement a stable weight balancing procedure30 to adjust for significant differences between respondents and the full sample. The depression symptom measures were then subjected to exploratory factor analysis in the weighted respondent sample. Factor-based scales were constructed with equal weighting across items with standardized partial regression coefficients of at least 0.40 after assigning means to item-missing score values. The resulting scales were then standardized in the weighted sample to a mean of 0 and variance of 1.0 to facilitate interpretation.

One-way analysis of variance was used to compare patients across settings and types of treatment on standardized (mean of 0, variance of 1.0) administrative variables, depression symptom scales, and comorbidity measures. Similar to prior studies of depression-related outcomes between PC and SMH settings,22,24 the analyses adjusted for age, sex, race and ethnicity, marital status, census region, urbanicity, percentage of population below 1.5 of poverty line, history of previously diagnosed mental disorders, number of previously diagnosed mental disorders, current depression treatment, and treatment location, setting, and type. Estimates were adjusted for the false discovery rate using the Benjamini–Yekutieli method.31 Ensemble machine learning32 was then used to assess distinctiveness of predictor profiles of patients in each setting-type of treatment. This method used a series of different classifiers (Appendix 3) to capture nonlinearities and interactions among predictors to obtain the best 10-fold externally cross-validated prediction of treatment setting-type. Strength of associations was quantified with AUC predicting individual setting-type combinations in the total sample.

## Results

### Comparison of Analysis Sample with the Full Original Sample

Of the 55,106 patients we attempted to contact, 17,000 were reached by telephone. The others either were not reached after 3 calls (n = 27,603), their phone numbers no longer worked (n = 6,828), or they moved without forwarding information (n = 3,675) (Appendix 4). Of the 17,000, 6,298 patients agreed to participate, and 4,164 completed the baseline questionnaire (24.4% cooperation rate). We subsequently excluded 1,554 respondents because they had a history of bipolar disorder not found in VHA records (n = 728), reported current suicidality (n = 84), said depression was not a primary or secondary presenting problem (n = 471), or reported no depression severity in the 2 weeks before baseline assessment (n = 271). Analysis focuses on the remaining 2,610 patients, most of whom were young (54.5% aged 49 years or less), male (82.7%), non-Hispanic White (60.8%), married (48.6%), living in the south (50.6%), and living in major metro areas (85.9%). About half reported a prior history of depression (48.0%). Most had 1 or more mental comorbidities (69.7%). Most reported that depression was their main reason to seek care (58.2%) (Table 1). Most patients were referred to psychotherapy (89.3%), while less than one third were prescribed an ADM (31.8%).

View this table:
[Table 1.](http://www.jabfm.org/content/34/2/268/T1)

Table 1. 
Distributions and Associations of Administrative Variables with Survey Completion (n = 55,106)†

Patients who completed the questionnaire were, on average, somewhat older than nonrespondents and more likely to be female, non-Hispanic White, and currently married, with reduced odds among the under-represented categories in the range OR = 0.58-0.83. Although these characteristics were related significantly to participation (χ235 = 401.2, *P* < .*001*), the multivariate association of predictors with participation was weak (AUC = 0.59). We nonetheless weighted the sample of survey respondents to adjust for these small differences.30

### Exploratory Factor Analysis of Depression Symptom Severity Measures

Sixteen percent (16.4%) of patients who completed the questionnaire and were eligible had one or more missing items (10.9% missing only 1 item, 2.2% 2, 1.5% 3, and 1.7% 4+, 0.6% overall item missing response rate). Exploratory factor analysis among respondents with complete data found 7 factors that, after promax rotation, were labeled depression symptom severity (14 items; Cronbach's α = 0.92), positive mental health (19 items; α = 0.81), anhedonia (5 items; α = 0.86), cognitive difficulties (7 items; α = 0.20), rumination (5 items; α = 0.72), dissociation (4 items; α = 0.89), and mixed features (6 items; α = 0.78) (Appendix 5). Correlations among factors were between 0.53 (depression symptom severity and low positive mental health) and 0.09 (cognitive difficulties and mixed features) (Appendix 6).

### Distribution and Administrative Correlates of Treatment Setting and Type

Thirty-four percent (34.2%) of depressed patients were treated in integrated PC and 65.8% in SMH during the initial visit and following 30 days. Patients with PC-MHI encounters receiving only ADM made up 32.4% of the weighted PC sample compared with 18.2% of the SMH sample. Patients with psychotherapy made up 46.9% of the PC sample and 51.7% of SMH. Patients with combined treatment made up the remaining 20.8% of PC and 30.2% of SMH samples.

Patients in PC differed only modestly from those in SMH in terms of sociodemographics and geographic variables. More consistent, albeit relatively modest, differences were found in history of prior mental disorders, which were all less common among PC than SMH patients, with PC standardized mean estimates (Est) ranging between −0.05 and −0.18 (Table 2). PC patients were somewhat less likely than SMH patients to have presented with depression secondary to another mental disorder (Est = -0.15) and less likely to receive a psychotherapy referral on the first visit (Est = -0.35). PC patients were more likely than SMH patients, in comparison, to have presented with depression secondary to a physical disorder and to receive an ADM prescription (Est = 0.14-0.16). PC patients were more likely than SMH patients to have received a PC-MHI encounter during their first visit (Est = 0.70).

View this table:
[Table 2.](http://www.jabfm.org/content/34/2/268/T2)

Table 2. 
Associations of Administrative Variables with Setting and Type of Treatment (n = 2610)†

Administrative variables were also associated with treatment type within and between settings. Sociodemographics were generally weak predictors, although the oldest patients (ages 60+) were less likely than others to receive combined treatment in both settings (Est = −0.13 to −0.18). Six out of 11 measures of prior mental disorders were predictors of treatment setting-type (F5 = 3.4-9.9, *P* = .005-<0.001), with increases in SMH and especially SMH combined treatment (Est = 0.13-0.18) strongest for prior PTSD, substance disorder, and 3+ prior diagnoses compared with other treatment types. Presenting problems were also predictors, with primary depression more likely to be treated with PC psychotherapy (Est = 0.27), depression secondary to a physical disorder with PC ADM (Est = 0.78), and depression secondary to another mental disorder with SMH ADM or combined treatment (Est = 0.16-0.18).

Patients seen initially by a PCP were more likely than others to end up in PC ADM (Est = 0.78) or combined (Est = 0.35) treatment, whereas patients receiving ADM on their initial visit were more likely than others to end up in ADM treatment either in PC or SMH (Est = 1.03-0.79). Patients receiving psychotherapy or a psychotherapy referral on their first visit were more likely than others to end up in psychotherapy either in PC (Est = 0.34) or SMH (Est = 0.35). Patients with a PC-MHI encounter on their first visit were more likely than others to end up in PC psychotherapy (Est = 1.25) or PC combined treatment (Est = 0.87).

### Depression Symptom Correlates of Treatment Setting and Type

The proportion of cases classified severe or very severe depression on the QIDS-SR/HRSD and 6 of the 7 depression symptom factors were all elevated among patients in SMH compared with PC (F1 = 7.7-19.1, *P* = .*006*-<0.001), but with relatively modest standardized associations (Est = 0.04-0.05) (Table 3). Treatment types within and between settings show 2 noticeable associations: very severe cases more likely to receive SMH combined treatment (Est = 0.17) and less likely to receive ADM treatment in PC (Est = -0.14); patients with anhedonia, were less likely to receive PC psychotherapy (Est = -0.12), and more likely to receive SMH combined treatment (Est = 0.14).

View this table:
[Table 3.](http://www.jabfm.org/content/34/2/268/T3)

Table 3. 
Associations of Depression Severity with Setting and Type of Treatment (n = 2610)†

### Comorbidity Correlates of Treatment Setting and Type

The results for self-reported comorbidity showed differences between settings on 5 of 10 measures (F1 = 7.5-30.7, *P* = .006-<0.001), mostly due to modestly higher comorbidities among SMH than PC patients (Est = 0.04-0.08) and associations for setting-type combinations (Table 4). Comorbidity was elevated for 3 measures among patients in SMH combined treatment (Est = 0.14-0.15; PTSD, other anxiety, and substance disorders) and for 1 measure among patients in SMH ADM treatment (Est = 0.17, other anxiety). Comorbidity was reduced, in comparison, for PTSD among patients in PC ADM treatment (Est = -0.25) and for anxiety disorder among patients in PC psychotherapy (Est = -0.13). Comorbidity prevalence estimates were much higher when based on EMR data than on self-reported data. Despite the higher prevalence, comorbidity patterns were similar between PC and SMH patients, with only 5 of 10 comorbidity measures showing significant differences (F1 = 9.2.1-32.0, *P* = .002-<0.001). Comorbid PTSD was high among SMH patients on ADM (Est = 0.15) or combined treatment (Est = 0.18) and reduced among PC patients with psychotherapy (Est = -0.16) or ADM (Est = -0.12). Comorbid substance use disorder patients were more likely to receive combined treatment in SMH (Est = 0.21) than in a PC setting (Est = -0.19).

View this table:
[Table 4.](http://www.jabfm.org/content/34/2/268/T4)

Table 4. 
Associations of Comorbidity with Setting and Type of Treatment (n = 2610)†

### Joint Predictive Associations

As many of the statistically significant associations in Tables 2⇑–4 were relatively modest in substantive terms, we estimated a series of ensemble machine learning models to quantify the joint predictive associations of all baseline variables with treatment setting-type (Appendix 7). Cross-validated AUC for integrated PC versus SMH was 0.64, for specific types of PC treatment in the range 0.53-0.68, and for specific types of SMH treatment in the range 0.50-.60. The highest AUC (0.68) was for PC ADM, the treatment type consistently associated with the lowest depression severity-complexity.

## Discussion

This analysis is among the first national studies of depression among VHA patients that linked administrative data with patient-reported symptoms. Three important findings emerged. First, depressed veterans seen in integrated PC have less severe and comorbid episodes on average than those seen in SMH. This finding contrasts with studies in other health care systems, which found mixed evidence for whether depression severity and psychiatric comorbidity were higher among SMH than PC patients.21⇓–23,33,34 Second, within-setting analyses showed that these broad patterns are due largely to patients with the lowest severity-complexity receiving PC ADM treatment and those with the highest severity-complexity receiving SMH combined treatment. These differences are broadly consistent with the goals of PC-MHI. However, third, patients with these setting-type treatment combinations were more similar than different with respect to the predictors examined, as indicated by the fact that sophisticated ensemble machine learning models using all predictors considered along with their interactions to optimize discrimination of patients across settings and treatment types yielded cross-validated AUCs of 0.50-.68. Clinically significant AUCs are typically considered to be at least .70.35

The premise that depression severity is the primary driver explaining treatment decisions is challenged by the weak association of severity with treatment assignment in our data. Other factors, unmeasured or unexplored in this analysis, likely played an important role in treatment decisions. These might include patient factors (eg, care preferences and barriers), provider factors (eg, preferences, time constraints), and system factors (eg, availability of referral resources, incentives). We do not consider the weak association with severity evidence for suboptimal performance of the PC-MHI system but a consequence of treatment providers attempting to adapt VHA recommendations12 to differing patient needs, preferences, and resource constraints. In comparing PC to SMH patients, we expected to see more severe and complex major depressive episode cases receiving SMH combined treatment. However, with no external benchmark against which to compare these results, we consider the weak statistically significant associations found between severity-complexity and treatment type useful information for generating hypotheses in subsequent analyses to explore other determinants.

One reason for weaker than expected associations may be incomplete PC-MHI implementation.36⇓–38 Structural barriers to implementation have been identified and initiatives have been launched to address these barriers,39,40 but this remains a work in progress throughout health care systems, including VHA. It is likely that variation in PC-MHI implementation across sites dilutes the ability of high-functioning collaborative care to optimally tailor the aforementioned factors in ways that are efficient and acceptable to patients. However, in this study we found it challenging to extract reliable indicators of evidence-based PC-MHI implementation from VHA records to examine measures of collaborative care and their relationships with treatment selection. Future studies should evaluate the extent to which patient, provider, and system factors mediate or moderate the relationship between severity-complexity and treatment setting-type.

It is also important to recognize that some mismatch between severity-complexity and treatment setting-type is inevitable even given VHA initiatives to guide treatment assignment given that both PC and PC-MHI function as a safety net for patients who refuse specialty treatment due to stigma or other concerns or are unable to access specialty care due to barriers. This means that the practical alternative to a severe-complex depressed patient getting PC monotherapy, with or without the collaborative assistance of PC-MHI, may be getting no treatment at all rather than getting SMH combined treatment. Both patients and providers can have strong preferences on treatment settings. In addition, patients can have strong feelings about medication or psychotherapy that lead them to demand or refuse treatment types.19,41,42

Controlled studies show that depression treatment engagement is higher and treatment response better when treatments match patient preferences.15,43,44 It is unclear how to weigh this fact in attempting to optimize treatment selection, although it is noteworthy that evidence suggests positive effects of patient preference on outcomes might be limited to situations in which patients had previous successful depression treatment.45 Questions about preferences and past treatment experience were included in our survey, allowing us in future analyses to investigate effects on what seem to be mismatches between severity-complexity and treatment setting-type and subsequently investigate effects of these different factors on treatment response.

These results need to be interpreted within the context of several limitations. First, the low survey response rate could have introduced sample bias despite small discrepancies on administrative variables between the sample and population. Second, the weight introduced because we undersampled patients with ADM only introduced differential sampling that affected statistical power even though it removed bias introduced by the sampling strategy. Third, the generalizability of our results is reduced by our exclusion of patients whose depression was not a presenting problem (Appendix 8) and those who received watchful waiting or active surveillance but did not either receive an ADM prescription or a psychotherapy referral. Fourth, the actual effect of PC-MHI is doubtlessly stronger than the attenuated estimate found here because of variation in PC-MHI implementation and the fact that use of the PC-MHI encounter code is not a guarantee that collaborative care existed in the treatment provided. Similar to coding inaccuracies of diagnostic data within VHA,46 coding of PC-MHI has been identified as a potential source of error in other studies.9,10 Fifth, baseline assessments were made between 4 and 7 days after the initial visit. To the extent that symptoms diminished within 4-7 days of a first visit and there is mood-congruent recall bias, the proportion of patients reporting severe depression might be lower than if assessment had occurred on the day of first visit. Sixth, we did not investigate influences of treatment history or patient preferences in determining setting or type of treatment. Given that interventions that incorporate patient preferences are associated with positive outcomes,13,15,20 further examination of these factors is warranted.

## Conclusions

Within the context of these limitations, we found statistically significant associations of depression severity-complexity with treatment setting-type similar to those found for other collaborative care applications in civilian samples. With increasing adoption of collaborative care principles (ie, shifting mental health services for less severe cases to PC, with shared treatment responsibilities) in the VHA7,47 and other health systems,48 continuous monitoring of the distribution of patients in primary and specialty settings as well as delivery of treatments consistent with the collaborative care model will aid in continuous improvement of programs that attend to specific mental health needs of the patient population.

## Acknowledgments

Thanks to Irving Hwang, Elizabeth Karras-Pilato, Janet McCarten, and Nancy Sampson for technical assistance and helpful comments on an earlier draft.

## Appendix

View this table:
[Appendix 1.](http://www.jabfm.org/content/34/2/268/T5)

Appendix 1. 
ICD-9-CM1 and ICD-10-CM2 Mental and Behavioral Health Codes

## Appendix 2. Self-Report Measures of Depression and Psychiatric Comorbidity

### Depression Symptom Severity

It was noted in the text that depression symptom severity in the 2 weeks before seeking treatment was assessed in the self-report survey with the 16-item self-report Quick Inventory of Depressive Symptomatology Self-Report Scale (QIDS-SR)3 and that scores on this scale were converted into estimated severity levels on the Hamilton Rating Scale of Depression (HRSD).4 In addition, we asked 8 questions about melancholic features from the full Inventory of Depressive Symptomatology Self-Report Scale (IDS-SR)5; 5 questions about anhedonia based on the Dimensional Anhedonia Rating Scale (DARS)6 and the Motivation and Pleasure Scale–Self-Report (MAP-SR)7; the 4-question reduced version of the Beck Hopelessness Scale8; 4 questions about difficulties with concentration and decision-making from the PROMIS Applied Cognition-Abilities Short Form 4a9; 6 questions to operationalize Criterion A of the DSM-5 Mixed Features Specifier for Depressive Disorders10 based on the cyclothymic temperament subscale of the Temperament Evaluation of Memphis, Pisa, Paris and San Diego-autoquestionnaire (TEMPS-A)11; and 7 questions from the 10-question Ruminative Responses Scale-Short Form (RSS-SF).12

We also assessed the 6 dimensions other than depressive symptom severity in the self-report Remission from Depression Questionnaire (RDQ).13 These dimensions were found in developmental research for the RDQ to be the most important ones for patients in defining recovery from a depressive episode. The 6 dimensions include other symptoms that often co-occur with depression (anxiety, irritability), role functioning, coping ability, and 3 positive dimensions (life satisfaction, general sense of well-being, and positive sense of mental health). Symptoms were assessed using the same 2-week recall period as in the QIDS-SR. The first dimension was operationalized with questions from the RDQ,13 the full IDS-SR,5 and the Composite International Diagnostic Interview (CIDI)14 on irritability and the DSM-5 Anxious Distress Specifier for Depressive Disorders.10 The second dimension was operationalized with the Sheehan Disability Scale.15 The last 4 dimensions were operationalized with 3 items per dimension from the RDQ.13

### Depression Persistence

Questions from the CIDI14 were used to obtain retrospective assessments of depression age of onset, number of years with at least 1 month of depression sufficiently severe to cause substantial distress or interference with functioning, and length of the current depressive episode.

### Comorbidity

Patients were asked which of their presenting problems were primary and secondary. As noted in the text, the few patients who reported bipolar disorder were excluded from further study. Other presenting complaints and diagnoses recorded as the focus of clinical attention in VHA records were collapsed into 6 categories: post-traumatic stress disorder (PTSD), other anxiety disorders, substance use disorder, anger problems, other emotional problems, and somatic symptom disorder. In addition, we administered brief dimensional self-report screening scales for comorbid disorder dimensions of special interest: a 6-question version of the PTSD Checklist for DSM-5 (PCL-5)16 calibrated to the full PCL-5 based on analysis of the Army STARRS data17; 4 questions from the CIDI14 to operationalize the DSM-5 PTSD Dissociative Symptom Specifier10; the 7-question PROMIS Alcohol/Substance Use Short Form-7a18 to assess symptoms of substance use disorder and scored using the recommended PROMIS scoring procedures19; and the 12-question Somatic Symptoms Disorder-B Criteria Scale (SSD-12)20,21 to operationalize the 3 psychological criteria (cognitive, affective, behavioral) of DSM-5 Somatic Symptom Disorder.10

## Appendix

View this table:
[Appendix 3.](http://www.jabfm.org/content/34/2/268/T6)

Appendix 3. 
Brief Description of Machine Learning Algorithms Included in the SuperLearner Library

## Appendix

![Appendix 4.](http://www.jabfm.org/https://www.jabfm.org/content/jabfp/34/2/268/F1.medium.gif)

[Appendix 4.](http://www.jabfm.org/content/34/2/268/F1)

Appendix 4. 
Patients Seen for Incident Depression as Reported in the Veterans Health Administration Electronic Medical Records from December 2018 through June 2020

## Appendix

View this table:
[Appendix 5.](http://www.jabfm.org/content/34/2/268/T7)

Appendix 5. 
Factor Analysis (n = 2,169)†

## Appendix

View this table:
[Appendix 6.](http://www.jabfm.org/content/34/2/268/T8)

Appendix 6. 
Correlations Among Factors in Promax Rotated Factor Solution

## Appendix

View this table:
[Appendix 7.](http://www.jabfm.org/content/34/2/268/T9)

Appendix 7. 
AUC of 10-fold Externally Cross-Validated SuperLearner Models†

## Appendix

View this table:
[Appendix 8.](http://www.jabfm.org/content/34/2/268/T10)

Appendix 8. 
Comparison of Primary Problems and Diagnoses between Analytic Sample and Those Who Did Not Report Depression as a Problem

## Notes

*   This article was externally peer reviewed.

*   *Funding:* This research was supported by the Office of Mental Health Services and Suicide Prevention and Center of Excellence for Suicide Prevention (RMB); RCK was supported by National Institute of Mental Health of the National Institutes of Health under award number R01MH121478; LBL was funded by Career Development Award IK2 HX002867 from the United States Department of Veterans Affairs Health Services Research & Development Service.

*   *Conflict of interest:* In the past 3 years, RCK was a consultant for Datastat, Inc., RallyPoint Networks, Inc., Sage Pharmaceuticals, and Takeda. The remaining authors report no conflict of interest.

*   To see this article online, please go to: [http://jabfm.org/content/34/2/268.full](http://jabfm.org/content/34/2/268.full).

*   Received for publication September 11, 2020.
*   Revision received December 1, 2020.
*   Accepted for publication December 2, 2020.

## References

1.  1.Liu Y, Collins C, Wang K, Xie X, Bie R. The prevalence and trend of depression among veterans in the United States. J Affect Disord 2019;245:724–7.
    
    

2.  2.Trivedi RB, Post EP, Sun H, et al. Prevalence, comorbidity, and prognosis of mental health among US veterans. Am J Public Health 2015;105:2564–9.
    
    

3.  3.Ziobrowski H, Sartor CE, Tsai J, Pietrzak RH. Gender differences in mental and physical health conditions in U.S. veterans: results from the National Health and Resilience in Veterans Study. J Psychosom Res 2017;101:110–3.
    
    

4.  4.Hoerster KD, Lehavot K, Simpson T, McFall M, Reiber G, Nelson KM. Health and health behavior differences: U.S. military, veteran, and civilian men. Am J Prev Med 2012;43:483–9.
    
    [CrossRef](http://www.jabfm.org/lookup/external-ref?access_num=10.1016/j.amepre.2012.07.029&link_type=DOI) 
    
    [PubMed](http://www.jabfm.org/lookup/external-ref?access_num=23079170&link_type=MED&atom=%2Fjabfp%2F34%2F2%2F268.atom) 

5.  5.Lehavot K, Hoerster KD, Nelson KM, Jakupcak M, Simpson TL. Health indicators for military, veteran, and civilian women. Am J Prev Med 2012;42:473–80.
    
    [CrossRef](http://www.jabfm.org/lookup/external-ref?access_num=10.1016/j.amepre.2012.01.006&link_type=DOI) 
    
    [PubMed](http://www.jabfm.org/lookup/external-ref?access_num=22516487&link_type=MED&atom=%2Fjabfp%2F34%2F2%2F268.atom) 

6.  6.Pemberton MR, Forman-Hoffman VL, Lipari RN, Ashley OS, Heller DC, Williams MR. Prevalence of past year substance use and mental illness by veteran status in a nationally representative sample [press release]. Available from: [https://www.samhsa.gov/data/sites/default/files/NSDUH-DR-VeteranTrends-2016/NSDUH-DR-VeteranTrends-2016.htm](https://www.samhsa.gov/data/sites/default/files/NSDUH-DR-VeteranTrends-2016/NSDUH-DR-VeteranTrends-2016.htm). November 2016.
    
    

7.  7.Post EP, Metzger M, Dumas P, Lehmann L. Integrating mental health into primary care within the Veterans Health Administration. Fam Syst Health 2010;28:83–90.
    
    [CrossRef](http://www.jabfm.org/lookup/external-ref?access_num=10.1037/a0020130&link_type=DOI) 
    
    [PubMed](http://www.jabfm.org/lookup/external-ref?access_num=20695668&link_type=MED&atom=%2Fjabfp%2F34%2F2%2F268.atom) 

8.  8.Jones AL, Mor MK, Haas GL, et al. The role of primary care experiences in obtaining treatment for depression. J Gen Intern Med 2018;33:1366–73.
    
    

9.  9.Leung LB, Rubenstein LV, Yoon J, et al. Veterans Health Administration investments in primary care and mental health integration improved care access. Health Aff (Millwood) 2019;38:1281–8.
    
    

10. 10.Leung LB, Yoon J, Escarce JJ, et al. Primary care-mental health integration in the VA: shifting mental health services for common mental illnesses to primary care. Psychiatry Services 2018;69:403–9.
    
    

11. 11.Leung LB, Yoon J, Rubenstein LV, et al. Changing patterns of mental health care use: the role of integrated mental health services in Veterans Affairs primary care. J Am Board Fam Med 2018;31:38–48.
    
    [Abstract/FREE Full Text](http://www.jabfm.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NToiamFiZnAiO3M6NToicmVzaWQiO3M6NzoiMzEvMS8zOCI7czo0OiJhdG9tIjtzOjIwOiIvamFiZnAvMzQvMi8yNjguYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

12. 12.U.S. Department of Veteran Affairs. Clinical practice guideline for the management of major depressive disorder. Available from: [https://www.healthquality.va.gov/guidelines/MH/mdd/VADoDMDDCPGFINAL82916.pdf](https://www.healthquality.va.gov/guidelines/MH/mdd/VADoDMDDCPGFINAL82916.pdf). Version 3.0. 2016.
    
    

13. 13.Dwight-Johnson M, Unutzer J, Sherbourne C, Tang L, Wells KB. Can quality improvement programs for depression in primary care address patient preferences for treatment? Med Care 2001;39:934–44.
    
    [CrossRef](http://www.jabfm.org/lookup/external-ref?access_num=10.1097/00005650-200109000-00004&link_type=DOI) 
    
    [PubMed](http://www.jabfm.org/lookup/external-ref?access_num=11502951&link_type=MED&atom=%2Fjabfp%2F34%2F2%2F268.atom) 
    
    [Web of Science](http://www.jabfm.org/lookup/external-ref?access_num=000170925200004&link_type=ISI) 

14. 14.Kirchner JE, Owen RR, Dockter N, et al. Equity in veterans' mental health care: Veterans Affairs medical center clinics versus community-based outpatient clinics. Am J Med Qual 2008;23:128–35.
    
    [CrossRef](http://www.jabfm.org/lookup/external-ref?access_num=10.1177/1062860607313144&link_type=DOI) 
    
    [PubMed](http://www.jabfm.org/lookup/external-ref?access_num=18230869&link_type=MED&atom=%2Fjabfp%2F34%2F2%2F268.atom) 

15. 15.Lin P, Campbell DG, Chaney EF, et al. The influence of patient preference on depression treatment in primary care. Ann Behav Med 2005;30:164–73.
    
    [CrossRef](http://www.jabfm.org/lookup/external-ref?access_num=10.1207/s15324796abm3002_9&link_type=DOI) 
    
    [PubMed](http://www.jabfm.org/lookup/external-ref?access_num=16173913&link_type=MED&atom=%2Fjabfp%2F34%2F2%2F268.atom) 
    
    [Web of Science](http://www.jabfm.org/lookup/external-ref?access_num=000232431600009&link_type=ISI) 

16. 16.Loeb DF, Bayliss EA, Binswanger IA, Candrian C, deGruy FV. Primary care physician perceptions on caring for complex patients with medical and mental illness. J Gen Intern Med 2012;27:945–52.
    
    [CrossRef](http://www.jabfm.org/lookup/external-ref?access_num=10.1007/s11606-012-2005-9&link_type=DOI) 
    
    [PubMed](http://www.jabfm.org/lookup/external-ref?access_num=22370766&link_type=MED&atom=%2Fjabfp%2F34%2F2%2F268.atom) 

17. 17.Loeb DF, Bayliss EA, Candrian C, deGruy FV, Binswanger IA. Primary care providers' experiences caring for complex patients in primary care: a qualitative study. BMC Fam Pract 2016;17:34.
    
    

18. 18.Maciejewski ML, Perkins M, Li YF, Chapko M, Fortney JC, Liu CF. Utilization and expenditures of veterans obtaining primary care in community clinics and VA medical centers: an observational cohort study. BMC Health Serv Res 2007;7:56.
    
    [CrossRef](http://www.jabfm.org/lookup/external-ref?access_num=10.1186/1472-6963-7-56&link_type=DOI) 
    
    [PubMed](http://www.jabfm.org/lookup/external-ref?access_num=17442115&link_type=MED&atom=%2Fjabfp%2F34%2F2%2F268.atom) 

19. 19.Moradveisi L, Huibers M, Renner F, Arntz A. The influence of patients' preference/attitude towards psychotherapy and antidepressant medication on the treatment of major depressive disorder. J Behav Ther Exp Psychiatry 2014;45:170–7.
    
    [CrossRef](http://www.jabfm.org/lookup/external-ref?access_num=10.1016/j.jbtep.2013.10.003&link_type=DOI) 
    
    [PubMed](http://www.jabfm.org/lookup/external-ref?access_num=24172145&link_type=MED&atom=%2Fjabfp%2F34%2F2%2F268.atom) 

20. 20.van Schaik DJ, Klijn AF, van Hout HP, et al. Patients' preferences in the treatment of depressive disorder in primary care. Gen Hosp Psychiatry 2004;26:184–9.
    
    [CrossRef](http://www.jabfm.org/lookup/external-ref?access_num=10.1016/j.genhosppsych.2003.12.001&link_type=DOI) 
    
    [PubMed](http://www.jabfm.org/lookup/external-ref?access_num=15121346&link_type=MED&atom=%2Fjabfp%2F34%2F2%2F268.atom) 
    
    [Web of Science](http://www.jabfm.org/lookup/external-ref?access_num=000221448500003&link_type=ISI) 

21. 21.Bhalla IP, Stefanovics EA, Rosenheck RA. Social determinants of mental health care systems: intensive community based care in the Veterans Health Administration. BMC Public Health 2020;20:1311.
    
    

22. 22.Kuramoto-Crawford SJ, Han B, Jacobus-Kantor L, Mojtabai R. Receipt of depression treatment from general medical providers and specialty mental health providers. Psychiatr Serv 2016;67:758–65.
    
    

23. 23.Mojtabai R, Olfson M. National patterns in antidepressant treatment by psychiatrists and general medical providers: results from the national comorbidity survey replication. J Clin Psychiatry 2008;69:1064–74.
    
    [CrossRef](http://www.jabfm.org/lookup/external-ref?access_num=10.4088/JCP.v69n0704&link_type=DOI) 
    
    [PubMed](http://www.jabfm.org/lookup/external-ref?access_num=18399725&link_type=MED&atom=%2Fjabfp%2F34%2F2%2F268.atom) 
    
    [Web of Science](http://www.jabfm.org/lookup/external-ref?access_num=000258007700004&link_type=ISI) 

24. 24.Kuramoto-Crawford SJ, Han B, Jacobus-Kantor L, Mojtabai R. Differences in patients' perceived helpfulness of depression treatment provided by general medical providers and specialty mental health providers. Gen Hosp Psychiatry 2015;37:340–6.
    
    

25. 25.Pomerantz JM, Finkelstein SN, Berndt ER, et al. Prescriber intent, off-label usage, and early discontinuation of antidepressants: a retrospective physician survey and data analysis. J Clin Psychiatry 2004;65:395–404.
    
    [PubMed](http://www.jabfm.org/lookup/external-ref?access_num=15096080&link_type=MED&atom=%2Fjabfp%2F34%2F2%2F268.atom) 

26. 26.Rush AJ, Trivedi MH, Ibrahim HM, et al. The 16-item Quick Inventory of Depressive Symptomatology (QIDS), Clinician Rating (QIDS-C), and Self-Report (QIDS-SR): a psychometric evaluation in patients with chronic major depression. Biol Psychiatry 2003;54:573–83.
    
    [CrossRef](http://www.jabfm.org/lookup/external-ref?access_num=10.1016/S0006-3223(02)01866-8&link_type=DOI) 
    
    [PubMed](http://www.jabfm.org/lookup/external-ref?access_num=12946886&link_type=MED&atom=%2Fjabfp%2F34%2F2%2F268.atom) 
    
    [Web of Science](http://www.jabfm.org/lookup/external-ref?access_num=000184906100009&link_type=ISI) 

27. 27.IDS/QUIDS. Interpretation: Inventory of Depressive Symptomatology (IDS) and Quick Inventory of Depressive Symptomatology (QIDS). 2020. Available from: [http://www.ids-qids.org/interpretation.html](http://www.ids-qids.org/interpretation.html).
    
    

28. 28.Kessler RC, Ustün TB. The World Mental Health (WMH) survey initiative version of the World Health Organization (WHO) Composite International Diagnostic Interview (CIDI). Int J Methods Psychiatr Res 2004;13:93–121.
    
    [CrossRef](http://www.jabfm.org/lookup/external-ref?access_num=10.1002/mpr.168&link_type=DOI) 
    
    [PubMed](http://www.jabfm.org/lookup/external-ref?access_num=15297906&link_type=MED&atom=%2Fjabfp%2F34%2F2%2F268.atom) 
    
    [Web of Science](http://www.jabfm.org/lookup/external-ref?access_num=000223439600004&link_type=ISI) 

29. 29.Zubizaretta JR, Li Y, Allouah A, Greifer N. sbw: stable balancing weights for causal inference and estimation with incomplete outcome data [computer program]. 2020. Available from: [https://cran.rstudio.com/web/packages/sbw/](https://cran.rstudio.com/web/packages/sbw/). Version 1.1.1.
    
    

30. 30.Zubizarreta JR. Stable weights that balance covariates for estimation with incomplete outcome data. J Amer Statist Assn 2015;110:910–22.
    
    

31. 31.Benjamini Y, Yekutieli D. The control of the false discovery rate in multiple testing under dependency. Ann Stats 2001;1165–88.
    
    

32. 32.Polley EC, Rose S, van der Laan MJ. Super learning. New York: Springer; 2011:43–66.
    
    

33. 33.Gaynes BN, Rush AJ, Trivedi M, et al. A direct comparison of presenting characteristics of depressed outpatients from primary vs. specialty care settings: preliminary findings from the STAR*D clinical trial. General Hospital Psychiatry 2005;27:87–96.
    
    [CrossRef](http://www.jabfm.org/lookup/external-ref?access_num=10.1016/j.genhosppsych.2004.10.003&link_type=DOI) 
    
    [PubMed](http://www.jabfm.org/lookup/external-ref?access_num=15763119&link_type=MED&atom=%2Fjabfp%2F34%2F2%2F268.atom) 
    
    [Web of Science](http://www.jabfm.org/lookup/external-ref?access_num=000228035000001&link_type=ISI) 

34. 34.Gaynes BN, Rush AJ, Trivedi MH, et al. Major depression symptoms in primary care and psychiatric care settings: a cross-sectional analysis. Ann Fam Med 2007;5:126–34.
    
    [Abstract/FREE Full Text](http://www.jabfm.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6ODoiYW5uYWxzZm0iO3M6NToicmVzaWQiO3M6NzoiNS8yLzEyNiI7czo0OiJhdG9tIjtzOjIwOiIvamFiZnAvMzQvMi8yNjguYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

35. 35.Mandrekar JN. Receiver operating characteristic curve in diagnostic test assessment. J Thorac Oncol 2010;5:1315–6.
    
    [CrossRef](http://www.jabfm.org/lookup/external-ref?access_num=10.1097/JTO.0b013e3181ec173d&link_type=DOI) 
    
    [PubMed](http://www.jabfm.org/lookup/external-ref?access_num=20736804&link_type=MED&atom=%2Fjabfp%2F34%2F2%2F268.atom) 

36. 36.Beehler GP, Lilienthal KR, Possemato K, et al. Narrative review of provider behavior in primary care behavioral health: how process data can inform quality improvement. Fam Syst Health 2017;35:257–70.
    
    

37. 37.Chang ET, Rose DE, Yano EM, et al. Determinants of readiness for primary care-mental health integration (PC-MHI) in the VA health care system. J Gen Intern Med 2013;28:353–62.
    
    [CrossRef](http://www.jabfm.org/lookup/external-ref?access_num=10.1007/s11606-012-2217-z&link_type=DOI) 
    
    [PubMed](http://www.jabfm.org/lookup/external-ref?access_num=23054917&link_type=MED&atom=%2Fjabfp%2F34%2F2%2F268.atom) 

38. 38.Ritchie MJ, Kirchner JE, Townsend JC, Pitcock JA, Dollar KM, Liu CF. Time and organizational cost for facilitating implementation of primary care mental health integration. J Gen Intern Med 2020;35:1001–10.
    
    

39. 39.Cohen DJ, Balasubramanian BA, Davis M, et al. Understanding care integration from the ground up: five organizing constructs that shape integrated practices. J Am Board Fam Med 2015;28:S7–20.
    
    [Abstract/FREE Full Text](http://www.jabfm.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NToiamFiZnAiO3M6NToicmVzaWQiO3M6MTg6IjI4L1N1cHBsZW1lbnRfMS9TNyI7czo0OiJhdG9tIjtzOjIwOiIvamFiZnAvMzQvMi8yNjguYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

40. 40.Solberg LI, Crain AL, Jaeckels N, et al. The DIAMOND initiative: implementing collaborative care for depression in 75 primary care clinics. Implementation Sci 2013;8:135.
    
    

41. 41.Kwan BM, Dimidjian S, Rizvi SL. Treatment preference, engagement, and clinical improvement in pharmacotherapy versus psychotherapy for depression. Behav Res Ther 2010;48:799–804.
    
    [CrossRef](http://www.jabfm.org/lookup/external-ref?access_num=10.1016/j.brat.2010.04.003&link_type=DOI) 
    
    [PubMed](http://www.jabfm.org/lookup/external-ref?access_num=20462569&link_type=MED&atom=%2Fjabfp%2F34%2F2%2F268.atom) 

42. 42.McHugh RK, Whitton SW, Peckham AD, Welge JA, Otto MW. Patient preference for psychological vs pharmacologic treatment of psychiatric disorders: a meta-analytic review. J Clin Psychiatry 2013;74:595–602.
    
    [CrossRef](http://www.jabfm.org/lookup/external-ref?access_num=10.4088/JCP.12r07757&link_type=DOI) 
    
    [PubMed](http://www.jabfm.org/lookup/external-ref?access_num=23842011&link_type=MED&atom=%2Fjabfp%2F34%2F2%2F268.atom) 
    
    [Web of Science](http://www.jabfm.org/lookup/external-ref?access_num=000321318900004&link_type=ISI) 

43. 43.Gelhorn HL, Sexton CC, Classi PM. Patient preferences for treatment of major depressive disorder and the impact on health outcomes: a systematic review. Prim Care Companion CNS Disord 2011;13:PCC.11r01161.
    
    

44. 44.Lindhiem O, Bennett CB, Trentacosta CJ, McLear C. Client preferences affect treatment satisfaction, completion, and clinical outcome: a meta-analysis. Clinical Psychology Review 2014;34:506–17.
    
    [CrossRef](http://www.jabfm.org/lookup/external-ref?access_num=10.1016/j.cpr.2014.06.002&link_type=DOI) 
    
    [PubMed](http://www.jabfm.org/lookup/external-ref?access_num=25189522&link_type=MED&atom=%2Fjabfp%2F34%2F2%2F268.atom) 

45. 45.Dunlop BW, Kelley ME, Aponte-Rivera V, for the PReDICT Team, et al. Effects of patient preferences on outcomes in the Predictors of Remission in Depression to Individual and Combined Treatments (PReDICT) Study. AJP 2017;174:546–56.
    
    

46. 46.Yoon J, Chow A. Comparing chronic condition rates using ICD-9 and ICD-10 in VA patients FY2014-2016. BMC Health Serv Res 2017;17:572.
    
    [PubMed](http://www.jabfm.org/lookup/external-ref?access_num=28818082&link_type=MED&atom=%2Fjabfp%2F34%2F2%2F268.atom) 

47. 47.Pomerantz AS, Sayers SL. Primary care-mental health integration in healthcare in the Department of Veterans Affairs. Fam Syst Health 2010;28:78–82.
    
    [CrossRef](http://www.jabfm.org/lookup/external-ref?access_num=10.1037/a0020341&link_type=DOI) 
    
    [PubMed](http://www.jabfm.org/lookup/external-ref?access_num=20695667&link_type=MED&atom=%2Fjabfp%2F34%2F2%2F268.atom) 

48. 48.Butler M, Kane RL, McAlpine D, et al. Integration of Mental Health/Substance Abuse and Primary Care No. 173 (prepared by the Minnesota Evidence-based Practice Center under Contract No. 290–02-0009). 2008. Agency for Healthcare Research and Quality.
    
    

## References for Appendices

1.  1.Centers for Disease Control and Prevention. International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM). 2013. Available from: [http://www.cdc.gov/nchs/icd/icd9cm.htm](http://www.cdc.gov/nchs/icd/icd9cm.htm).
    
    

2.  2.Centers for Disease Control and Prevention. International Classification of Diseases, Tenth Revision, Clinical Modification (ICD-10-CM. 2019. Available from: [https://www.cdc.gov/nchs/icd/icd10cm.htm](https://www.cdc.gov/nchs/icd/icd10cm.htm).
    
    

3.  3.Rush AJ, Trivedi MH, Ibrahim HM, et al. The 16-Item Quick Inventory of Depressive Symptomatology (QIDS), Clinician Rating (QIDS-C), and Self-Report (QIDS-SR): a psychometric evaluation in patients with chronic major depression. Biol Psychiatry 2003;54:573–83.
    
    [CrossRef](http://www.jabfm.org/lookup/external-ref?access_num=10.1016/S0006-3223(02)01866-8&link_type=DOI) 
    
    [PubMed](http://www.jabfm.org/lookup/external-ref?access_num=12946886&link_type=MED&atom=%2Fjabfp%2F34%2F2%2F268.atom) 
    
    [Web of Science](http://www.jabfm.org/lookup/external-ref?access_num=000184906100009&link_type=ISI) 

4.  4.IDS/QUIDS. Interpretation: Inventory of Depressive Symptomatology (IDS) and Quick Inventory of Depressive Symptomatology (QIDS). 2020. Available from: [http://www.ids-qids.org/interpretation.html](http://www.ids-qids.org/interpretation.html).
    
    

5.  5.Rush AJ, Gullion CM, Basco MR, Jarrett RB, Trivedi MH. The Inventory of Depressive Symptomatology (IDS): psychometric properties. Psychol Med 1996;26:477–86.
    
    [CrossRef](http://www.jabfm.org/lookup/external-ref?access_num=10.1017/S0033291700035558&link_type=DOI) 
    
    [PubMed](http://www.jabfm.org/lookup/external-ref?access_num=8733206&link_type=MED&atom=%2Fjabfp%2F34%2F2%2F268.atom) 
    
    [Web of Science](http://www.jabfm.org/lookup/external-ref?access_num=A1996UK25900004&link_type=ISI) 

6.  6.Rizvi SJ, Quilty LC, Sproule BA, Cyriac A, Michael Bagby R, Kennedy SH. Development and validation of the Dimensional Anhedonia Rating Scale (DARS) in a community sample and individuals with major depression. Psychiatry Res 2015;229:109–19.
    
    

7.  7.Llerena K, Park SG, McCarthy JM, Couture SM, Bennett ME, Blanchard JJ. The Motivation and Pleasure Scale–Self-Report (MAP-SR): reliability and validity of a self-report measure of negative symptoms. Compr Psychiatry 2013;54:568–74.
    
    [CrossRef](http://www.jabfm.org/lookup/external-ref?access_num=10.1016/j.comppsych.2012.12.001&link_type=DOI) 
    
    [PubMed](http://www.jabfm.org/lookup/external-ref?access_num=23351831&link_type=MED&atom=%2Fjabfp%2F34%2F2%2F268.atom) 

8.  8.Aish AM, Wasserman D, Renberg ES. Does Beck's Hopelessness Scale really measure several components? Psychol Med 2001;31:367–72.
    
    [CrossRef](http://www.jabfm.org/lookup/external-ref?access_num=10.1017/S0033291701003300&link_type=DOI) 
    
    [PubMed](http://www.jabfm.org/lookup/external-ref?access_num=11232923&link_type=MED&atom=%2Fjabfp%2F34%2F2%2F268.atom) 
    
    [Web of Science](http://www.jabfm.org/lookup/external-ref?access_num=000167004300017&link_type=ISI) 

9.  9.Saffer BY, Lanting SC, Koehle MS, Klonsky ED, Iverson GL. Assessing cognitive impairment using PROMIS® Applied Cognition-Abilities scales in a medical outpatient sample. Psychiatry Res 2015;226:169–72.
    
    

10. 10.American Psychiatric Association. Diagnostic and statistical manual of mental disorders: DSM-5. Fifth Edition. Arlington, VA: American Psychiatric Association; 2013.
    
    

11. 11.Akiskal HS, Mendlowicz MV, Jean-Louis G, et al. TEMPS-A: validation of a short version of a self-rated instrument designed to measure variations in temperament. J Affect Disord 2005;85:45–52.
    
    [CrossRef](http://www.jabfm.org/lookup/external-ref?access_num=10.1016/j.jad.2003.10.012&link_type=DOI) 
    
    [PubMed](http://www.jabfm.org/lookup/external-ref?access_num=15780675&link_type=MED&atom=%2Fjabfp%2F34%2F2%2F268.atom) 
    
    [Web of Science](http://www.jabfm.org/lookup/external-ref?access_num=000228409400006&link_type=ISI) 

12. 12.Treynor W, Gonzalez R, Nolen-Hoeksema S. Rumination reconsidered: a psychometric analysis. Cognit Ther Res 2003;27:247–59.
    
    [CrossRef](http://www.jabfm.org/lookup/external-ref?access_num=10.1023/A:1023910315561&link_type=DOI) 
    
    [Web of Science](http://www.jabfm.org/lookup/external-ref?access_num=000183205900002&link_type=ISI) 

13. 13.Zimmerman M, Martinez JH, Attiullah N, et al. A new type of scale for determining remission from depression: the Remission from Depression Questionnaire. J Psych Res 2013;47:78–82.
    
    

14. 14.Kessler RC, Ustün TB. The World Mental Health (WMH) survey initiative version of the World Health Organization (WHO) Composite International Diagnostic Interview (CIDI). Int J Methods Psychiatr Res 2004;13:93–121.
    
    [CrossRef](http://www.jabfm.org/lookup/external-ref?access_num=10.1002/mpr.168&link_type=DOI) 
    
    [PubMed](http://www.jabfm.org/lookup/external-ref?access_num=15297906&link_type=MED&atom=%2Fjabfp%2F34%2F2%2F268.atom) 
    
    [Web of Science](http://www.jabfm.org/lookup/external-ref?access_num=000223439600004&link_type=ISI) 

15. 15.Leon AC, Olfson M, Portera L, Farber L, Sheehan DV. Assessing psychiatric impairment in primary care with the Sheehan Disability Scale. Int J Psychiatry Med 1997;27:93–105.
    
    [CrossRef](http://www.jabfm.org/lookup/external-ref?access_num=10.2190/T8EM-C8YH-373N-1UWD&link_type=DOI) 
    
    [PubMed](http://www.jabfm.org/lookup/external-ref?access_num=9565717&link_type=MED&atom=%2Fjabfp%2F34%2F2%2F268.atom) 
    
    [Web of Science](http://www.jabfm.org/lookup/external-ref?access_num=A1997XX22000002&link_type=ISI) 

16. 16.Blevins CA, Weathers FW, Davis MT, Witte TK, Domino JL. The Posttraumatic Stress Disorder Checklist for DSM-5 (PCL-5): development and initial psychometric evaluation. J Traum Stress 2015;28:489–98.
    
    

17. 17.Zuromski KL, Ustun B, Hwang I, et al. Developing an optimal short-form of the PTSD Checklist for DSM-5 (PCL-5). Depress Anxiety 2019;36:790–800.
    
    

18. 18.Gibbons LE, Fredericksen R, Merrill JO, et al. Suitability of the PROMIS alcohol use short form for screening in a HIV clinical care setting. Drug Alcohol Depend 2016;164:113–9.
    
    

19. 19.Patient-Reported Outcomes Measurement Information System (PROMIS) Health Organization. PROMIS Alcohol Use Scoring Manual. 2014. Available from: [https://www.healthmeasures.net/images/promis/manuals/PROMIS\_Alcohol\_Use\_Scoring\_Manual.pdf](https://www.healthmeasures.net/images/promis/manuals/PROMIS\_Alcohol_Use_Scoring_Manual.pdf).
    
    

20. 20.Limburg K, Sattel H, Dinkel A, Radziej K, Becker-Bense S, Lahmann C. Course and predictors of DSM-5 somatic symptom disorder in patients with vertigo and dizziness symptoms—a longitudinal study. Compr Psychiatry 2017;77:1–11.
    
    

21. 21.Toussaint A, Murray AM, Voigt K, et al. Development and validation of the Somatic Symptom Disorder-B Criteria Scale (SSD-12). Psychosomatic Medicine 2016;78:5–12.
    
    [CrossRef](http://www.jabfm.org/lookup/external-ref?access_num=10.1097/PSY.0000000000000240&link_type=DOI) 
    
    [PubMed](http://www.jabfm.org/lookup/external-ref?access_num=26461855&link_type=MED&atom=%2Fjabfp%2F34%2F2%2F268.atom) 

22. 22.Friedman J, Hastie T, Tibshirani R. Regularization paths for generalized linear models via coordinate descent. J Stat Softw 2010;33:1–22.
    
    [CrossRef](http://www.jabfm.org/lookup/external-ref?access_num=10.1111/j.1467-9868.2005.00503.x&link_type=DOI) 
    
    [PubMed](http://www.jabfm.org/lookup/external-ref?access_num=20808728&link_type=MED&atom=%2Fjabfp%2F34%2F2%2F268.atom) 
    
    [Web of Science](http://www.jabfm.org/lookup/external-ref?access_num=000275203200001&link_type=ISI) 

23. 23.Friedman JH. Multivariate adaptive regression splines. Ann Statist 1991;19:1–67.
    
    

24. 24.Stone CJ, Hansen MH, Kooperberg C, Truong YK. Polynomial splines and their tensor products in extended linear modeling. Ann Stats 1997;25:1371–425.
    
    

25. 25.Breiman L. Random forests. Machine Learning 2001;45:5–32.
    
    

26. 26.Friedman JH. Greedy function approximation: a gradient boosting machine. Ann Statist 2001;29:1189–232.
    
    [CrossRef](http://www.jabfm.org/lookup/external-ref?access_num=10.1214/aos/1013203451&link_type=DOI) 

27. 27.Friedman J, Hastie T, Tibshirani R. Additive logistic regression: a statistical view of boosting (with discussion and a rejoinder by the authors). Ann Statist 2000;28:337–407.
    
    

28. 28.Venables WN, Ripley BD. Modern applied atatistics with S. 4th ed. New York: Springer; 2002.
    
    

29. 29.Crawford AV, Green SB, Levy R, et al. Evaluation of parallel analysis methods for determining the number of factors. Educ Psychol Measurement 2010;70:885–901.