Abstract
Background: The Veterans Health Administration (VA) Women's Health Practice-Based Research Network (WH-PBRN) was created to foster innovations for the health care of women veterans. The inaugural study by the WH-PBRN was designed to identify women veterans' own priorities and preferences for mental health services and to inform refinements to WH-PBRN operational procedures. Addressing the latter, this article reports lessons learned from the inaugural study.
Methods: WH-PBRN site coordinators at the 4 participating sites convened weekly with the study coordinator and the WH-PBRN program manager to address logistical issues and identify lessons learned. Findings were categorized into a matrix of challenges and facilitators related to key study elements.
Results: Challenges to the conduct of PBRN-based research included tracking of regulatory documents; cross-site variability in some regulatory processes; and troubleshooting logistics of clinic-based recruitment. Facilitators included a central institutional review board, strong relationships between WH-PBRN research teams and women's health clinic teams, and the perception that women want to help other women veterans.
Conclusion: Our experience with the inaugural WH-PBRN study demonstrated the feasibility of establishing productive relationships between local clinicians and researchers, and of recruiting a special population (women veterans) in diverse sites within an integrated health care system. This identified strengths of a PBRN approach.
Women veterans, as only 6.5% of Veterans Health Administration (VA) users,1 represent an extreme numeric minority group in a predominantly male system. Created in June 2010, the VA Women's Health Practice-Based Research Network (WH-PBRN) seeks to address historic gaps in the evidence base informing the health care of women veterans2 by providing research infrastructure supporting multisite women's health research across VA facilities.3 The WH-PBRN is among the first PBRNs in the VA, the largest integrated health care system in the United States.
Integral to the launch of the WH-PBRN were a series of implementation evaluation projects designed to provide timely data relevant to enhancing women veterans' care, while at the same time clarifying how to refine WH-PBRN operational procedures. The inaugural project characterized women veterans' priorities for gender-specific mental health care to inform patient-centered design of women's mental health services; results are presented elsewhere.4 The a priori secondary aim of the inaugural project was to identify PBRN-related takeaways relevant to future multisite WH-PBRN studies. This article describes these lessons learned; some may apply to future VA PBRNs, some may help other PBRNs in their start-up phase, and some may resonate with nontraditional PBRNs that, like ours, sit within complex, integrated health care systems or focus on special populations.
Methods
WH-PBRN Context
The WH-PBRN has a national coordinating center with a director, program manager, coordinator, and other part-time staff (research assistant, data analyst, economist, statistician). During its initial phase, the 4 founder sites were each staffed with a site coordinator reporting to the local WH-PBRN site lead (each a health services researcher and clinician). In collaboration with the WH-PBRN, the study principal investigator and study coordinator oversaw the inaugural project.
Overview of the Inaugural Project's Methods
Figure 1 summarizes the study processes. After regulatory approvals, we used VA administrative data to identify women veterans with recent primary care use at a participating site and sent them 1 bulk mailing (N = 3129) to introduce the study. Women with upcoming clinic appointments who had not opted out after the bulk mailing were approached before (by telephone) or during their clinic visit to invite participation (a total of 687 were contacted). Other than the bulk mailing and introductory call, all study procedures (consent process, survey assessing the need for mental health services, and an interviewer-administered interview where patients ranked mental health care priorities) were conducted by site coordinators in person, in conjunction with scheduled clinic visits (March through November 2012); 515 women participated (75% of those contacted). Site coordinators also reviewed electronic medical records (EMRs).
Study processes (letters reference the study elements in Table 1). IRB, institutional review board.
Synthesizing Lessons Learned
WH-PBRN site coordinators from each local site research team convened weekly with the study coordinator and WH-PBRN program manager to troubleshoot logistic issues solicited from site coordinators; minutes were recorded. At the close of the study, the WH-PBRN program manager reviewed call minutes to identify study “elements” (core study processes); WH-PBRN leadership approved this elements list. Every WH-PBRN site coordinator, collaborating with her local WH-PBRN site lead, then provided written suggestions and feedback for each element regarding lessons relevant to future WH-PBRN studies. After combining the 4 sites' responses for each element, the WH-PBRN coordinating center team re-reviewed the elements (consolidating some) and then re-reviewed responses within each element (consolidating some). Three key themes emerged: challenges, facilitators, and lessons learned. We populated a matrix with key points mapping to each theme for each study element. The core WH-PBRN coordinating center team (AP, DVC, SMF) reviewed and arrived at consensus on representative key points to include.
Results: What Have We Learned?
Study processes (Figure 1) mapped to 7 key study elements: regulatory procedures, building the sampling frame, participant contact, participant tracking systems, logistics of clinic-based recruitment, participant enrollment, and ensuring effective clinic partnerships. Table 1 outlines lessons learned (challenges, facilitators) around each of these themes, summarizing key recommendations.
Regulatory Procedures
Our study benefited from availability of the VA's Central Institutional Review Board (IRB)5 and from a strong working relationship between the WH-PBRN coordinating center and central IRB. The WH-PBRN coordinating center created templates streamlining the creation of regulatory documents across sites, as well as tracking systems to address challenges sites encountered with sequencing of the multiple regulatory steps required in the VA context: IRB (for the overall study and then for each site); local research & development (R&D) committee review at each site (with processes varying across sites); Office of Management and Budget waiver; and data access requests. The centralized tracking system also addressed document version control across sites, reducing the risk of accidental use of outdated study forms.
Building the Sampling Frame
Sitting within a national health care system, our PBRN was able to capitalize on centralized VA databases derived from nationally linked VA EMR data to identify the sampling frame, stratified by visit frequency (to avoid oversampling of high utilizers). A centralized folder permission structure developed by the WH-PBRN coordinating center allowed each site to access their local sampling frame.
National WH-PBRN standardization of study processes streamlined local efforts, but study implementation profited from some flexibility to accommodate differences in local context. Identification of eligible women with an upcoming clinic appointment required the engagement of local nonresearch information technology (IT) staff because the approach to pairing eligible participants with appointment schedules varied by facility. This challenge was lessened by having a stable cohort of local WH-PBRN site leads with long-term local relationships, which will continue to benefit future WH-PBRN studies.
Participant Contact
To address the IRB's prohibition against cold calls, women received an introductory mailing. Because of the site coordinators' observation that primary care appointments at these facilities were often scheduled within days of the patient's appointment request, the study team recognized that it would not be feasible to send a mailing to women with upcoming appointments, wait for the required opt-out period to pass, and then call to introduce the study. Instead, we sent a bulk mailing to all eligible women, informing them that they might be contacted about the study. Since site coordinators called women shortly before a clinic appointment, there was a time gap between the mailing and telephone recruitment, impairing some participants' recall of the mailing. Rolling mailings may have helped.
Women reached by phone often had limited time to speak; site coordinators found that a flexible phone script helped to accommodate women's time constraints. This included various scenarios that could arise during a telephone interview and prompts to guide the site coordinator through the script's logic pattern, depending on real-time patient responses. Supplemental information was available when desired, but it was not imposed on women simply wanting core facts. Site coordinators were formally trained around diverse scenarios that could arise during telephone contacts.
For women who were unreachable by telephone, site coordinators found that approaching women in clinic, while labor-intensive, often led to successful recruitment. A contributor to the site coordinators' ability to establish rapport may have been that the majority of women were recruited from women's health clinic settings, characterized by a team approach to care and an emphasis on establishing a safe and welcoming environment for women veterans.
Throughout the study, and especially during participant recruitment, trainings provided to site coordinators by the WH-PBRN coordinating center (including trainings about working with a special population with unique military experiences and high rates of prior trauma), combined with study-specific protocol materials provided by the study principal investigator, ensured cross-site fidelity in study procedures. Investing in the training of local site WH-PBRN staff is expected to pay off in future studies, which will increasingly profit from common operating procedures across sites and local research staff with expertise around recruitment of women veterans.
Participant Tracking Systems
The WH-PBRN coordinating center developed participant tracking systems in Microsoft Access (Microsoft Corp., Redmond, WA). Each site accessed its site-specific database behind the VA firewall on a secure, limited-access server. The tracking system template addresses multiple potential study needs (managing participant contacts, calculating response rates, responding to regulatory audits) and will serve as a resource for future WH-PBRN studies. Overcoming IT challenges associated with multiple users accessing the same database from a central server and establishing database backup schedules (with naming conventions to distinguish backups from “live” files) were considered critical.
Logistics of Clinic-Based Recruitment
The clinic-based WH-PBRN milieu allowed us to approach eligible participants regarding their care preferences while they were actively embedded in their usual care environment. Despite strong relationships with clinic staff, site coordinators encountered complexities when working in dynamic clinic settings, such as needing to become familiar with clinic rhythm and layout so as to anticipate shifts in clinic schedule flow and interruptions to room availability. Since women veterans are a numeric minority group in the VA, a small number of primary care providers see most women; therefore, provider absences from the clinic disproportionately affected recruitment. With strong links to the women's clinic team, site coordinators were able to anticipate such scheduling issues and plan accordingly.
Based in large VA medical centers, at some sites the clinic and research offices were in different cities, making it important for the site coordinator to have a mechanism for maintaining contact with the site lead. Locked bags were identified as a solution for the secure transport of sensitive paper documents between locations.
Research on a special population like women veterans requires recruitment from a limited pool. Accordingly, some sites encountered recruitment overlaps with other ongoing studies. Being based in the WH-PBRN has the advantage that the site lead is a stable position: the site lead can maintain contact with facility research leadership and women's clinic leadership regarding potential recruitment conflicts. Research participant “burnout” could also become an issue when recruiting from a small pool of eligible participants. The WH-PBRN site lead can work with the clinical team to stage studies appropriately, accepting only studies that the clinic has the capacity to support.
Participant Enrollment
Even after securing clinic space and finding willing participants, further hurdles to participation arose. The legal language in Health Insurance Portability and Accountability Act (HIPAA) forms seemed to be threatening or overwhelming to some women. Using the Flesch-Kincaid grade level readability statistics function within Microsoft Word, we attempted to simplify HIPAA language to an eighth-grade reading level, but because of required language we could reach only a grade level of 10.7. Some women were concerned about study access to their EMRs; future studies might consider allowing “opt-outs” from EMR review, weighing the benefits of chart reviews against the adverse effects on recruitment. This could be particularly important for studies engaging women veterans, given the high prevalence of trauma histories among this population.6
In addition, we did not have an option to provide childcare services, transportation, or financial incentives to participants, and at the time of this study the clinics did not typically offer night or weekend clinic appointments to accommodate women's scheduling constraints. Despite such logistic challenges, credit for successful recruitment goes to the women veterans: Site coordinators across all sites remarked on their perception that women were motivated to participate by a strong desire to help fellow women veterans. This may have contributed to the study's high (75%) participation rate.4
Ensuring Effective Clinic Partnerships
Site leads at the 4 sites were selected for their passion and commitment to women veterans' health care and because, as clinicians themselves, they had existing relationships with women's health clinical leaders at their sites. Site leads garnered leadership support to endorse women's health research to clinic staff as a facility priority, and they personally introduced site coordinators to clinic staff. Indeed, at several sites the site lead and/or the site coordinator attended clinic team meetings on an ongoing basis, as part of the WH-PBRN′s philosophy regarding maintenance of a long-term research-clinical partnership.3 The research team and clinic team carefully considered together how to avoid having research interviews interrupt clinic flow. Clinic staff at all sites generously provided tremendous assistance with interview space, scheduling issues, and notifying site coordinators when eligible women had checked in. A key characteristic of PBRNs is that investigators do not “helicopter” in to clinics, recruit participants, and leave; instead, the local PBRN staff develop longstanding relationships with local clinicians and clinical leaders7 that support ongoing collaborative efforts. It is essential for the local PBRN team to acknowledge the added demands that research imposes on clinic staff, who are at the heart of any successful PBRN activity.
Discussion
While many complexities of conducting multisite, clinic-based research in an integrated health care system with a special population (women veterans) had to be addressed, our experience with the inaugural WH-PBRN study demonstrated the feasibility of establishing productive relationships with local clinicians and recruiting women veterans in diverse sites, and it illustrated strengths of a PBRN approach. We identified 7 critical study elements and described barriers to and facilitators of conducting multisite research within the newly formed WH-PBRN.
Multisite Research
While conducting research across multiple sites, we encountered challenges consistent with the experience of other PBRNs. The need to coordinate across diverse IRBs can impede PBRN research8,9; shared or ceded reliance models of IRB review, like IRBShare and centralized IRBs, help to overcome this challenge.10,11 Access to the VA's centralized IRB streamlined our multisite regulatory approval processes and ensured study-wide protocol consistency. Following central IRB approval, VA studies must still obtain approval from the local R&D committee at each site; for this reason, a VA-based PBRN might still experience study start-up delays resembling those seen in other PBRNs12 as a result of cross-site variability in local R&D committee forms, procedures, and meeting dates.
In addition to such regulatory complexities, other challenges of conducting research across multiple facilities included training geographically dispersed study staff, ensuring universal consistency in data entry, and addressing site-level IT idiosyncrasies, such as systems for accessing local clinic schedules. Our experience, mirroring that of others,13⇓⇓–16 was that central standardization (eg, tracking databases, backup conventions, training protocols) via a PBRN coordinating center attenuates these challenges. As has been observed elsewhere,17 a centralized EMR facilitated our multisite work, as did the ease of data-sharing across a national secure network.
Ongoing input on this and other WH-PBRN studies from the VA's Cooperative Studies Program, with its well-established procedures for multisite research,18 has also greatly enhanced our PBRN′s effectiveness. VA funding of our central infrastructure—a known key to sustaining networks19,20—likewise facilitates our ability to support multisite research.
Health Care System–Based PBRNs
PBRNs were originally composed of private practices (eg, ASPN, PROS),21 but over time other models have emerged, such as PBRNs embedded within an academic institution and its surrounding counties (eg, Primary Care Research Consortium,22 SPUR-Net23) or within health care systems (eg, HMORN17). Special challenges arise in systems-based research, including the need to navigate across the various layers of departments and disciplines and engage stakeholders at multiple levels in the organization.24 As the largest integrated health care system in the United States, the VA has a particularly complex structure. This includes organizational levels (national, regional, and local leadership); settings (medical centers and community-based outpatient clinics); units (primary care, specialty outpatient, inpatient, rehabilitative); and disciplines (nursing, medicine, pharmacy, and others). While this complexity introduces challenges for a VA-based PBRN and necessitates extra effort to ensure multilevel stakeholder engagement, it also introduces opportunities to partner with frontline providers and with leaders poised to translate research findings into local practice and national policy.18,25⇓⇓–28 Future VA-based PBRNs can profit from this context and from the experience of the WH-PBRN3 and other VA-based networks (such as the VA Office of Dentistry's VADER-PBRN, VA Mental Health PBRN,29 Heart Failure Provider Network,30 and the National Center for Posttraumatic Stress Disorder's Practice-Based Implementation Network31).
Special Populations
Many PBRNs address special populations.32 While a number identify women's health as a research interest, to our knowledge no other PBRNs focus exclusively on women.33 Women who served in the military have distinct demographic and clinical profiles1 and unique occupational exposures, potentially including combat34 or military sexual trauma,6 and many receive their VA care in women's health clinics.35 As in our study, such characteristics can have implications for recruitment (eg, embedding the site coordinator in the women's health clinics that partner with the WH-PBRN promoted recruitment) and study procedures (eg, women's hesitancy around HIPAA form completion could sometimes have been due to trust issues related to prior trauma). Strong support of the WH-PBRN from top VA leadership,36⇓–38 combined with the WH-PBRN′s close partnership with the highly engaged frontline clinicians who care for women veterans, were central to the study's success. Our experience demonstrates that a PBRN can be an effective vehicle for amplifying the voices of a minority population in research.
Conclusions
Many new PBRNs may encounter some of the experiences we did.19,39 This inaugural WH-PBRN study supports the feasibility of conducting meaningful PBRN research in a complex, integrated health care system like the VA and lays the groundwork for future VA PBRNs. The Blueprint for Excellence,40 the VA's recently published strategic plan, prioritizes research on women veterans and proposes to capitalize on innovations obtained through the WH-PBRN, making this study's “lessons from the field” particularly timely.
Acknowledgments
The authors thank the clinical staff at the participating VA Women's Health PBRN sites, including the women veteran program managers (Kara Carter, Joan Galbraith, Linda Kleinsasser, and Callie Wight); the women's health clinic directors, program associates, physicians, psychologists, nurse practitioners, nurses, and clerks; along with facility leaders at the sites, for their support and assistance during the conduct of the study. We also appreciate additional core research team members (Ciaran Phibbs, Joanne Pavao, Julia Lin, Aileen Baylosis, Andres Busette, Jennifer Gierisch, Eric Berg, Pon Su, Natalia Llarena, and Yasmin Romodan) and additional interviewers (Nikki Hozack, Casey Latting, Lesley Rohrer, Fay Saechao, Ioana Schmidt, and Hollis Weidenbacher), without whose input this inaugural PBRN study could not have been successful. We are also grateful for the guidance we received throughout the study from the PBRN Advisory Board (Rowena Dolor and Michael Parchman), and from the VA Women's Health Research Network's national Steering Committee. We express our gratitude to the women veterans who have served our country, to whom the WH-PBRN is dedicated.
Notes
This article was externally peer reviewed.
Funding: This material is supported by the Department of Veterans Affairs, Health Services Research & Development (HSR&D) services grant SDR 10–012. EMY received support from VA HSR&D Senior Research Career Development Award no. 05–195.
Conflict of interest: none declared.
Disclaimer: The views expressed are those of the authors and do not necessarily reflect the position or policy of the Department of Veterans Affairs or the US government.
- Received for publication January 15, 2015.
- Revision received May 27, 2015.
- Accepted for publication June 8, 2015.
