INTRODUCTION

Patients and other key healthcare stakeholders are increasingly becoming engaged in the planning and conduct of biomedical research.1,2 Patients and stakeholders can be engaged across stages of research including identifying study topics, choosing hypotheses, analyzing data, and disseminating findings35 (see Table 1 for illustrative examples). Levels of engagement range from consultation, to collaboration in bi-directional partnerships with researchers, to stakeholder-directed projects.6 Participatory research approaches have been increasingly recognized as potentially beneficial in several countries,710 and a robust community-based participatory research literature documents key issues in including communities in research.11 In the US, growing support for engaged health research is demonstrated by patient engagement programs at several federal health agencies.1215 In 2010, the Patient-Centered Outcomes Research Institute (PCORI) was established to fund comparative clinical effectiveness research (CER) to assist patients, clinicians, and other healthcare stakeholders in making informed health decisions.16 PCORI requires research engagement in all its funded CER studies.17

Table 1. Illustrative Examples from the Peer-Reviewed Literature of Patient and Stakeholder Activity by Research Stage

Despite increased attention to engagement, descriptive information about engagement in health research is limited. Some reviews describe engagement in published studies and report impacts of engagement,1,3,1821 but the sparse detail on how engagement is implemented,22,23 and the lack of systematic characterization and evaluation of engagement5,24,25 highlight the need for additional approaches for obtaining information on engagement. Some earlier studies describe engagement in research in the U.K.,26,27 but no systematic measures exist to characterize or evaluate engagement. The literature provides insufficient information to permit identification of best practices and guidelines for replication by others. Researchers have acknowledged the need for a better understanding of optimal approaches to engagement, and the opportunity to learn more about engagement from PCORI’s funded projects.24 This paper presents our work to collect information about engagement of patients and other stakeholders in PCORI’s first awardees, the 50 PCORI Pilot Projects.

MATERIALS AND METHODS

Sample and Data Collection

In 2012, PCORI awarded contracts totaling $31 million for 50 PCORI Pilot Projects (http://www.pcori.org/assets/PCORI-Pilot-Projects-Funding-Announcement-Amendment-1-_v2_-09302011.pdf). The intent of this funding was to advance methods for patient-centered outcomes research. Unlike subsequent PCORI funding, these first projects were intended to have an explicit focus on methods (analytic methods, outcome measures, or other specific aspects of research methods). Projects were funded for a maximum of 2 years and $250,000 in direct costs. Engagement of patients and other stakeholders was one criterion by which PCORI evaluated the proposals for funding (as well as significance, investigators, innovation, approach, and environment). PCORI required engagement unless the applicant sufficiently explained why engagement was not pursued.

AcademyHealth, a national health services and policy research organization, was competitively selected to support active management of the PCORI Pilot Projects. PCORI and AcademyHealth collected data regarding engagement of patients and other stakeholders in the design, conduct, and dissemination of the projects (response was voluntary). MaGil IRB provided Institutional Review Board (IRB) approval for secondary analysis of these data.

All 50 PCORI Pilot Project principal investigators (PIs) or their designees were invited to complete an online questionnaire28 in July 2013 (approximately 6 to 8 months following project initiation). We sent two reminders 3 and 6 weeks later and a final personalized reminder in early November.

Measures

The instrument (Appendix A, online) was developed by the PCORI and AcademyHealth team based on PCORI’s conceptual model for patient-centered outcomes research,29 key informant interviews with five Pilot Project PIs, qualitative analysis of Pilot Project research proposals, discussion with the PCORI Patient Engagement staff, and literature review, as noted below. In addition to capturing descriptive elements (who, when, and extent of engagement), the instrument content followed the conceptual model categories of ways to partner, communication, capture and use of perspectives, and perceived influence of partners. To inform future practice, the instrument also addressed challenges to and facilitators of engagement.

Stakeholders Engaged

Respondents reported whether they had engaged patients or stakeholders in ways other than as research subjects, the types of stakeholders engaged in their project, and the number of individuals engaged.

Level of Engagement

Respondents classified the extent to which each stakeholder type was engaged: 1) Consultation: researchers use stakeholder views to influence decision-making regarding research. Consultation allows researchers to obtain stakeholder views without necessarily being committed to act on them; 2) Collaboration: stakeholders have an ongoing partnership with researchers and greater ownership of the project; or 3) Stakeholder-led: stakeholders initiate, design, and undertake the research process.6

Nature of Relationships

Respondents reported on the relationship duration (classified as < 1 year, 1 to 3 years, > 3 years); this provided insight into whether the relationship was pre-existing or newly established.

Stages of the Research Project

Respondents indicated the stages of research in which each stakeholder type was engaged.3,5

Facilitators of and Challenges for Engagement

Respondents classified a list of possible engagement facilitators on a four-point Likert scale (“Not at all important” to “Critically important”) or “n/a” if not used by their team. Respondents identified challenges to engagement experienced by their team from a list and rated the extent to which challenges were resolved on a three-point Likert scale (“Not at all” to “Completely”). We developed these items based on a review of the literature5,3036 and knowledge of the investigators’ experiences obtained through interviews and facilitated group discussions with investigators.

Evaluation of Stakeholder Influence

Respondents indicated (yes/no) whether they planned to evaluate the influence of stakeholders on the research project.

Open-Ended Questions

Respondents answered several open-ended questions about how relationships with stakeholders were established, challenges to engagement, contributions of patients and other stakeholders, and initial learnings about engagement.

Analysis

Descriptive statistical analyses (including frequencies, proportions) were performed using the survey platform.28 The majority of items were created as closed-ended Likert-scaled ratings; we examined percent of sample reporting each response category and interpreted based on response scale. We analyzed responses from open-ended questions (motivations for engagement, establishment of relationships, challenges, lessons learned, and contributions of stakeholder partners) using a descriptive qualitative approach, with each question treated as its own deductive code. Responses to each question were grouped by sub-codes representing themes identified inductively by one member of the team (LEE), and reviewed by another member (LPF). Disagreements were resolved through consensus.

RESULTS

Researchers from 47 of the 50 projects completed questionnaires (94 %; 33 PIs and 13 other designated research staff). These projects were geographically distributed (Northeast = 30 %, South = 27 %, Midwest = 12 %, West = 30 %). Almost half (46 %) of the projects focused on four health care decision supports; while 30 % focused on research prioritization and analytic methods, 15 % were about developing outcomes instruments, and 13 % related to developing technology and infrastructure for patient-centered outcomes research.

Engagement of Patients and Other Stakeholders

Most respondents (83 %, N = 39) reported engaging patients or other stakeholders in their project in ways other than as research subjects. Among those, investigators most commonly reported engaging patients/consumers (90 %), clinicians (87 %), clinic/hospital or health system representatives (44 %), caregivers/family members (41 %), or patient or caregiver advocacy organizations (41 %) (Fig. 1). Most projects (84 %) engaged at least two types of stakeholder; 22 % reported engaging four or five types of stakeholders.

Figure 1.
figure 1

Types of stakeholders engaged in the PCORI pilot projects (among those projects who reported any engagement, n = 39).

The primary motivation for engaging patients, their caregivers, and advocacy organizations was to understand their values and perspectives on items such as the impact of a disease, the most appropriate interventions, the importance of specific study outcomes, and research products that meet patients’ needs. The primary motivations for engaging clinicians included their “clinical and methodological expertise” and “insights” into care delivery, obtaining clinician “buy-in,” and ensuring that the research outcomes were clinically meaningful and translatable.

A notable proportion of relationships between researchers and stakeholders existed for < 1 year (37 % of patients, 36 % of caregivers/family members, 19 % of clinicians, 33 % of patient or caregiver advocacy organizations, 27 % of health system representatives). However, some researcher-stakeholder relationships were more long-standing (> 3 years), particularly relationships between researchers and clinicians (50 % of clinicians, 36 % of health system representatives, 27 % of caregivers/family members, 23 % of patients, 22 % of patient or caregiver advocacy organizations). Researchers reported establishing new relationships with patient stakeholders through traditional recruitment in the clinical care setting (e.g., staff outreach, mailings, and flyers); clinician referrals; community outreach; or utilizing investigators’ existing networks with colleagues, patients, and advocacy organizations.

Regarding engagement of patients specifically, 15 % of projects reported engaging one patient, while 56 % of projects reported engaging ≥ six patients. Most projects reported engaging patients as consultants (35 %) or collaborators (53 %) rather than as co-leaders (6 %). At the time of assessment, patients had engaged most commonly at the stages of research question development, proposal development, study design, data collection, topic solicitation, and results review (Fig. 2). Respondents engaging other stakeholder types gave similar responses regarding the level of engagement and research stages (Appendix B, online).

Figure 2.
figure 2

Stages of the research project in which patients were engaged (among PCORI pilot project investigators reporting engagement of patients, n = 34).

Challenges to and Facilitators for Engagement

The most commonly reported challenges to engagement were lack of stakeholder time (46 %), lack of research team time (35 %), lack of stakeholder training/background (30 %), difficulty in finding the appropriate representatives to engage (27 %), lack of research team resources (24 %), and lack of research team training/background (22 %). Respondents generally described these challenges as “partially resolved,” and shared some tangible strategies (Table 2). Respondents also reported establishing trust and learning how to work together as challenges. Some solutions to these challenges included allowing ample opportunity for stakeholders to ask questions, communicating through multiple channels, having face-to-face meetings, making time to socialize, and training researchers on engagement.

Table 2. Challenges to Engaging Patients and Other Stakeholders in Research Reported by PCORI Pilot Projects Investigators (N = 37)

Respondents most commonly endorsed communication strategies and shared leadership approaches as “critically important” facilitators (49 and 41 %, respectively). Examples of communication strategies included: trying different arrangements (big group, small group, individual) to see what works best; maintaining regular contact with stakeholders; interacting through multiple channels (online, teleconference, in-person); using non-technical language; and holding pre-meetings with stakeholders to go over items to be discussed in larger meetings. Examples of shared leadership strategies included: adopting a perspective of shared responsibility for decision-making, training of research team members, and involving research partners in research team meetings. Respondents also endorsed remuneration, training of stakeholders, and training of researchers as “important” (41 %, 38 %, and 30 %, respectively) (Fig. 3).

Figure 3.
figure 3

Facilitators of engaging patients and other stakeholders in research (among PCORI pilot projects reporting any engagement, n = 36).

Early Lessons Learned about Engagement

Early lessons learned (Table 3) suggest the benefits of establishing meaningful partnerships with stakeholders rather than engaging stakeholders merely to fulfill a requirement. Researchers reported the importance of selecting stakeholders strategically to fit project needs, continuously involving stakeholders, adapting to the practical needs of stakeholders, and clearly defining roles and expectations for research partners. Respondents also emphasized the importance of in-person contact for relationship building.

Table 3. Summary of Main Themes Identified from Responses to Select Open-Ended Questions

Contributions of Patients and Other Stakeholders

Twenty-one projects (57 %) planned to assess the influence of stakeholders on the research process. The themes that emerged regarding early tangible contributions to the research projects from patients and stakeholders included changes to the project outcomes or goals, changes to the project methods, enhanced access to study populations or study settings, modifications to the study interventions, and refinement of the study instruments or interview questions. Although still in the early stages, several projects identified contributions of patients and stakeholders to results interpretation and dissemination of findings (Table 3).

DISCUSSION

To address the need for more information about engagement in research, we created and fielded a data collection tool to obtain information about stakeholder engagement practices and strategies employed in PCORI’s Pilot Projects. In the first 6 to 12 months of these projects, researchers most commonly engaged patients, and the majority of projects engaged multiple types of stakeholders. Compared to the published literature, which reflects lesser engagement of other types of stakeholders,1,18 many PCORI Pilot Projects reported engaging clinicians (> 85 %), and a substantial number of projects reported engaging with caregivers, advocacy organizations, and health system representatives (>40 % for each). However, similar to previous studies, few investigators reported engaging purchasers, payers, policy makers, and the life sciences industry. This pattern may reflect the basic methods research focus of the funding; the stakeholders engaged by these awardees are those most proximal to the methods questions addressed. Subsequent PCORI funding that has a focus on comparative effectiveness research rather than methods research is expected to expand the universe of stakeholders engaged. Our findings highlight the need for a better understanding of how the views from multiple types of stakeholders are incorporated into a study and of the impact of relationship length and quality on the research process. While engagement was most frequently reported in developing research questions, consistent with other reports,1,37 the extent of engagement in data collection, data analysis, interpretation of results, and dissemination is noteworthy. Engagement in these later phases is expected to increase as projects progress. While the challenges, facilitators, and lessons learned reported here are generally consistent with the existing literature,3,5,19,20 , 3840 this study is unique in capturing this information systematically and assessing the relative importance across a group of projects. Our findings collectively highlight the effort, attention, resources, and flexibility required for concerted engagement. Across these survey responses, for example, respondents emphasized meaningful and continuous partnerships, shared leadership strategies, communication through multiple strategies that fit specific stakeholder partners, and adaptation to the practical needs of stakeholders. Similarly, others highlighted the importance of sharing power for successful research engagement.41 Lack of time—both for stakeholders and researchers—was the most frequently noted challenge and was identified as a top barrier to engagement in a recent survey of patients and clinicians.39 Thus, it is critical to recognize resource needs and develop strategies that maximize stakeholder input while minimizing the time burden, and the responses provide support for elements in the conceptual model that helped to inform these items.29

Most challenges experienced were only partially resolved. Challenges to engagement remain to be addressed in terms of individual research team process as well as through institutional and funding agency infrastructure. Some issues have tangible solutions (e.g., financial support of partnership in research budgets, matching researchers with stakeholder partners); others require change in social norms (e.g., developing meaningful partnerships). PCORI supports several efforts to address these issues, including funding opportunities for developing research partnerships (http://www.pcori.org/funding-opportunities/pipeline-to-proposal-awards/) and for developing training and building capacity for engagement (http://www.pcori.org/funding-opportunities/eugene-washington-pcori-engagement-awards/).

This study also demonstrates the contributions that engaged stakeholders had already made in the first year of the projects. Researchers reported that patient and stakeholder input resulted in changes to study outcomes, goals, methods, interventions, and materials. These contributions may enhance the appropriateness of study materials,19,21 and the relevance of the research questions and outcomes1,3,18,21,22,4244 to the study population. Some projects reported stakeholder contributions to the interpretation of qualitative data, which may help researchers to contextualize findings for specific audiences.3,21 Increased access to study populations or settings suggests that engagement may mitigate commonly encountered challenges with recruitment and retention.3,18,22,43 These results are consistent with available evidence on positive impacts of engaged research on research process and dissemination of results.1,6 , 18,21,22,4245 More research is needed to better understand how engagement impacts research quality, relevance, and dissemination.

While the findings represent significant progress towards systematically understanding engagement in research, several limitations should be considered. We obtained only researchers’ perspectives, which may differ from the views of other stakeholders engaged.26 This tool did not assess principles of engaged research (e.g., trust, respect) that have been identified as important to the success of engagement. While others have noted important negative impacts of engagement,21 the current tool did not specifically ask about these effects and thus did not provide new insights on drawbacks. Obtaining information at later stages of research projects will permit evaluation of engagement outcomes.

Additional information about construct validity of the measure is needed. Results obtained here should be compared to other forms of project reporting, including open-ended project status updates required of awardees and conversations between funding agency staff and awardees. Future data collection efforts should include appropriately timed repetition of key items. Additionally, further cognitive testing of items is warranted to ensure item intent is understood by respondents and that responses reflect respondent intent.

The generalizability of these findings may be limited because respondents were investigators on methods projects rather than health outcomes or CER. Because these respondents were the first PCORI awardees, they also may have more interest in and experience with engagement than the broader health research community. Thus, their knowledge and attitudes about engagement may be more positive. Finally, these self-report data may overestimate research engagement, particularly given the role of the research funder in data collection. Many of these limitations will be addressed in future data collection efforts with PCORI awardees.

Engagement in research is becoming more widespread as the value of end-user needs and perspectives is recognized, and as funding initiatives requiring or encouraging engagement proliferate. Since the PCORI Pilot Projects were funded, researchers have become more familiar with the concept of research engagement. PCORI has provided additional guidance to applicants regarding its funding requirements related to engagement in research and is continuing to collect information on engagement from PIs and their patient and stakeholder partners to inform the literature and PCORI’s policies regarding research engagement.

The evidence on engagement will grow if researchers systematically document more information about engagement and its impact on individual projects in publications,1,23 if relevant medical subject headings (MESH) indexing terms for patient engagement are developed,3 and if information on engagement is systematically collected from researchers and research partners. PCORI continues to collect and use learnings from engagement in funded projects. Future research should evaluate the impact of engagement at each stage of the research process, including the effects of engagement in interpretation of results and dissemination on the clinical uptake of findings. Studies should consider both positive impacts and potential undesirable consequences of engagement, and strategies for mitigating unintended consequences. Furthermore, future evaluations should identify which strategies for engagement—which stakeholders to involve, how to involve them, and at what stages of process—have the greatest positive impact on research quality and usefulness to patients and their caregivers and other end users of the research. Ultimately, evaluations demonstrating links between engagement and longer-term outcomes such as health decision making and health outcomes29 are needed to develop evidence-based guidance about the best approaches to engagement in research.