Abstract
Background: Quality improvement collaboratives (QICs) are used extensively to promote quality improvement in health care. Evidence of their effectiveness is limited, prompting calls to “open up the black box” to better understand how and why such collaboratives work.
Methods: We selected a cohort of 5 primary care practices that participated in a 6-month intervention study aimed at improving colorectal cancer screening rates. Using an immersion/crystallization technique, we analyzed qualitative data that included audio recordings and field notes of QICs and practice-based team meetings.
Results: Three themes emerged from our analysis: (1) practice staff became empowered through and drew on the QICs to advance change efforts in the face of leader/physician resistance; (2) a mix of content and media in the QIC program was important for reaching all participants; (3) resources offered at the QIC did little to spur practice change efforts.
Conclusion: QICs offer a potentially powerful way of disseminating health care innovations through enhanced strategies for learning and change. Creating collaborative environments in which diverse participants learn, listen, reflect, and share together can enable them to take back to their own organizations key messages and change strategies that benefit them the most.
Quality improvement collaboratives (QICs) are used extensively to promote quality improvement (QI) in health care.1–7 Although different QIC models have been developed in various countries,6 a common framework includes multidisciplinary teams from multiple health care departments or organizations coming together periodically to learn about ways to improve their provision of care.8 Additional features typically include facilitators, content experts, and project management coaches who guide the implementation and change process and provide the group with periodic instruction.9 QICs have attempted to overcome the notoriously slow dissemination of medical and health care innovations through enhanced strategies for learning and change.4,10 Although there have been some QIC success stories,11–13 the popularity of QICs seems to be based largely on the assumption that teams are likely to be more effective in generating and implementing improvement ideas when working together rather than in isolation.10 In fact, evidence of QIC effectiveness is limited, and their effects cannot be predicted with certainty.8 This has prompted calls to open the “black box” of QICs,6,9–10 that is, to focus on understanding how and why QICs work—the determinants of their effectiveness—rather than simply measuring whether they work or not.
Several recent articles have begun to answer this call.6,14,15 Vos et al15 explored the QIC black box by assessing the applicability of the QIC method for process redesign based on 18 project teams from 8 different hospitals. They found that the QIC method did not empower project teams to implement process redesign in a short time frame. Nevertheless, they did find that it was important to customize redesign solutions for different problems, which ran counter to their use of standardized QIC methods. Also, they did not find evidence to support their hypothesis that using QICs to focus on one central idea of change would lower resistance to change. Instead, they found that departmental interdependencies created resistance to change requiring additional consultations and wider organizational buy-in. Finally, they found that project teams did not benefit from peer stimulus because participants perceived few similarities between projects and rarely shared their experiences.
In another study based on self-reported questionnaires of team leaders from 24 Dutch hospitals who participated in QICs, Duckers et al14 found that support from organizational and external change agents (ie, outside QI facilitators) had a positive influence on the number of changes adopted by QIC teams during implementation. The authors urged more research explaining how the support of an external change agent influences team organization, how team learning takes place within a QIC, and how QICs contribute to organizational learning.
Although these studies shed light into the QIC black box, significant questions remain about understanding how ideas and plans generated during a QIC are translated into actual QI changes in the participating organizations. In this article we present a qualitative analysis of primary care practices that participated in a QI intervention to further explore this black box. The intervention design called for each practice to form their own QI team comprising clinicians and staff who met regularly with an external facilitator. This team was the driving force for generating and implementing ideas to improve the practice's colorectal cancer screening rates. In addition, 2 QICs were held over the course of the intervention where representatives from each practice gathered for day-long programs of didactic presentations and shared learning. The intent was to use the QICs to enhance idea generation, problem solving, resource sharing, and motivation among the practice representatives who would then bring ideas, information, and resources back to their own QI team.
We used audio recordings of and field notes from the QICs and local QI team meetings to explore the flow of ideas, information, and resources between QIC leaders, QIC participants, and the practices. Our central research question was, How do QICs shape local organizational QI change efforts? Several targeted questions drove the analysis: (1) How did QIC participants engage with the discussions, ideas, and resources they encountered in the QICs? (2) How did QIC participants then draw on what they encountered at the QICs during their local QI team meetings? and (3) What conversations and activities took place during each practice's QI implementation process?
Methods
Project SCOPE
This analysis is based on a subset of primary care practices that participated in a federally funded QI trial called SCOPE (Supporting Colorectal Cancer Outcomes through Participatory Enhancements).16–18 Beginning in 2005, practices were randomized into either an initial intervention arm (n = 13) or delayed intervention arm (n = 12). The primary aim was to test the effectiveness of a facilitated, team-based approach to improve colorectal cancer (CRC) screening rates in primary care practices.
SCOPE used an organizational change model that incorporated an initial practice assessment, facilitated QI team meetings, and 2 QICs over approximately 6 months (see Figure 1). For the initial practice assessment (called the “multimethod assessment process”),19 study facilitators used ethnographic techniques (eg, observations and interviews) to understand the initial conditions of each practice (eg, aspects of leadership, communication, work relationships, patient population) and to identify current cancer screening processes.
For the facilitated QI team meetings (called the “reflective adaptive process” [RAP]),20 facilitators met with a team of 4 to 9 members at each practice for up to 10 weekly, 1-hour meetings (see Table 1). RAP teams comprised clinicians and support staff; these meetings were held at each of the primary care offices at a time that was convenient for the practice members.
Facilitators followed a loosely structured QI plan designed to help each team better understand their current cancer screening practices, brainstorm cancer screening improvement ideas, reach consensus on a plan, and implement and monitor this plan.
Two SCOPE QICs were held during the 6-month intervention period. Each QIC took place on a Saturday at the Cancer Institute of New Jersey and lasted approximately 6 hours. Two practice representatives (including one clinician) from each practice were expected to attend, although in some cases more than 2 attended. Each QIC brought together a relatively small gathering of approximately 14 to 16 practice representatives in addition to project researchers and guest speakers. The QIC program consisted of cancer-specific presentations and discussions of QI change strategies, such as how to generate staff buy-in and sustain motivation to engage in team-based QI efforts. After each session, facilitators prompted participants to reflect together on how they could utilize this information in their own practices. Time also was allotted for participants to share ideas with their peers and discuss their practice's accomplishments and barriers to improving cancer screening rates.
The University of Medicine and Dentistry of New Jersey—Robert Wood Johnson Medical School Institutional Review Board approved this study. Medical directors and/or lead physicians from each practice gave informed consent, as did participating staff members. All names have been changed to protect confidentiality.
Selection of Practices for Analysis
We chose to conduct an in-depth analysis on a subset of SCOPE practices. Within the intervention arm, practices were grouped into 2 “waves” (or groups) of 5 to 6 practices each. We randomly selected wave 2, the second group of the intervention arm, for intensive analysis. Initially, 7 practices were randomized into this wave, but 2 failed to complete the intervention requirements, leaving 5 practices available for analysis. Differences between the practices in our sample and SCOPE practices as a whole are minimal in terms of the average number of physicians (our sample, n = 4; remaining SCOPE practices, n = 3) and type of practice (most were single-specialty family medicine). Our sample included a larger number of staff (average, 22) compared with the remaining practices (average, 14), which was due to a large residency-affiliated practice (n = 66) in our sample.
Data Collection
Two types of data informed this analysis: audio recordings and field notes. The study facilitator used a digital audio-recorder to record every RAP meeting held at each practice site. The study facilitator also wrote field notes of each RAP meeting to capture elements that were unavailable from the audio recordings (eg, body language, facial expressions, group dynamics). Multiple audio-recorders were used at both QICs to ensure that all dialogue was adequately recorded. In addition, multiple study team members wrote field notes during the QICs to capture dialogue and elements that were unavailable from the audio recordings. All field notes were de-identified and stored as text documents.
Data Analysis
Three authors (EKS, SMC and JH) performed the primary data analyses using an immersion/crystallization technique.21 This consisted of cycles of reading and listening to data followed by discussion/reflection, repeated until reportable interpretations were reached. Because audio recordings were not transcribed, the authors listened to the QICs and targeted RAP meetings together. Remaining audio recordings and field notes from QICs and RAP meetings were then read/listened to independently by the authors, who wrote descriptive summaries that enriched the group analytic process. Regular analytic meetings allowed the authors to pose emerging questions, cross-check each other's interpretations, and identify patterns and themes related to our research questions. In total, approximately 37 hours of audio recordings and 200 pages of field notes were analyzed.
Results
Overall, RAP teams within each practice averaged 6 people per team, and each practice held an average of 7 RAP meetings (see Table 1). Two practices (practices 4 and 5) failed to provide physician representation at the first QIC. All practices sent at least two practice representatives to both QICs. Four practices sent more than the required 2 representatives to one or both QICs. Because each team implemented practice-specific plans to improve CRC screening, there was some variation of QI objectives/plans across teams. Improving documentation and enhancing CRC education were the most common strategies chosen.
SCOPE QIC Context
In this section we highlight several presentations and discussions that took place during the first QIC. This provides context for exploring, in the next section, how elements of the QIC were then translated into each practice's QI efforts. SCOPE QIC agendas are shown in the Appendix.
The first QIC began with members of the research team enacting a humorous skit depicting a physician enthusiastically returning from a conference with ideas for practice improvement. In the skit, the physician encountered skepticism and resistance from the staff. Following the skit, study facilitators led a group discussion about taking information back to their practice and generating buy-in and motivation. Practice representatives responded positively to the skit and engaged in conversations about similar experiences of their own.
This led into a presentation about “effecting change in the primary care setting.” During this session, the physician from practice 1 shared a personal experience about implementing changes in his practice:
“… I didn't tell the front desk about [a new idea for the practice] and I didn't really sell it to anyone else…so it didn't happen because I [can] see, now, like 20 different process things that I didn't bother doing, like getting anyone else involved, which I'm guilty of.”
In group discussions such as this, participants were encouraged to reflect on what worked or did not work in their previous change efforts and why.
Another presentation, led by a local CRC surgeon, focused on CRC screening. Key points included (1) the low clinical value of single, in-office fecal occult blood tests (FOBTs) for screening; (2) colonoscopy as a CRC prevention tool; and (3) reasons why patients do not get screened (eg, “My doctor never talked to me about it”). During the group discussion, all practice representatives made at least one comment and many offered positive remarks about the new information they learned. Hearing this speaker's emphasis on colonoscopy as the best screening option, some practice representatives raised concerns about costs and the availability of colorectal surgeons.
The QIC program also included a cancer survivor who shared her life story. She explained that although her father had passed away from prostate cancer, her doctor had to recommend a colonoscopy 4 times before she was screened, and by that time she had developed colon cancer. She emphasized the need for physicians to be patient and keep recommending screening even in the face of noncompliance or resistance.
Participants were less engaged in the subsequent session about “educational resources for practices.” Although the speaker outlined multiple ways in which they could use these resources, there was little discussion of the materials and few questions.
The final session prompted the group to reflect on their QIC experience and to talk about what they could take back to their own practices. Cross-practice conversations included ideas about improving patient compliance (eg, by reducing fears of CRC screening), coding for CRC screening/education, and strategies to move away from single, in-office FOBT cards as a screening tool.
QIC Translation into Practice Change Efforts
Three themes emerged from our analysis that shed light on how the QIC shaped each practice's QI efforts: (1) how practice staff used the QICs to advance QI changes despite physician resistance, (2) how variation in QIC content and media impacted each practice's QI changes, and (3) how QIC resources were (not) utilized.
Staff Use QICs to Advance Changes despite Physician Resistance
We found instances in which practice staff who attended the first QIC drew on information and insights they acquired there to advance change efforts at their own practices, even in the face of disengaged or resistant practice leaders/physicians. For example, the lead physician at practice 5 had not attended the first QIC while all RAP team staff had. At the first RAP meeting after the QIC, the team discussed the idea of improving CRC education during visits. The physician on the RAP team asserted: “It's not going to be changed because there's not time to do that and there's no need…” He emphasized low reimbursements and time constraints as key reasons for not changing his current screening/education routines. A nurse responded, “But we want to make sure you're educating [patients].” Another staff member added, “You [the physician] missed all the percentages that [the speaker] went over at [the QIC].” Another team member argued that “patients don't really understand that colonoscopies are a cure.” The physician interrupted this discussion, saying: “Well I tell them that… that's actually part of my speech.” The nurse then responded [with some sarcasm]: “Alright, we should do a survey—do [the patients] really understand what you're telling them, because I didn't know that before attending [the QIC].”
During subsequent RAP meetings at practice 5, staff members continued to focus on ways to improve patient education for CRC, including making welcome packets with CRC educational materials. Although the lead physician was willing to be part of the QI process, he did not act as a CRC improvement “champion,” that is, one who actively promotes a desired change. Rather, he tended to react defensively to proposed changes that impacted patient care, which he perceived to be solely the physician's domain. Having gained information and insights from the QIC, the staff became the primary drivers of their CRC screening efforts. Toward the end of the intervention, however, the physician became more active in their QI plans to improve CRC education, even indicating that he would attend the second QIC and share their CRC educational DVD and handouts with other practice representatives.
Mixing QIC Content and Media Optimizes Participant Impact
The SCOPE QICs included a mixture of content (about cancer-specific topics and practice change strategies) and media (such as a skit, PowerPoint-based educational presentations, a cancer survivor's life story, and group discussions/reflections). We found that these different formats resonated with QIC participants in a variety of ways and consequently influenced individual practice change efforts in diverse ways.
The educational presentations impacted some QIC participants, as evidenced by how they remembered key points and then drew on these in their own RAP team discussions to shape their QI changes. The CRC surgeon's key message that single FOBT cards are virtually worthless as a CRC screening tool was remembered and mentioned at subsequent RAP meetings by at least one QIC representative from all 5 practices. For example, after attending the QIC, a nurse at practice 2 told the RAP team, “I learned [at the QIC] that the [single] stool cards are not a good indicator of colon cancer…but we have a lot of underinsured patients [who cannot afford colonoscopies].” This prompted a brainstorming activity from which this RAP team generated ideas for using 3-card FOBTs and increasing access to colonoscopies.
For some other QIC representatives, we found that the emotional nature of the CRC survivor's story resonated more strongly. For example, after attending the QIC, a nurse at practice 3 returned to her practice and shared with the RAP team that, “The lady [at the QIC] who survived colon cancer, she was very good. I've been telling [patients] about her since!” QIC representatives from practice 2 also shared details from the cancer survivor's talk during one of their own RAP team meetings. This prompted a nonclinical team member who did not attend the QIC to ask about the removal of polyps during a colonoscopy. The physician responded with an impromptu mini-lecture for both clinical and nonclinical team members about how colonoscopies prevent CRC. Across multiple practices, these kinds of conversations—all of which stemmed from the cancer survivor's story—helped RAP teams identify the need to educate patients and practice members about CRC screening as part of their QI efforts.
We also found that some QIC representatives drew specifically on “change process” messages from the QIC, such as how to generate staff buy-in and sustain motivation to engage in team-based QI efforts. For example, both the physician and office manager from practice 1 were highly engaged in discussions during the first QIC, asking numerous questions about how to run effective team meetings and how to gain buy-in from their staff for improvement ideas. When they returned to their own practice, they both exhibited signs that they were intentionally importing strategies learned at the QIC to build teamwork and generate buy-in among their own staff. For example, at the first RAP meeting after the QIC, the physician facilitated the process by saying, “I want to hear what the silent [team members] have to say… [Any] thoughts on how we can get our ideas into something that will work for the practice?” These kinds of comments were not heard in RAP meetings at practice 1 before the first QIC. In fact, during the QIC this physician reflected on his own past change efforts that failed because he did not get staff buy-in. Given that practice 1 did not hold any staff meetings before the SCOPE intervention, these kinds of process changes seemed to be substantial for this practice.
QIC Resources Had Minimal Impact on Local Change Efforts
Unexpectedly, the resources and materials provided at the QICs did little to spur practice change efforts. These resources included take-home binders containing articles and QI tools, the opportunity to consult with a colorectal surgeon, and assorted bilingual educational materials. Only 2 RAP teams minimally discussed using the educational materials and one team (practice 2) ignored them and, instead, spent considerable time researching CRC educational materials on its own.
Discussion
In a quality improvement collaborative, teams within a health care organization or across multiple organizations are tasked with making improvements by implementing ideas to address a particular need or problem. There are complex interactional dynamics that occur throughout such a process, but little research has focused on them or examined how QIC participants use information and insights gained from QICs in their own particular QI change efforts. Our article begins to fill this gap by exploring how a QIC impacted the attendees and how the attendees subsequently used QIC information and insights to shape their own QI change efforts. In doing so, this study benefitted from the SCOPE intervention design, which called for each practice to form its own QI team comprising clinicians and staff and to send representatives to 2 QICs. Audio recordings and written field notes from these meetings allowed the authors to study what took place at the QIC, how representatives then shared information and insights with their own RAP teams, and how these interactions shaped each practice's QI change efforts.
Based on this analysis, we offer 3 recommendations for QIC planners. First, it is important to ensure that both leaders and support staff from each organization attend the QIC because they are likely to take different roles in promoting QI efforts later. Previous research has shown the importance of involving key stakeholders or “project champions” in QICs.6–8 A key finding from our study showed that staff who gained new insights from the QIC played a critical role in advancing the change efforts at their own practices despite physician resistance. Although empowering team members outside traditional leadership positions can be valuable, it also has the potential to create conflict between individuals of differing status and power within organizations. As Edmondson and Smith's22 work has shown, when team conflicts encompass opposing values or deeply held interests, emotional and relational conflict is virtually inevitable. Rather than avoiding or dismissing such conflict, teams can put it to good use. Having a skilled facilitator to guide a team through such conflicts can be critical for ensuring that communication and productivity are not hindered.23,24
Second, it is also important to ensure that a QIC program incorporates a variety of offerings, including fact-based, educational presentations, real-life stories when possible, and group reflection/discussion. Our analysis showed that some QIC participants drew on cancer/screening facts and figures whereas others resonated with the emotional life story of the cancer survivor. It was also apparent that group reflections provided unique opportunities for cross-communication and shared learning. Because people learn differently,25 QIC leaders cannot necessarily predict in advance which key messages or media will resonate with QIC participants. Therefore, a mixture of content and media can help ensure that important ideas and insights reach all participants.
Third, planning time and energy should be shifted away from the provision of extensive resources and toward the inclusion of cross-communication and shared learning at the QIC. This recommendation is bolstered by previous findings that suggest that less sharing among participants results in slower improvements.6 Because participants in our study placed little value on the QIC resources offered, we suggest that they be customized for each group when possible; this could be part of a follow-up process. We recognize that the provision of resources will vary depending on the scope, structure, and objectives of the QIC.
Our analysis has several limitations. First, the small sample limits its generalizability. Nevertheless, because of the dearth of empirical research about how QICs shape change, our qualitative analysis spanning the entire intervention adds a valuable perspective that is currently missing in the QIC literature. A second limitation is that we cannot report on how our QICs impacted clinical outcomes (ie, cancer screening rates) because of the small sample size. We note, however, that our focus was on how QICs impact participants' change processes, not whether or not they ultimately improved cancer screening rates. Our focus, rather, is at the heart of the call to open up the QIC black box. Last, we are mindful of issues of researcher bias and the validity and reliability of our themes and findings. We took steps to minimize biases through ongoing discussions among the authors to verify our coding scheme, interpretations, and conclusions.
Conclusions
The US health care system is undergoing rapid and continual change. Effective strategies to facilitate the adoption and implementation of new information, guidelines, technologies, and procedures into practice are needed. QICs offer a potentially powerful way of disseminating health care innovations through enhanced strategies for learning and change. Creating collaborative environments in which diverse participants learn, listen, reflect, and share together can enable them to take key messages and change strategies that benefit them the most back to their own organizations.
Appendix: Supporting Colorectal Cancer Outcomes Through Participatory Enhancements (SCOPE) Quality Improvement Collaborative Agendas
Notes
This article was externally peer reviewed.
Funding: The quality improvement trial, on which this secondary data analysis is based, is supported by a grant from the National Cancer Institute (R01 CA112387-01).
Conflict of interest: none.
- Received for publication March 8, 2011.
- Revision received July 15, 2011.
- Accepted for publication August 15, 2011.