Abstract
Purpose: Registry implementation is an important component of successfully achieving patient-centered medical home designation and an important part of population-based health. The purpose of this study was to examine what factors are evident in the successful implementation of a registry in a selection of Colorado practices involved in quality-improvement activities.
Methods: In-depth, small-group interviews occurred at 13 practices. The data were recorded, transcribed, and qualitatively analyzed to identify key themes regarding elements of successful registry implementation. Key elements were described as conditions, then calibrated and analyzed using qualitative comparative analysis (QCA).
Results: The QCA revealed several formulas to successful registry implementation. Key conditions included the importance of Resources and Leadership along with either a Quality Improvement Mindset or a Key Person driving efforts (or both). Health System membership affected the specific formula.
Discussion: This study is innovative in that it examines which factors and in what combination are necessary for successful implementation of a registry. The findings have implications for primary care quality-improvement efforts.
Many changes are occurring in primary care as a result of the patient-centered medical home (PCMH), which emphasizes whole-patient care coordinated by a primary care medical practice.1⇓–3 A key part of the PCMH is data-driven decision making.4,5 Such decision making is facilitated by a registry, defined as “a searchable list of all patients with a particular (chronic) condition that often interfaces with an electronic health record (EHR).”6 Registries are also required and relevant to maintenance of certification requirements for physicians and other providers continuing their medical licensure.
There is considerable research on disease registries, their essential properties, and issues in registry implementation.4⇓⇓–7 Conclusions from these various studies in diverse clinical settings indicate that registries are a necessary, but not sufficient component of PCMH and chronic illness care. Research and reviews suggest that keys to successful registry implementation include being compatible with or part of the practice's EHR and workflow; being interactive, easy to use and actionable; providing information to both physicians (providers) and patients/families; identifying high-risk subgroups among identified patients; sending reminders, prompts, and feedback to teams on care; and importantly, linking all members of the health care team.4,6,8 Team-based care has received somewhat less research but is considered necessary for effective quality improvement.9⇓–11
Implementing a registry is often challenging, and better understanding is needed about the factors that impact successful registry implementation. Literature on the use of registries in primary care is extensive, but there is relatively little evidence regarding how to overcome challenges as they occur during registry implementation.12,13 Creating, refining, and updating disease registries and successfully integrating them with EHRs can be time consuming and frustrating. Delays in establishing effective and actionable registries are a frequent impediment to the success of chronic care quality-improvement efforts.14,15 Therefore, adaptations to registries and how they are implemented are common and often necessary.16,17 Much more information and data are needed on what types of registry implementation adaptations practices make along the way to achieve successful implementation.
To address these issues, we initiated a project to examine the characteristics, use, and frequency of adaptations made during the process of working toward PCMH designation and other quality improvement efforts. One key aspect of this work was to examine registry implementation as a part of PCMH. Although the work of other investigators has focused on the frequency of adaptations and how they relate to models of adaptation,18,19 our work was to describe the process and factors associated with successful implementation of a registry in primary care.20 The purpose of this article is to illustrate, using the innovative method called qualitative comparative analysis (QCA), how different factors contribute to the successful implementation of a registry in primary care.
Methods
This study was approved by the Colorado Multiple Institutional Review Board at the University of Colorado Denver. Data were collected during 2015 and analyses were performed in 2015 and 2016.
Setting and Participants
As many primary care practices work to achieve PCMH recognition, grant programs and training initiatives have emerged to help support this goal. We studied the experiences of practices in 1 practice transformation initiative in Colorado. The initiative provided trained external Practice Facilitators21,22 who met with practice clinicians and staff on a regular basis to provide training in quality improvement concepts and PCMH principles to support transformation. Our study team worked with the leaders of this initiative to identify practices for this study.
A first round of the study identified practices' adaptations as they made various changes in support of PCMH. Twenty-seven practices participated in this initial round of interviews (out of 54 original participants). From these 27, we identified 15 for additional in-depth interviews to provide greater clarity about the registry implementation. Practices were selected for this second round of data collection based on a) selecting registry as an adaptation topic, b) to provide geographic distribution of the practice locations across rural and urban areas of Colorado, and c) to obtain diversity in perceived impacts resulting from the adaptation during registry implementation. Thirteen of these 15 completed an interview.
Practices not participating in either round of interviews (n = 29) were not notably different from practices that participated in initial (n = 27) or subsequent interviews (n = 13 subset of the 27) on primary care discipline, practice size, or ownership status. Targeted participants for interviews were individuals who were most familiar with the registry implementation at the practice. This usually included the practice manager and the key implementer (such as the clinical staff member making the registry changes). Practices that did not respond after 3 emails and 3 telephone calls (including voicemail messages) were designated as “declined to participate.”
Data Collection
A semi-structured interview guide was developed by the study team (all authors) to assess the following topics: how the practice became involved with the initial PCMH transformation initiative, initial goals regarding participation, the story of how the practice implemented the registry, and the status of the registry use at present. Question probes included understanding processes, resources, and people involved in implementing the registry and how issues that occurred during the process of implementation were handled.
Interviews were completed in person at the practice location by a coinvestigator (JSH) and a professional research assistant (TH). Both had extensive experience in qualitative research and interviewing primary care practice members. Interviews were approximately 1.5 hours in length. Interviews were audio recorded and professionally transcribed. The interview data were cleaned, labeled, and placed into the qualitative analysis program, Atlas.ti (Atlas.ti, version 6, Scientific Software Development, GmbH, Berlin, Germany). Participants each received a $100 gift card as a thank you for their participation, for their personal use or to share with practice team members, depending on their choice and/or organizational policies.
Data Analysis
After completing all interviews, the study investigator and research assistant (the analysis team) reviewed notes from all interviews both independently and then together to identify key thematic areas and determine potential codes to be used for a coding scheme. They utilized two approaches to analyzing the data. First, a template organization approach23 was used to sort responses into categories consisting of the preliminary codes identified in the first phase of the data review. These were emergent codes using a grounded theory editing approach.24 In addition, we had established in advance to use macrocognition theory25,26 to determine whether this theory was instructive in understanding use of registry, as a priori codes. The interview team qualitatively coded the data providing monthly reports and vetting thematic results with the entire team (all authors).
Second, using a table format, the analysis team organized the emergent responses into a matrix27 consisting of rows of cases (practices and practice groups) and columns for key dimensions for consideration. The conditions included both those emergent from the analysis (such as quality improvement [QI] mindset, leadership, EHR capability, etc. described below in results) as well as important demographic variables (Health System). They then reviewed this matrix in multiple passes to highlight and group categories of adaptations consisting of similar dimensions. These categories were presented to the larger study team including the principal investigator for discussion and feedback, and refined and finalized as described below. The larger study team also reflected on and discussed the analysis team's experience.
Qualitative Comparative Analysis
Combinations of conditions seemed to be important to determining the successful implementation of a registry, rather than single conditions operating in isolation. Furthermore, it seemed that these combinations of conditions might have been clustered by whether the practice was part of a hospital-owned health system or an independent or rural practice. Although the thematic analysis would produce insight regarding the participants' experiences, it would not reveal how these patterns and combinations of conditions might be tied to the outcome of “successful implementation of the registry,” and in what context.
Therefore, the research team decided to use qualitative comparative analysis (QCA). QCA is an analytical method used to study complex causality. Compared with traditional statistical analysis, it may be used to analyze qualitatively-derived, nonrepresentative data.28 QCA relies on qualitative data for the analysis but does include data transformation to numbers. Therefore, it does not represent a purely qualitative process. Given that it is based on Boolean, rather than linear, algebra, it is not constrained by degrees-of-freedom and may applied to small, medium, and large sample sizes. It is therefore well suited to “small-N” comparative case studies with samples of 5 to 50 cases, such as the present one. To complete the QCA, we used the analysis process outlined by Rihoux and Ragin.29
To conduct the QCA, we first reviewed the emergent codes identified through the grounded theory analysis. After consulting our code list and extensive team discussion, we arrived on an initial set of likely explanatory variables (“conditions” in the terminology of QCA). We then rescaled (“calibrated”) these conditions onto a 0.0 to 1.0 scale for each practice, as is required for QCA. The initial set of 7 conditions and the calibration scale is provided in Table 1. Both members of the analysis team then independently scored each practice according to the previously defined calibrations, before coming together to compare and resolve any discrepancies. The final calibration involved going back to the transcripts, recalibrating, and determining their final calibrated scores. This process was repeated for the outcome of “registry implementation success.” Next, the QCA was conducted using Kirq,30 a software program created by coauthor CR. The analysis was completed by the coauthors (JSH, TH with CR) over a series of 3 meetings of approximately 2 hours each. As described below, Kirq was used to test for the existence of both necessary and sufficient conditions. QCA can produce a range of results of varying complexity; for this study, we report the complex solution, which retains all causal complexity and incorporates no simplifying assumptions.31
Factors (Conditions), Condition Descriptors, and Qualitative Comparative Analysis Scoring (Calibration) Used to Identify Key Ingredients in Successful Registry Implementation
Results
Practice Characteristics
Thirteen interviews were conducted. Interviews usually included 1 to 2 (range, 1 to 5) practice members most knowledgeable about the registry implementation in their setting. Characteristics of the practices participating in the interviews are included in Table 2. The calibrated results for each condition are noted in Table 3.
Characteristics of Practices and Participants of In-Depth Interviews
Calibrated Data Table for the Qualitative Comparative Analysis
Combinations of Conditions That Produce Registry Implementation Success
QCA was used to test for both necessary and sufficient conditions. Two goodness-of-fit measures, consistency and coverage, indicate the strength of the relationship between the condition and the outcome. A necessary condition with consistency of 1.0 indicates that the outcome never occurs outside the presence of the cause. For sufficient conditions, a consistency score of 1.0 indicates that whenever the cause occurs, the outcome also occurs. A consistency score of 0.9, in contrast, would indicate that a condition (or combination of conditions) is “almost always necessary” or “almost always sufficient.” For example, it is almost always necessary to be exposed to significant amounts of cigarette smoke to contract lung cancer, because a relatively small number of cases occur due to other causes.
Coverage, a second goodness of fit measure, gauges empirical importance and measures the fraction of cases in the data to which the relationship applies. Coverage is measured in 2 ways. Raw coverage identifies those cases described by a given combination of conditions (“recipes”). Recipes may overlap and >2 recipes may describe the same cases; therefore, unique coverage identifies those cases that exclusively belong to a single recipe. A recipe with high raw coverage explains many cases; a recipe with high unique coverage explains many cases that are not also explained by other recipes. In general, consistency and coverage values over 0.8 indicate strong relationships.
Our initial results, presented in Table 4, revealed 3 recipes that almost always produced successful registry implementation. Consistency for each of these recipes was very high, at least 0.97. The third recipe explains the most cases, as indicated by its high raw and unique coverage scores. This recipe describes instances of successful registry implementation that combine a QI mindset with the presence of a Key Person(s), Resources, and Leadership. Importantly, this combination of conditions operates regardless of Health System membership (ie, owned by a health system or independent practice).
Sufficiency Consistency and Coverage, including the Condition of QI Mindset
The other 2 recipes in Table 4 describe instances of successful registry implementation among large, hospital-owned systems (HEALTHSYSTEM). These recipes combine Resources and Leadership with either the presence of a Key Person(s) or a QI Mindset (both were not required). Note that these 2 recipes are very similar to one another—the only difference is that practice 3 (P3) belongs to the first one, whereas P4 and P7 belong to the second. More importantly, these 2 recipes are largely a subset of the third (which explains the low unique coverage scores). The only observations present in the first 2 recipes that are not in the third are P3, P4, and P7. There is a great deal of overlap between these 2 groups.
Therefore, we interpret these results as providing slightly different explanations of the same data, rather than thinking of these are 2 (or 3) distinct recipes. The first 2 recipes suggest that QI Mindset and Key Person are substitutable for one another, whereas the third recipe suggests that they are complementary and that both are needed. And perhaps the main finding is that Resources and Leadership are always needed (eg, these results indicate that having a Key Person and/or a QI Mindset would not be enough to make up for a lack of Leadership).
The Importance of Necessary Conditions
These results suggested the possible presence of necessary conditions, so we also ran a necessity analysis, using the same conditions and outcomes. A necessity analysis identifies those conditions that must be present for the outcome to occur, all or most of the time. The absence of a necessary condition almost always prevents the outcome from occurring. Table 5 shows that both QI Mindset and Leadership emerged as “almost always necessary” with consistency scores of 0.93 and 0.91, respectively; whereas Resources did not. Excluding QI Mindset from the sufficiency analysis, as is common practice for conditions identified as necessary, produces Table 5. These results indicate that Resources and Leadership, when combined with either Key Person(s) or a large hospital-owned system, almost always resulted in the successful implementation of a registry. This suggests the role of Key Person and of large Health System are in some way serving the same function and interchangeable for one another.
Sufficiency Consistency and Coverage, Excluding the Condition of QI Mindset
Notably, EHR Capability and Incentives did not emerge as components of any of the sufficiency recipes for this analysis. This indicates that practices with limited EHR Capability are able to use alternate means to create a registry. This usually indicated a registry that was not built within the EHR. Thus, some practices were able to implement a registry outside of their EHR. Although our qualitative interviews indicated that Incentives are often an initial driving reason for the decision to have a registry, the QCA results indicate that the presence of Incentives are not relevant to actual successful registry implementation.
The Story from the Qualitative Interviews
Although the results here suggest essential components of successful registry implementation, they do not tell the full story. During qualitative interviews, participants shared their experience of registry implementation, often revealing how these key elements developed over time. For example, the health system leadership would hear about PCMH and payment changes “coming in the future” and communicate this to practice leaders to investigate and participate in the PCMH transformation initiative. This was often followed by the practice providers and staff complaining about making changes, but with good leadership (usually at both the health system and practice level), this resistance would fairly quickly turn into an understanding of how transformation could improve patient care and result in continued reimbursement for patient care (which was often the start of the QI Mindset).
Leadership would then further support continued efforts to implement the registry (a key component of PCMH) by providing Resources such as staff time, training, and support to implement needed changes. Sometimes this was in the form of identifying a Key Person and providing training and time to change the query patterns in the EHR or identify a registry system external to the EHR. Sometimes this was in the form of dedicating information technology staff time to make the changes to the EHR so that the registry functions could be used properly. Although EHR Capability was important, it did not emerge as a key condition because successful implementers either worked around it (using a registry external to the EHR system) or figured out a way to make it work at a minimal level of sufficiency for their needs. The QI Mindset emerged again throughout the discussion as stories were told of using the registry data to drive new goals for patient care improvement and sharing success stories of patients affected by the changes.
Discussion
Patient registries should help front-line clinicians identify and improve health outcomes for patients who could benefit from additional care. Functioning disease management registries are generally viewed as a prerequisite during the initial stages of successful quality improvement efforts.14,15 However, transitioning from not having a registry to implementing and using one can be challenging for the typical primary care practice. Understanding the factors important to success might help practices with their implementation.
In this study, among the 13 practices included in our sample, our analysis identified 1 necessary condition required for successful registry implementation, the development of a QI Mindset. Among those practices that developed a QI mindset, we found 2 separate contributions of other facilitators (recipes) that consistently resulted in successful registry implementation—a combination of sufficient Resources with Leadership, plus either the involvement of a Key Person or being part of a large Health System. These findings extend and provide a more in-depth and empirical answer to the question of what predicts successful registry implementation and successful adaptation.4,7
These results confirm previous studies7,8 of successful QI and registry implementation in that issues around the EHR and quality improvement interfaces were important, as were policy and reimbursement issues.4 In addition consistent with prior reports, leadership, and linkage across team members for QI efforts were important components.8,12 In addition, this study adds to the literature in 2 important ways. First, our focus was specifically on understanding adaptations made during the implementation process. A second important addition to the literature is that EHR Capacity did not emerge as essential to success while QI Mindset was revealed as critical. Surprisingly, hypothesized factors such as Incentives were not part of emerging recipes for success. The results of this QCA analysis identified new ways of understanding how adaptations are considered and attempted.
Limitations
This study is limited by the practice population we included: all practices participated in a single PCMH transformation initiative in Colorado, and only those that were working on registry implementation. However, that group of practices included a variety of practices and situations that may broaden the generalizability of our findings. In other contexts, additional financial, cultural, and organizational factors may affect the process of registry implementation. In addition, only 1 of several quality improvement initiatives undertaken by these practices was studied and reported here. In addition, although much thought went into the variables (or conditions) for consideration, it is possible that we did not formulate the ideal combination of conditions or calibrate them in ways that accurately reflect their occurrence in practice. We lacked definitive outcome data on the extent to which the registry was actually implemented. However, these analyses are inherently qualitative in nature with QCA adding a data transformation element to bring to light the patterns of the qualitative data. Finally, it remains to be seen whether similar conditions are necessary or sufficient for successful implementation or adaptation or other aspects of practice transformation such as team-based care.
In summary, conducting the QCA provides a new perspective on the qualitative data and raises new questions about how to successfully implement a registry in primary care and make adaptations to aid in its implementation. Future research is needed to replicate these findings and explore the generalizability using different settings, practice transformation issues, and analysis approaches.
Notes
This article was externally peer reviewed.
Funding: Funded by Robert Wood Johnson Foundation grant No. 71732.
Conflict of interest: none declared.
To see this article online, please go to: http://jabfm.org/content/30/5/657.full.
- Received for publication March 1, 2017.
- Revision received May 25, 2017.
- Accepted for publication May 28, 2017.
