Abstract
Introduction: Primary care risk stratification (RS) has been shown to help practices better understand their patient populations' needs and may improve health outcomes and reduce expenditures by targeting and tailoring care to high-need patients. This study aims to understand key considerations practices faced and practice experiences as they began to implement RS models.
Methods: We conducted semistructured interviews about experiences in RS with 34 stakeholders from 15 primary care practices in Oregon and Colorado and qualitatively analyzed the data.
Results: Three decisions were important in shaping practices' experiences with RS: choosing established versus self-created algorithms or heuristics, clinical intuition, or a combination; selecting mechanisms for assigning risk scores; determining how to integrate RS approaches into care delivery. Practices using clinical intuition found stratification time-consuming and difficult to incorporate into existing workflows, but trusted risk scores more than those using algorithms. Trust in risk scores was influenced by data extraction capabilities; practices often lacked sufficient data to calculate their perceived optimal risk score. Displaying the scores to the care team was a major issue. Finally, obtaining buy-in from care team members was challenging, requiring repeated cycles of improvement and workflow integration.
Discussion: Practices used iterative approaches to RS implementation. As a result, procedural and algorithmic changes were introduced and were influenced by practices' health IT, staffing, and resource capacities. Practices were most successful when able to make iterative changes to their approaches, incorporated both automation and human process in RS, educated staff on the importance of RS, and had readily accessible risk scores.
- Chronic Disease
- Colorado
- Disease Management
- Electronic Health Records
- Health Expenditures
- Information Technology
- Medical Informatics
- Oregon
- Primary Health Care
- Qualitative Research
- Risk Adjustment
- Surveys and Questionnaires
- Workflow
Risk stratification in health care involves the division of patient panels into tiers or strata based on health and social factors to identify and address potentially avoidable and expensive adverse health outcomes. Ideally, risk levels should correspond to the likelihood that a patient will experience negative health outcomes, higher rates of unnecessary or preventable health care utilization, and/or increased financial burden. The process generally involves several steps: defining the included population, systematically assigning risk levels through algorithmic calculations and/or clinical intuition, and subsequently employing a population-based approach to mitigate or address risk, such as complex care management or care coordination.
Risk stratification can help care teams better understand their populations' needs and reduce costs by targeting and tailoring care to high-needs patients with complex medical conditions and social issues1⇓⇓–4 who constitute a large proportion of overall health care costs and utilization.5,6 Patients are not uniform in their needs, and developing interventions to manage complex patients requires parsing those needs to match the approach. The potential benefits of risk stratification for population health management have led national programs such as the Comprehensive Primary Care (CPC) initiative (and the subsequent CPC+ model) to require participating practices to risk stratify their empaneled populations.7 Similarly, Accountable Care Organizations and state-led initiatives have also emphasized the importance of identifying and directing resources toward complex patients to better address their needs, lower costs, and improve health outcomes.8 Ultimately, risk stratification is one of the first steps for allocating resources for more intensive care management for high-risk and rising risk patients.
Selecting and implementing a stratification approach is complicated. Many computerized algorithms exist to predict high utilizers, such as the Hierarchical Conditions Categories,9 each with varying risk criteria. For example, some analyses indicate patients' self-management capabilities may play a critical role in predictive utilization models,10 while others argue that patient-reported health measures alone may suffice in predicting high-needs patients.11 There is also extensive variation in implementation strategies and workflow processes,7 thereby creating additional challenges for primary care practices when selecting optimal approaches for their patients and care teams. In addition, primary care practices often lack adequate reporting and analytic tools, thus creating another barrier in adopting stratification approaches.12
In primary care settings, risk stratification remains relatively new. Few practices have substantial experience in the process.13⇓–15 While existing research focuses largely on the predictive validity of specific algorithms, less emphasis has been placed on understanding how practices approach and experience implementation.16 This article uses a qualitative approach to describe practices' experiences implementing risk stratification procedures and to identify key considerations when developing these workflows.
Methods
This analysis was conducted as part of a mixed-methods study to understand the implementation and impact of risk stratification on clinical workflows and patient outcomes. The Institutional Review Board at Oregon Health and Science University approved this study. All participants consented to participation.
Sample
Using publically available lists from Oregon, Colorado, and Kentucky, we identified practices participating in health care initiatives requiring risk stratification. Thirty-seven practices agreed to participate in the study and completed surveys.17 From these 37 practices, we used survey data to select a maximum-variation sample of practices to participate in interviews based on practice size, location, ownership, and scores on a self assessment of risk stratification accuracy and utility. We contacted 20 practices that completed the initial risk stratification survey and a total of 34 individual participants representing 15 practices (Range, 1 to 4 respondents per practice) participated in interviews. To recruit participants for interviews, we asked selected practices to identify staff members who were involved in risk stratification processes including development, implementation, and using the risk scores, and invited them to participate in an interview. We then used a snowball sample to identify others within a particular practice who were knowledgeable about risk stratification. We discontinued participant recruitment on reaching saturation, when further interviews stopped providing novel insights into the practices' risk stratification processes.
Data Collection and Data Management
Semistructured interviews were conducted by experienced qualitative researchers and followed a semistructured guide (see Appendix 1 for interview guide). We referenced the University of Wisconsin's Systems Engineering Initiative of Patient Safety 2.0 (SEIPS 2.0)18 and the Consolidated Framework for Implementation Research19 to develop interview questions to identify potentially relevant contextual elements and preintervention components (eg, what relevant internal and external components were in place) that might affect practices' experiences. Interviews were conducted by telephone and lasted approximately 1 hour. Interviews were audio recorded, professionally transcribed, and reviewed for accuracy before being deidentified and entered into Atlas.ti software (Berlin, Germany) for data management and analysis.20
Analysis
We used an inductive, thematic analytic approach using an immersion-crystallization process involving iterative cycles of data analysis and reflection to identify patterns and themes.21 During the first phase of analysis, our research team read interview transcripts, listened to audio recordings, and collaboratively established codes to identify recurring concepts in the data. We used an inductive process for developing our list of codes. We continued to review the transcripts together until our codebook was no longer changing, and each code had a clear definition. Next, we divided the remaining data among team members to code individually, meeting weekly to discuss emerging findings. We revisited earlier transcripts to update and refine codes as needed. During this phase we identified features of different algorithms and gained a deeper understanding for how practices used risk scores to inform care delivery. When we identified outstanding questions not answered by our interviews, we contacted interview participants to gain clarity. Finally, we analyzed data across practices to understand the varied experiences practices had developing risk stratification processes and adopting those processes into practice.
Results
The 15 practices that participated in interviews varied in demographic characteristics (Table 1). Eight were large (>8,000 patients), Five were independent, and 10 were urban. Practices were evenly distributed between algorithm type (automated, manual, and hybrid), and all incorporated diagnoses to calculate risk scores. We interviewed 15 care managers, 8 administrators, 5 physicians, 4 quality improvement staff, and 2 medical assistants. Compared with participating practices, those that declined interviews, but completed the survey tended to be small, rural practices. Practices of all sizes cited time constraints as the primary reason for declining interviews.
Characteristics of Participating Practices
We found 3 key decisions were important in shaping practices' experiences with risk stratification: a) choosing to develop novel criteria or using existing instruments and criteria, b) selecting mechanisms for assigning risk scores to patients (automated, manual, or a combination of the 2), and c) determining how to integrate the risk stratification approach into care delivery. For all practices, this process was highly iterative, with refinements required to improve score trustworthiness and usability. Although participants identified areas for improvement in their risk stratification workflows, they also reported that using this population management tool provided a more accurate view of their patient population, therefore helping their teams target resources more effectively.
Choosing Novel or Existing Criteria
Participants considered the following factors when choosing between creating a novel approach and adopting an existing model for risk stratification: availability of data and resources, existing clinical workflows, electronic health record (EHR) and other health information technology (HIT) capabilities, and staffing capacity.
Practices that adopted their own approaches wanted to tailor the stratification criteria to their patient population or include criteria not present in existing algorithms. A physician explained that after reviewing existing approaches, his practice concluded that, “Although there were reasonable approaches, they were… too broad and did not encompass some of the things that we thought would place a patient in higher risk” (Practice G.1). For example, many existing approaches did not emphasize psychosocial or behavioral health criteria, payer reports, or emergency department data, and practices felt these factors would improve the predictive validity of their approach.
The most commonly selected existing approach was the American Academy of Family Physicians model which consists of a 6-tier rubric guiding care team members through risk stratification of patients using variables to assess health status, hospital and emergency department utilization, and social determinants of health.22 Existing approaches were appealing because practices perceived them to be reputable and established, which helped mitigate many of the practices' concerns with deciding which criteria were important when initiating this process. Table 2 displays the benefits and challenges associated with using an existing approach versus creating a new one that practice representatives identified in interviews.
Benefits and Challenges Associated With Adopting Existing Risk Calculation Criteria Versus Developing Novel Criteria
Selecting Mechanisms for Assigning Risk Scores
Practices implemented automated algorithms, manual approaches, or hybrid combinations of approaches to assign risk scores to patients; practices encountered different efficiencies and challenges specific to their selected approach. Table 3 shows the variation in risk stratification approaches, the practices that selected each, their associated workflows, and the challenges and benefits practices experienced.
Variation Among Risk Stratification Approaches
Automated stratification approaches required selecting data sources and programming risk score calculation into the EHR or other database. These approaches were efficient for stratifying large patient panels, particularly when the HIT could automatically update scores. However, using these algorithms required technical skills, significant time to integrate into existing HIT, and occasional assistance from external consultants or vendors.
Practices often looked to EHR vendor add-ons when they encountered technical problems postimplementation. However, this created additional challenges when discrete software did not always “speak well [with other modules]…. And, there is a lot of development that happens [for risk stratification] after you have bought these pieces of technology” (Practice A.1; Manager of Quality Improvement). These technical issues sometimes stalled progress while practices sought solutions through vendors or IT departments.
To establish consistency in score calculations, automated algorithms required EHR users to standardize documentation workflows. Practices found staff often needed repeated reminders to ensure accurate data entry:
“Emergency department visits, for example, could be entered 3 different ways… We did provider training and nurse training to make sure that they all entered it that way. And then from time to time we go through, usually it is about every 3 to 4 months, and run a report to see if it is being entered incorrectly again. And then remind everybody to enter it the correct way” (Practice I.1; RN Care Coordinator).
Algorithm software not aligning with existing workflows often led to inconsistent documentation behavior and contributed to practices using automated algorithms to question the trustworthiness of their data more compared with practices using manual and hybrid methods for risk score assignment.
Manual approaches used rubrics containing criteria related to patients' conditions and/or health care utilization to calculate patient risk scores. Manual approaches allowed practices to generate and update risk scores without advanced technical expertise, and to incorporate information not available in the EHR. As a process improvement director described, “It is a manual process here. It is not something that our computer system can pull. Because we felt like it would not be as accurate if the system was just pulling it, especially when we're talking about the social factors” (Practice C.1). Practices that selected a manual approach valued the ability to integrate social information, and this helped them trust the risk scores.
Practices using manual approaches faced several challenges: time requirements to stratify patients, difficulty incorporating the process into existing workflows, and issues related to the lack of integration with EHRs. As one physician explained, “It is hours of work, and so that is the drawback. It is getting the providers that can sit down and do that. And do it well” (Practice G.1). To mitigate these challenges, some practice managers gave clinicians shorter lists of patients to review on a daily basis in place of longer lists or more complex workflows. In some cases, medical assistants assigned the score before clinician adjudication to complete the process faster.
Hybrid approaches combined elements of manual and automated algorithms. Practices using these approaches experienced the benefits of automation (ie, more efficient than fully manual systems) and manual assignment (ie, the inclusion of more nuanced psychosocial data that is difficult to pull from an EHR). One quality improvement coordinator at a practice that transitioned from a hybrid to automated approach due to an EHR change described this approach compared with a purely automated one:
“The best was the algorithm combined with provider intuition. That was by far the most useful…. It did not rely on somebody noticing something or somebody coding something right. That was just a much better way of going about it” (Practice G.0).
This respondent further explained how the hybrid model helped ensure data were captured accurately and all relevant information could be included in the risk score assignment. Overall, practices using hybrid approaches trusted the scores and found the manual component was worth the additional time because it offered a more accurate picture of their patients' individual circumstances affecting risk.
Integrating the Approach Into Care Delivery Through Access and Adjustments
Following risk score assignment, care teams needed to be able to access, update, and modify the scores to use them in care delivery and decision making processes. Most practices stored risk scores in their EHR, but the location varied. To make the score easy to view some placed it in the patient record banner (the banner refers to an element across the top of EHR screens that identifies the patient and carries key information, such as age or location), often incorporating a color-coded display to identify high-risk patients. Others stored the score in commonly accessed locations such as the health maintenance and patient demographics sections of the EHR. However, simply storing the score in the EHR did not always make it useful; if the score was in a rarely-accessed location, it was not actively utilized in care decisions.
Practices not storing the score in the EHR either relied on external spreadsheets, or indicated that patient scores were not well integrated into care workflows. Ultimately, practices with easily accessible scores reported using risk scores to drive care delivery (eg, triaging high-risk patients, referring them to care managers or other care team members, or automatically giving them longer appointments).
Practices needed the ability to adjust risk scores to continuously portray an accurate reflection of patients' needs. An adjudication process allowed clinicians to weigh factors that were not captured by the practices' initial stratification approach, such as patient history and social determinants. Practices using adjudication indicated that it enhanced their overall perception of risk stratification accuracy and identified patients who were missed by initial approaches:
“We had patients that might have only scored a 3 [on a 5-point scale] and some physicians would say: Well, this patient should be a lot higher. This patient's a lot sicker than your models. Some patients they looked at, and [they] moved them higher in the scale. Some patients they looked at and said: Although you have these components, this patient does not really hit a 5. And so they might have downgraded them” (Practice I.1; Administrator).
In addition to capturing nuanced information, practices needed to update scores when patient risk status changed over time. Some practices had processes for updating the score on the availability of new information (eg, hospitalizations or new diagnoses), or on a predetermined schedule for the entire patient panel (eg, at point of care, quarterly, or annually). A physician in an independent practice using a manual approach explained his views on this process, simultaneously acknowledging the necessity and challenge of updating scores as health statuses evolve:
“Over the course of time, patients change in terms of risk level and making sure the providers are in tune with documenting changes in risk and documenting to our care managers changes in risk, so care managers can change their focus and their intensity of interventions with patients as they move up and down the risk scale. So not only just implementation but ongoing maintenance of the risk list is quite a bit of work” (Practice G.1).
Though all practices had some process in place to update risk scores, the variation in frequency corresponded to the ability to update them rapidly and the utility in reviewing the scores at set times.
Discussion
Practice members reported 3 crucial decisions for selecting and implementing risk stratification processes in their practices. These included selecting a pre-existing or novel approach, determining mechanisms for risk score assignment, and developing processes for integrating risk scores into care delivery. These important implementation decisions were often revisited—through trial and error—to improve stratification processes over time. These practices iteratively refined their risk stratification processes, which aligns with findings by Hong and colleagues,23 who showed that prediction of high-needs patients improved after several changes were made to the initial stratification process.
Our findings indicate practices should balance automation with human review when possible. Practices relying entirely on automated algorithms did not trust their scores to the same degree, and staff using the scores were not always able to describe criteria they used to establish risk levels. Interviews substantiated and added detail to similar findings from our initial survey of trust in various risk stratification approaches from a larger sample of practices and staff.17 Recent research analyzing the types of risk stratification approaches in primary care reported similar findings, emphasizing the critical role of integrating clinical intuition in successful stratification outcomes and subsequent care management service allocation.24 Furthermore, participants emphasized the importance of including social factors in risk stratification and found these items difficult to identify through typical EHR data, thereby necessitating interactions with patients and considering additional information when creating or updating risk scores. These findings support the insertion of a human element into risk stratification to increase trust in scores, preferably through comprehensive adjudication. Adjudication provides an opportunity to validate automated scores, assess accuracy of algorithms, and identify areas for refinement; most importantly, it allows clinicians and care teams to incorporate clinical judgment and other information into risk scores to improve stratification and patient care.
Our findings also show it is imperative for patient scores to be readily accessible and highly visible to everyone on the care team, preferably in EHR locations frequently accessed when delivering care. The EHR's role is substantial, though practices should be careful not to rely on technological solutions too heavily for all obstacles throughout the risk stratification process, as doing so can result in stagnation when challenges arise regarding algorithm updates or weighting risk criteria, and also in disjointed approaches when purchasing add-ons that may not integrate well together.
When considering approaches, it is important to remember that risk stratification implementation remains highly iterative, and refinements are likely to be required. If primary care practices plan for refinements in advance, they may be better equipped to address barriers and make changes as they arise. In addition, educating staff on the purpose behind risk stratification is critically important, and practices endeavoring to implement this work would do well to emphasize these aspects routinely.
Limitations
Additional research with a more representative sample has the potential to build on this initial work. Our sample included practices with a range of characteristics, but did not include any with fewer than 3,000 patients, and smaller practices' experiences may differ from those of larger practices. It is also possible that our respondents were more positive about risk stratification compared with those who declined.
All participating practices were involved in a federal initiative requiring and incentivizing risk stratification (among other practice transformation efforts); many of the practices cited this as the primary reason for implementing risk stratification. Through these programs, practices received support from other practices and program staff dedicated to providing technical assistance. This could potentially influence the approaches they chose and how they dealt with challenges. Nevertheless, this support made it possible for practices to implement risk stratification and gain this experience, which was critical to understanding practices' challenges and opportunities, and underscores the importance of focusing on this group of implementers. Currently, all primary care practices in the United States can receive special payments—through Chronic Care Management codes or similar programs—to identify and care manage high-needs patients; this work could inform their processes but would need to be revalidated among those practices. Furthermore, recent work has highlighted that not all high-needs patients have the same risks or should receive the same interventions12,25 and that there are differences in nominal and ordinal assignments of risk. Most practices we interviewed employed categories: they stopped at identifying high risk patients, but did not further segment patients into additional groups. Practices indicated they understood this goal, though further research into these processes is needed to better understand the implications it has for implementation and care delivery. In addition, our findings were influenced by implementation science frameworks to explore interactions between local personnel, tasks, and technology to better understand processes and goal outcomes, but the primary results were derived directly from data, represent exploration of a nascent area, and future interventions should address the challenges and barriers before identifying core components.
Finally, this work focuses primarily on integrating risk stratification into clinical workflows rather than downstream effects on care management. Surveys in our other work also analyzed staff perceptions of care management,17 though additional research exploring approaches for capturing patient information not available in EHRs and how this might impact care management processes and perceptions is warranted.
Conclusion
Practices engaged in risk stratification had a number of choices to make: how to implement and refine their risk stratification methods, how to balance trust and efficiency of the process, and how best to facilitate action to support patients stratified into higher-need categories. Our findings identify 3 key considerations that clinicians, researchers, and policy makers should consider.
Acknowledgments
Thank you to Christie Pizzimenti, PhD and Sumeet Singh for reviewing and assisting in preparing this manuscript.
Appendix 1:
Risk Stratification Interview Guide
The questions below cover the general topic areas we will explore with interview participants. These questions will be modified in light of what is learned during practice observation and to fit the expertise of the interviewee.
Thank you for participating in this interview. We are speaking with you today because we are interested in your experiences with risk stratification in your practice. During the interview, we will ask you to tell us a little bit about yourself as well as your thoughts and experiences regarding risk stratification and care management.
Rapport Building
Please tell me a little about yourself.
Tell me about your background
What is your role in the organization?
Please describe your practice.
Ownership and affiliation?
Patient population?
Risk Stratification
Now I am going to ask you some questions about risk stratification in your practice.
3. How do you define the term risk stratification?
Our plain-language definition, if asked: In every system there are patients who have multiple complex problems that are more likely to use the health system. We see risk stratification as a way of looking at the patient panel so patients can be proactively managed and preventing those high risk patients from high cost outcomes like going to the emergency department.
4. Why did you begin using risk stratification in your practice?
What were your goals or expectations for using risk stratification?
5. How did your practice prepare for risk stratification?
How did you initially stratify your patient population?
How do you define your active patient population?
6. What were the challenges in getting started with risk stratification in your practice?
How did you overcome these challenges?
What do you think is driving those challenges?
Now I am going to ask you some questions about your practice's current approach to risk stratification.
7. We are interested in understanding the risk stratification process at your practice. Can you walk me through each of the steps for assigning a risk score to a new patient?
Where does the work start?
What happens next?
[Please make brief jottings to note each step in their process. Then, ask the following questions for each step:]
Who in the practice does this step?
What tools or technology are used to do this step? Where, if at all, does your risk score seem in your EHR?
How does your practice adjust a patient's risk score? Who makes these adjustments?
Under what circumstances do you adjust the risk score?
Which groups of patients do you reduce the risk score?
For which groups of patients do you increase the risk score?
Do you use an algorithm to stratify patients?
If so, please describe it.
Describe the different risk stratification tiers/categories and how you define them.
How did you select the criteria for your tiers?
If not mentioned all ready, what was the process for assigning existing patients a risk score? (Frequency)
Care Management
8. Did you have Care Management in place before RS? Once you have identified high-risk patients through risk stratification, what happens next?
What practice team members are involved?
Care management activities
Integrated behavioral health activities
Others?
What tools or technology are used?
Are services provided within the practice or by an external organization?
If external, what is your relationship and what does that organization do?
How are services targeted or tailored based on risk stratification?
Once a patient is identified for more intensive care, how often are they contacted/seen? For which risk tiers do you target CM interventions?
For how long do they receive more intensive care?
9. How has your risk stratification approach changed over time?
Why has it changed?
10. How useful do you think your risk stratification approach is for improving patient care?
Describe your experience with your practices' risk stratification approach?
How accurate is it?
What information does it provide beyond what you'd known or would have done otherwise?
How are patients distributed across the tiers?
11. Knowing what you know now, what is the most important lesson you have learned about risk stratification?
What has helped your practice conduct risk stratification?
What are the challenges to conducting risk stratification in your practice?
How did you overcome these challenges? – Or, how do you anticipate overcoming these challenges?
[If needed further] What was the source of that challenge? What do you think is driving those challenges?
Thinking about your risk stratification approach today, what would make it more effective?
[If time allows]
12. What is your practice's vision for risk stratification in the future?
Notes
This article was externally peer reviewed.
Funding: DAD received a grant from The Commonwealth Fund to conduct this research.
Conflict of interest: none declared.
To see this article online, please go to: http://jabfm.org/content/32/4/585.full.
- Received for publication November 12, 2018.
- Revision received February 8, 2019.
- Accepted for publication February 11, 2019.
