Abstract
Background: Submission of clinical quality measures (CQMs) data are 1 of 3 major requirements for providers to receive meaningful use (MU) incentive payments under the 2009 Health Information Technology for Economic and Clinical Health (HITECH) Act. Some argue that CQMs are the most important component of MU. Developing an evidence base for how practices can successfully use electronic health records (EHRs) to achieve improvement in CQMs is essential and may benefit from the study of exemplars who have successfully implemented EHRs and demonstrated high performance on CQMs.
Methods: Conducted in PPRNet, a national primary care practice-based research network, this study used a multimethod approach combining an EHR-based CQM performance assessment, a provider survey, and focus groups among high CQM performers. Practices whose providers had attested for stage 1 MU were eligible for the study. Performance on 21 CQMs included in the 2014 MU CQM set and a summary measure was assessed as of October 1, 2013, through an automated data extract and standard analytic procedures. A web-based provider survey, conducted in November to December 2013, assessed provider agreement, staff education, use of EHR reminders, standing orders, and EHR-based patient education related to the 21 CQMs. The survey also had more general questions about the practices' use of EHR functionality and quality improvement (QI) strategies. Statistical analyses using general linear mixed models assessed the associations between responses to the survey and CQM performance, adjusted for several practice covariates. Three focus groups, held in early 2014, provided an opportunity for clinicians to provide their perspectives on the validity of the statistical analyses and to provide context-specific examples from their practice that supported their assessment.
Results: Seventy-one practices completed the study, and 319 (92.1%) of their providers completed the survey. There was wide variability in performance on the 21 CQMs among the practices. Mean performance ranged from 89.8% for tobacco use screening and counseling to 12.9% for chlamydia screening. In bivariate analyses, more positive associations were found between CQM performance and staff education, use of standing orders, and EHR reminders than for provider agreement or EHR-based patient education. In multivariate analyses, EHR reminders were most frequently associated with individual CQM performance; several EHR, practice QI, and administrative variables were associated with the summary quality measure.
Conclusions: Purposeful use of EHR functionality coupled with staff education in a milieu where QI is valued and supported is associated with higher performance on CQM.
The 2009 Health Information Technology for Economic and Clinical Health (HITECH) Act was intended to improve the quality of health care provided to the American public by incentivizing the meaningful use (MU) of electronic health records (EHRs), defined as their use by providers to achieve significant improvements in care.1 The legislation empowered the secretary of health and human services to develop specific MU objectives; the Centers for Medicare and Medicaid Services, in collaboration with the Office of the National Coordinator for Health Information Technology, are the federal leads for this development and implementation process. The process was designed to evolve in stages, with stage 1 (beginning in 2011) focusing on data capture and sharing, stage 2 (delayed until 2014) on advanced clinical processes, and stage 3 (delayed till 2017) on improved outcomes. For each stage, submission of data on clinical quality measures (CQMs) is 1 of 3 major requirements for providers to receive MU incentive payments.
Some have argued that CQMs are the most important component of MU2 as tools that measure and track the quality of care3 and because a focus on outcomes is a critical requirement for reengineering our health care system.4 That more needs to be learned about the process of how to use EHRs in a manner to improve health care quality has been recognized,5 however, given mixed evidence that EHR use alone improves quality.6⇓⇓⇓–10 EHR use is likely necessary, but not sufficient, and organizational changes such as team-based care, population management, and other strategies are required to achieve improvements in quality.11,12
Developing an evidence base for how practices can successfully use their EHRs to achieve improvement in outcomes is essential and may benefit from the study of exemplars—those who have successfully implemented EHRs and demonstrated high performance on CQMs. In this report we present the results of a multimethod study combining an EHR-based CQM performance assessment, a provider survey, and focus groups among high CQM performers.
Methods
Setting
The study was conducted in PPRNet, a national, EHR-based, primary care practice-based research network and Agency for Health care Research and Quality Center for Practice-Based Research and Learning.13 PPRNet maintains a longitudinal clinical database derived from regular electronic data extracts from the EHRs used by participating practices. The PPRNet database is used for quality reporting and research.14⇓⇓⇓⇓–19 Current reports provide feedback on the practice, provider, and patient level for >60 quality measures encompassing primary and secondary prevention, disease management, and safe medication prescribing and monitoring, as well as summary measures.20 Twenty-one of the PPRNet measures are comparable to the 2014 Centers for Medicare and Medicaid Services CQMs in the eligible provider incentive program.21
Participating practices were recruited through a series of E-mail messages sent in August and September 2013. PPRNet practices whose providers had attested for MU stage 1 were eligible for the study. Those that met this eligibility requirement and agreed to participate in the provider survey—and the focus groups, if invited—were enrolled. Enrolled practices that either did not submit data for an October 1, 2013 PPRNet report or failed to have a majority of their providers complete the survey were excluded from the analyses.
EHR-Based CQM Performance Assessment
Performance among participating practices on the 21 CQMs (Table 1) was assessed as of October 1, 2013, using a method described in detail elsewhere.22 Briefly, practices ran an extract program that generated a clinical data file for all clinical documentation since their last extract. Extracted data were posted to a secure server and added to the PPRNet database, where a longitudinal, anonymized record is maintained for each patient. Data processing included matching uncoded text strings to common terminologies and standard data dictionaries (International Classification of Diseases, Ninth Revision, Clinical Modification, for diagnoses/problems and Logical Observation Identifiers Names and Codes for laboratory results). Data tables (patient demographics, diagnoses, laboratory results, procedures, vital signs, medications) were combined across the PPRNet database using structured query language joins to calculate a CQM. Programs searched relevant patient data to identify those eligible for the CQM and subsequently dichotomized each patient as having met the criteria or not. For example, to identify patients eligible for the “controlling high blood pressure (BP)” measure, the algorithm joined patient demographics with diagnoses to identify patients 18 to 85 years old who also had a diagnosis of hypertension. Those patient records then were joined with vital sign data to identify the patients with most a recent systolic BP <140 mmHg and a diastolic BP <90 mmHg; these patients were categorized as meeting the measure.
Practice-level performance for each CQM was determined by dividing the number of eligible patients who met the measure by the total number of eligible patients; means and standard deviations across all practices were calculated. In addition, calculated for each patient was the Summary Quality Index (SQUID) CQM,20 the quotient of the number of eligible measures the patient met and the total number of measures for which the patient was eligible. A practice-level SQUID CQM, representing the mean SQUID CQM for all patients in the practice, also was calculated. The practices with the 27 highest SQUID CQMs (approximately the top tertile) were deemed “exemplar” practices.
Provider Survey
Development of the provider survey was guided by 2 theoretical frameworks to systematically ensure that the questions assessed a broad range of organizational factors that could impact EHR implementation. The first was the PPRNet quality improvement (QI) model Improving Primary Care through Health Information Technology23; the second was the Consolidated Framework for Implementation Research.24 Five iterative rounds of development, review, and refinement by the study authors were completed to develop the survey. The final survey included 100 specific questions related to the 21 CQMs addressing provider agreement with the CQMs, staff education, use of EHR reminders, standing orders, and EHR-based patient education and 27 general questions divided into 4 categories, including practices' use of EHR functionality (including stage 1 and 2 MU core and menu requirements), clinical QI strategies, the respondent's beliefs about the value of their EHR and QI activities, and practice administrative QI strategies (Table 2).
The survey was constructed using Research Electronic Data Capture, an online survey tool.25 A pilot test by 7 PPRNet members not participating in the study estimated that the survey took about 15 minutes to complete and suggested a few clarifying edits. The survey was available to respondents between November 6 and December 29, 2013. At least 3 emails, 1 fax, and 1 telephone reminder were sent to practice coordinators and individual clinicians in the practices to ensure an adequate response rate.
Statistical Analyses
Responses to categorical items in the survey were quantified using a scale from 0 to 100. For items with dichotomous responses (ie, yes/no or agree/disagree), a “no” or “disagree” response was coded as 0 and a “yes” or “agree” response was coded as 100. Items with 5 possible responses—that is, “never,” “rarely,” “sometimes,” “mostly,” and “always”—were coded as 0, 25, 50, 75, and 100, respectively. A similar strategy was used for items with 3 possible responses. For practices with more than 1 survey respondent, scores were averaged across respondents to construct a single set of responses for each practice.
Descriptive statistics (means and standard deviations) were used to characterize practices' item response scores, which, by design, could theoretically range from 0 to 100. One set of analyses was used to assess the degree to which survey responses in the 5 specific question categories (provider agreement, staff education, EHR reminders, standing orders, and EHR patient education) were correlated with the practice-level performance for the corresponding CQM. Within these analyses, bivariate (unadjusted) correlations, as well as partial (multivariate-adjusted) correlations were calculated. The partial correlations adjusted for other statistically significant (P < .05) practice covariates (eg, provider demographics, practices' patient demographic and clinical characteristic profiles), which were selected for inclusion in the correlation model only if they met inclusion (P < .1) and exclusion (P > .05) criteria within CQM-specific, stepwise regression models.
A separate set of analyses was used to examine the extent to which practice responses to the general survey items were correlated with their SQUID CQM score. Before calculating these correlations, a stepwise regression model revealed that among a set of patient, provider, and practice characteristics (all expressed at the level of the practice), the average number of chronic conditions among a practice's patients was the strongest independent factor correlated with a practice's SQUID CQM score (ρ = 0.60; P < .0001). This variable was used as a covariate within multivariable regression models, in which the SQUID CQM score served as the dependent variable of interest and each of the general survey item scores served as independent variables of interest in separate models. Partial (adjusted) correlations were calculated from these regression models, and significant (P < .05) correlations were identified. All statistical analyses were conducted using SAS version 9.3 (SAS Institute, Cary, NC).
Focus Groups Among High CQM Performers
To triangulate the survey findings, lead physicians from each of the 27 designated exemplar practices were invited to participate in 1 of 3 focus groups held on consecutive Saturday mornings in geographically separate cities in late January and early February 2014. Standard approaches for conducting focus groups were followed, including obtaining verbal consent to participate and to record the discussions for analysis, emphasizing the importance of each person's response to each question and participation in the general discussion.26,27 The groups were moderated by 2 of the study authors (SMO, LSN) and observed by 2 others (AMW, CBL), who took detailed notes using an observation template to supplement the audio recordings.
Each focus group lasted 2.5 hours. During the first 45 minutes, one of the moderators presented the overall study methods and key findings from the EHR-based CQM performance assessment, the provider survey, and the associations between them. During the remaining 1.75 hours, participants were asked for general perspectives on the overall findings and for detailed perspectives on each of the specific findings. Comments both on their assessment of the validity of each finding and for context-specific examples from their practice that supported their assessment were sought. The process for each of the 3 focus groups was similar, although minor refinements were made to reflect feedback from the first group.
Digital audio files were transferred via secure file servers for professional transcription shortly after each focus group session. Transcripts and field notes were imported into NVivo 10.0 for Windows software (QSR International, Doncaster, Australia) by the primary qualitative analyst (LSN). A second investigator (AMW) independently analyzed the field notes and transcripts. A deductive coding scheme was developed using key concepts and phrases from focus group questions, followed by an inductive process of constant comparison to identify the final themes.28 A microinterlocutor analysis based on the observation template enabled us to identify areas of consensus and variation among participants' responses.29 Extracted themes were reviewed by the 2 coder investigators and minor differences were resolved.30 Review of final themes and field notes by both analysts allowed us to immerse and crystallize the findings.31
Results
A total of 95 PPRNet practices expressed interest in the project through response to one or more email solicitations; 78 practices consented to participate. Two of these practices did not submit data for the CQM assessment and were excluded. Five other practices failed to have the majority of their providers complete the survey and were excluded, resulting in 71 practices from 28 states for the analyses. Among these practices there were 349 providers; 319 (92.1%) completed the provider survey. Among those completing the survey, 73% were physicians, 16% were nurse practitioners, and 11% were physician assistants. A slight majority (51%) were male, and most were white (91%), and non-Hispanic (97%).
Summary data on practice performance for each of the CQMs are presented in Table 1. Performance varied widely among the 21 CQMs. The practice mean for tobacco use screening and counseling was 89.8%, whereas the mean for chlamydia screening was only 12.9%. Large standard deviations on most of the CQMs are indicative of wide variation among practices in measure performance. The mean SQUID CQM across all practices was 37.4%, with a standard deviation of 10.7% and a range from 17.9% to 63.1%.
The 27 practices with the highest SQUID CQM (approximately the top tertile) came from 18 different states and had a SQUID CQM >40%. Lead physicians from 23 of the practices attended a focus group. Five (21.7%) of these physicians were women, 2 (8.7%) were Hispanic, and all were white. Their median age was 56 years. All came from practices with a long history of EHR use: 9 of their practices had been using their EHR for 6 to 10 years, 7 for 11 to 15 years, and 7 for >16 years.
Findings from the specific questions on the provider survey and their association with performance on the 21 CQMs are presented in Table 3. Provider agreement was high for all the CQMs, ranging from 88.5% for depression screening to 100% for pneumococcal immunization for older adults. In the bivariate analyses, provider agreement was associated with higher performance on colorectal cancer screening, avoiding use of high-risk medications in the elderly, and chlamydia screening; however, the association with chlamydia screening did not persist in multivariate analyses.
Providing clinical staff member education on specific quality goals was commonly reported for CQMs, with practice scores ranging from a low of 54.7 for heart failure therapies to a high of 90.6 for influenza immunization. In the bivariate analyses, staff education was associated with performance on 11 of the CQMs; in multivariate analyses associations persisted for breast cancer screening, urine protein screening for diabetes mellitus, use of antithrombotic medication for ischemic vascular disease, and depression screening. Focus group participants endorsed the notion that staff education was critical for improved performance, particularly if the education was directly tied to specific activities the staff could perform, for example, using the health maintenance (HM) reminder features of the EHR to prompt them to obtain urine for protein screening in patients with diabetes. A participant noted that there was “more direct impact when there are clear ways [for staff] to apply the education to actually improve performance.” Another commented: “when the staff understands the clinical importance of measures it makes them more enthusiastic about getting things done and the work is already done when we walk in the door.”
Use of EHR reminders (eg, flags, HM, progress note templates with prompts) as decision support to help meet clinical quality goals was variably reported for the CQMs. For cancer screening (breast, cervical, and colorectal), immunizations (influenza and pneumococcal), and tobacco use screening, practice scores were >80. For appropriate treatment of children with upper respiratory infection and warfarin management, scores were <45. Scores for EHR reminders for other CQMs were between these extremes. In the bivariate analyses, EHR reminder use was associated with higher performance on 12 of the CQMs; in multivariate analyses, associations persisted for the 3 cancer screening CQMs, urine protein screening for diabetes mellitus, heart failure therapies, chlamydia and depression screening, and immunizations. Focus group participants endorsed the strengths of their EHR for flexibility in its ability to display deficient HM items in different ways and during various types of patient contact. In addition to display on the patient summary screen and HM section of the chart, one participant indicated that “HM out of date is pulled into every [progress] note.” Another pointed out that “I can customize and embed reminders. The flexibility allows us to do this.” That EHR reminder use was associated with better performance on CQMs that reflect clinical processes rather than outcomes also was supported by the participants. One explained that “the EHR reminders for things that staff cannot do by themselves, we do not perform nearly as well, mostly because there are more steps”; another noted that “achieving targets is tough; more patient issues play into targets versus process.”
Standing order protocols were used frequently for immunization CQMs but less commonly for other CQMs. In bivariate analyses, use of standing orders was associated with 7 CQMs; however, none of these associations persisted in multivariate analyses. Focus group discussions suggested that the explanation was that standing orders were always mediated through the EHR reminder features (mostly the HM reminders). Indeed, several participants noted that implementing standing orders helped the practice realize significant financial gains, both directly by freeing providers to perform other clinical services and through pay-for-performance programs. One noted: “we are considering adding another [staff] person so that we can have them do more standing orders and that we can do our work.”
Practice scores for use of EHR-based patient education to reinforce population management or public health goals ranged from 38.6 to 64.2, with no discernable pattern in the variability across the CQMs. In bivariate analyses, associations were found between EHR-based patient education and hemoglobin A1C control, urine protein screening for diabetes mellitus, appropriate treatment of children with upper respiratory infection, and depression screening. In multivariate analyses, associations were present only for cervical cancer screening, hemoglobin A1C control, and β-blocker therapy in heart failure. Among focus group participants, the consensus was that the patient education resources embedded in their EHR were not very helpful; the resources were out of date and not tailored to patient needs. A variety of patient education websites were viewed as more current and were more regularly used by participants, some of whom added links to these resources in their EHR templates. Participants argued for a more personalized approach to patient education, with one suggesting, “what should be happening is a care plan with the nurse. The patient should be deciding what would be useful for them.”
Findings from the general questions on the provider survey and their association with performance on the SQUID CQM are presented in Table 2. In the category “Use of EHR Functionality,” practice means ranged from a high of 82.9 for maintaining up-to-date problem lists to a low of 44.8 for the use of a web-based patient portal. A statistically significant correlation was found only between use of registries for population management and the SQUID CQM. A number of the focus group participants indicated that they used the registry functionality provided by PPRNet for CQM population health management, reaching out to patients with care deficiencies. There was, however, no clear consensus that they had observed an impact of these activities on CQM performance. Participants were not surprised that use of patient portals was not associated with the SQUID CQM, noting that the purpose of portals was to improve patient access and likely affected patient satisfaction but not CQM performance. Nor were they surprised that use of after-visit summaries was not associated with performance; one participant noted: “…[summaries] can be confusing to the patient. Knowledge alone does not change behavior.” Others pointed out the lack of actionable, individualized information, and several participants shared the observation that these printed summaries were often left in the office and not taken home.
For the category “Clinical QI Strategies,” practice means ranged from a high of 88.3 for medication reconciliation to 64.9 for follow-up with patients who did not complete recommended services. None of the clinical QI strategies were correlated with the SQUID CQM. Focus group participants supported these findings. Medication reconciliation was noted as not directly tied to the CQMs assessed in this study and might be more likely to impact utilization, safety, or transition of care measures. A participant also noted, “I do not think reconciliation truly correlates with compliance, getting refills.” Follow-up with patients not completing recommended services was thought to be ineffective for several reasons. One participant noted, “patients who are resistant are resistant.” Another acknowledged that simple patient outreach, absent exploration of barriers and problem-solving, was unlikely to be effective in motivating patients to adhere to recommendations. He noted, “we do not necessarily explore barriers. Calling and just saying they need to get it done is not really effective.”
For the category “Beliefs about EHR and QI Activities,” practice means ranged from a high of 72.7 for EHR is helpful in achieving high-quality clinical care to a low of 36.5 for finding assistance from a regional extension center32 helpful in achieving high-quality clinical care. Statistically significant positive correlations were found between “EHR is helpful in achieving high-quality clinical care,” “EHR is customized in practice to facilitate high-quality clinical care,” and “Participation in PPRNet motivating to achieve high-quality clinical care” and the SQUID CQM. A statistically significant negative correlation was found between the belief that the EHR is difficult to use to achieve high-quality clinical care and the SQUID CQM. As noted above, focus group participants endorsed the view that the ability of their practice to customize their note templates and HM reminder features helps to improve performance across a number of CQMs. Participants also noted that PPRNet activities such as reports, network meetings, and site visits helped practices pinpoint areas for improvement and try new approaches to care improvement.
For the category “Practice Administrative QI Strategies,” practice means ranged from a high of 77.3 for “Members working consistently to achieve improvement” to a low of 39.0 for provider incentives. Statistically significant positive correlations were found between practice member knowledge about practice improvement priorities, members evaluating progress together, leaders seeking input from team members, regular staff meetings, and leadership commitment and the SQUID CQM. One participant noted, “The staff is critical. If you just have leadership without your staff behind you, you are in trouble.” Another commented, “Staff education … creates an interest in looking at our QI reports and gives us a context for discussing them as a group and creates this quality culture that makes change possible.” Regarding provider and staff incentives, current incentives were noted to be too small to have much impact, and there was a feeling that their practice teams were inherently motivated to provide high-quality care and did not need incentives to do so.
Discussion
Several pertinent findings emerged from this multimethod study. First, this study of how practices can successfully use their EHRs to achieve improvement in outcomes (“meaningful use” in its optimal sense) is interesting to clinicians. Indeed, a large majority of PPRNet practices with clinicians who had certified for MU by the third quarter of 2013 agreed to participate in the study, and the survey response rate among clinicians in practices completing the study was 92.1%. Second, use of an EHR clearly does not ensure high performance on CQMs, even when the CQMs are widely endorsed by clinicians. The mean practice SQUID CQM for measures supported by almost 90% of clinicians across this group of practices—all of which used an EHR—was only 37.4%. Third, practices are using theoretically sound improvement strategies23 for many CQMs, such as staff education, EHR reminders, standing orders, and EHR-based patient education, though there was great variability in their use among practices and by CQM. Practices also widely reported use of many EHR functions, including those required for stage 1 or 2 MU, and a majority had favorable beliefs about the potential impact of EHR on quality. Fourth, although there was some variability by CQM, and for 8 CQMs no measured strategies were associated with higher performance, use of EHR reminder functions seemed to be the most effective strategy. That a number of bivariate associations between standing orders and staff education did not persist after adjusting for other variables suggests that these interventions were operationalized through the use of EHR reminders. Fifth, the only correlation between the SQUID CQM and use of specific EHR functionality was for registry use for population management. Several requirements for stage 1 and/or 2 MU, including maintaining problem lists, providing after-visit clinical summaries, and patient portal functionalities were not associated with the SQUID CQM. Sixth, a number of factors, including EHR customization, participation in PPRNet, and leadership commitment to improved EHR use and a practice culture of quality, were associated with the SQUID CQM.
This study has several prominent strengths. To our knowledge it is the first study assessing the association between measured CQM performance and organizational factors related to EHR implementation in a broad set of primary care practices. A recent national study suggesting that physicians using EHRs that meet MU criteria, particularly those with longer EHR experience, have improved clinical care was based on survey data, not clinical data,33 as was an earlier study finding no association between EHR use and quality.8 Other studies examining the associations between EHR use and quality have been limited to 1 primary care practice,34 1 state,9,35 and the use of administrative data to assess quality.9,35,36 This study is likely to be more representative because it included a relatively large number of primary care practices, all whose physicians had sufficient enough experiences with EHR to be certified for MU. In addition, the physicians participating in the focus groups had years of experience using an EHR in their practices.
The study also has several notable weaknesses. It reflects findings among users of only 1 EHR. The magnitude of the statistically significant associations between CQM performance and the survey variables were low to modest, indicating that there are other determinants of quality other than those assessed in the study. Although all associations assessed were specified in advance, the large number of them suggests that some of the statistically significant associations may have occurred by chance. As with any study of EHR CQM performance, the findings are dependent on the accuracy of the data in the EHR and the fidelity with which these data were extracted and incorporated into CQMs—an issue noted in a recent report.37 Finally, because one of the frameworks used in the survey design (Improving Primary Care through Health Information Technology) was developed in PPRNet practices, the observed associations may be unique to this population of clinicians.
Conclusion
Despite these limitations, our findings support the conclusion that among clinicians who have certified for MU, organizational factors related to EHR implementation, such as purposeful use of EHR functionality coupled with staff education in a milieu where QI and the EHR are valued and supported, are associated with higher performance on primary care–relevant CQMs. High-quality care requires more than MU certification.
Notes
This article was externally peer reviewed.
Funding: This study was funded by the Agency for Healthcare Research and Quality (grant no. 1R18HS022701). CBL is supported by Agency for Healthcare Research and Quality grant no. K08HS018984. The funding agency had no role in the study design; in the collection, analysis, and interpretation of data; in the writing of the report; and in the decision to submit the paper for publication.
Conflict of interest: none declared.
- Received for publication July 31, 2014.
- Revision received September 29, 2014.
- Accepted for publication October 7, 2014.