Abstract
Introduction: Patient experience, 1 of 3 aims for improving health care, is rarely included in studies of patient-centered medical home (PCMH) transformation. This study examines the association between patient experience and National Committee on Quality Assurance (NCQA) PCMH transformation.
Methods: This was a cross-sectional study of parent-reported child patient experience from PCMH and non-PCMH practices. It used randomly sampled experience surveys completed by 2599 patients at 29 pediatric and family medicine PCMH (n = 21) and non-PCMH (n = 8) practices in Vermont from 2011 to 2013. Patient experiences related to child development and prevention were assessed using the Consumer Assessment of Health care Providers and Systems (CAHPS).
Results: A 10-point increase in NCQA score at PCMH practices is associated with a 3.1% higher CAHPS child prevention score (P = .004). Among pediatric practices, PCMH recognition is associated with 7.7% (P < .0005) and 7.2% (P < .0005) higher CAHPS child development and prevention composite scores, respectively. Among family medicine practices, PCMH recognition is associated with 7.4% (P = .001) and 11.0% (P < .0005) lower CAHPS child development and prevention composite scores, respectively.
Conclusions: Our results suggest that PCMH recognition may improve child patient experience at pediatric practices and worsen experience at family medicine practices. These findings warrant further investigation into the differential influence of NCQA PCMH transformation on family medicine and pediatric practices.
The term medical home has grown to describe primary health care that is patient-centered, comprehensive, coordinated, accessible, committed to quality and safety, and delivered in the context of family and community. Nearly 8000 patient-centered medical homes (PCMHs) are recognized by the National Committee for Quality Assurance (NCQA) across the United States, with the hope that these transformations will improve the triple aim dimensions of health outcomes, cost, and patient experience.1 The evidence for association of the PCMH model with improvements on the triple aim dimensions has been mixed; early reviews indicate modest positive, inconclusive, and negative impacts on these dimensions,2⇓–4 coming mostly from demonstration projects with limited payers, little financial support, and inconsistent use of NCQA's PCMH framework.5,6
In a recent pilot study evaluating multipayer PCMH initiatives involving substantial financial support, researchers found no cost savings and modest improvement in health outcomes.7 That study did not assess patient experience data and therefore did not fully address the triple aim. In our study, patient experience is measured at practices within the context of a state-led, financially supported, multipayer health care reform program in Vermont.
Significant improvements in the measurement of adult and child patient experience have been made over the years. The Consumer Assessment of Health Care Providers and Systems (CAHPS) survey for adult and child patients has expanded to include questions to better assess patients' experiences with PCMHs.8 Results from the CAHPS survey may provide insights into whether consumer perceptions of health care are influenced by NCQA score or are different among PCMH and non-PCMH practices. A recent review of 8 published studies found a small, positive effect of PCMH recognition on patient experience; however, only 3 studies included children,3 and 1 of those found no effect on patient experience.9 Therefore, results are mixed, and more research on child patient experiences is needed to inform the next steps in the transformation of health care systems.
The primary objective of this study is to evaluate differences in CAHPS child patient experience outcomes in relation to 2 research questions: (1) Are higher NCQA scores associated with better patient experience? (2) Is PCMH recognition associated with better patient experience? While both family medicine and pediatric practices provide quality primary health care for children, there may be practice-level contextual differences in their approach to the NCQA PCMH process. Practice-level changes made in family medicine to meet NCQA standards may be different from those made at pediatric practices, and those differences may affect child patients' experiences. Though modified in 2011 to be more relevant to pediatric practices, NCQA PCMH standards focused on conditions and processes that are less common in pediatric practice.10 As such, a secondary objective of this study is to test whether type of practice (pediatric vs family medicine) moderates the associations tested in our primary objective.
Methods
Study Sample
The Vermont Child Health Improvement Program (VCHIP) contacted all primary practices serving children in Vermont and offered participation in CAHPS, and 29 practices volunteered to participate: 21 PCMHs and 8 non-PCMHs at the time the CAHPS surveys were administered. The sample for our first research question (NCQA score associated with CAHPS scores) started with 21 practices first completing the NCQA PCMH process and later voluntarily participating in CAHPS. Three practices scoring below the 2011 NCQA standards were excluded because NCQA scores from 2011 and 2008 were not comparable because of programmatic changes in NCQA PCMH standards, leaving 18 practices (5 pediatric and 13 family medicine) with 1577 completed CAHPS surveys for our first research question.
The sample for the second research question (PCMH recognition associated with CAHPS scores) included all of the initial 21 PCMH practices (scoring below both 2008 and 2011 standards; 7 pediatric, 14 family medicine) and added 8 non-PCMH practices as the comparison group (4 pediatric, 4 family medicine). Practices were selected for the non-PCMH comparison group if they later became PCMHs (after 2013); practices that never became PCMHs were excluded. This decision was made to minimize potential bias related to practices that chose never to become PCMHs. Table 1 reports the number of completed CAHPS surveys for the different variables for PCMH, non-PCMH, family medicine, and pediatric practices.
PCMH Recognition
In 2003, a statewide health care reform program was launched as a governor's initiative, including offering support for NCQA PCMH recognition11 to all primary care practices. The NCQA PCMH scoring program was designed as a team approach, with evaluators from VCHIP at the University of Vermont assisting practice staff to assess policies and processes, ensure accurate documentation, review patient medical records, and submit annotated survey materials to NCQA. Having a single evaluation team helping all primary care practices was a unique environmental contextual factor.
Practices were recognized using NCQA's 2008 or 2011 PCMH standards, which introduced programmatic differences in the requirements for recognition.12 On the 2008 NCQA survey, scores ranged from 0 to 100, and to be recognized as a PCMH, practices needed to earn at least 25 points and achieve at least 50% of the attainable points on a minimum of 5 of 10 essential “must-pass” elements. On the 2011 NCQA survey, scores also ranged from 0 to 100, but practices were required to earn a minimum of 35 points and achieve at least 50% of the attainable points on each of 6 “must-pass” elements to be recognized. Because of these differences, NCQA score data were not combined across NCQA standard years; therefore our first research question focuses on NCQA scores from only practices scoring using the 2008 standards.
Patient Experience Survey
The CAHPS survey was developed jointly by the NCQA, the Agency for Health care Research and Quality, and the Commonwealth Fund. This survey is a measure of patient experience, going beyond satisfaction8 to assess experience with certain aspects of pediatric clinical care. While there is almost no published psychometric testing of the CAHPS, one study reported the reliability of the child-specific scales to be good and the convergent validity to be much higher than the discriminant validity, as expected, yet goodness-of-fit statistics were outside of the acceptable range to claim construct validity of all scales.13
In this study, our team administered the CAHPS Clinician & Group Expanded 12-Month Child Primary Care Survey with PCMH Items (hereafter referred to as “CAHPS survey”) in 2 cross-sectional waves, first in 2011 and again in 2013. There were 6 standard composite scores on this CAHPS survey; of these 6, this study focused on the 2 child-specific composites for “child's growth and development” and “child's health and safety.” The child's growth and development composite measure (hereafter referred to as “child development”) included 6 yes/no questions: “In the past 12 months, did you and anyone in this provider's office talk about …” (1) your child's learning ability; (2) the kinds of behaviors that are normal for your child at this age; (3) how your child's body is growing; (4) your child's moods and emotions; (5) how much time your child spends on a computer and in front of a TV; and (6) how your child gets along with others. The “child's health and safety” composite measure (hereafter referred to as “child prevention”) included 5 yes/no questions: “In the past 12 months, did you and anyone in this provider's office talk about …” (1) things you can do to keep your child from getting injured; (2) how much or what kind of food your child eats; (3) how much or what kind of exercise your child gets; and (4) whether there are any problems in your household that might affect your child; and, finally, (5) “In the past 12 months, did anyone in this provider's office give you information about how to keep your child from getting injured.”
Each composite score was calculated with the number of questions with a “yes” answer in the numerator divided by the total number of questions in the composite, multiplied by 100, for a final composite score representing the percentage answering yes. The percentage answering yes to the CAHPS survey child development or child prevention composite measures are our 2 outcome measures, and the higher the percentage, the more positive the experience of care.
Statewide efforts were made to inform pediatric and family medicine practices about the free opportunity to participate in the CAHPS survey. Parents and guardians of patients <18 years old from participating practices were eligible to receive the CAHPS survey if their child had at least 1 visit with a primary care provider at the practice during the previous 12 months. Random samples were drawn from the population of eligible patients based on the number of primary care providers in each practice. All data were collected by VCHIP, a certified vendor of the CAHPS survey, following the 2012 CAHPS sampling protocol for mail-based surveys.
Surveys were addressed to the parent or caretaker of the child, and survey questions were geared toward parents/caretakers. Samples were de-duplicated using the patients' contact information so that no more than 1 patient per household was selected. To be eligible, a patient had to have ≥1 visit during the measurement period. The surveys were administered in 2 mailing waves. Participants had the option of completing the printed or Web-based survey, and a follow-up mailing was sent to everyone 3 weeks later to improve response rates.
The sample size for each practice was based on the number of providers and an assumed response rate of 35%, as recommended by the 2012 CAHPS sampling protocol. For example, if there was 1 provider, then we sampled 128 patients to obtain 45 completed surveys; if there were 4 to 9 providers, we sampled 343 patients to obtain 120 completed surveys. If a practice's child population had fewer patients than the required sample size, the group was excluded from participation. A total of 55,508 eligible patients were reported by 29 participating practices, and based on the CAHPS sampling protocol described above, we mailed 10,726 surveys to randomly selected patients (19.3% of the eligible population). Our overall response rate was 25.5% (response rates based on PCMH and type of practice are included in Table 1).
Data Analysis
Linear mixed (multilevel) models in SPSS Statistics version 22 (IBM, Chicago, IL) were used to account for the correlation of self-reported measures from patients at the same practice. A variance components covariance structure was used to account for this correlation in all outcome models. This structure yielded the lowest Akaike information criterion in most instances. This covariance structure also resulted in models that converged without errors and supported our data as having nonconstant variances and independent confounding variables. Results and inferences from other covariance structures were similar, and using a variance component covariance structure yielded the most conservative results with slightly smaller effects.
The conceptual framework that guided our research design was based on the emerging evidence that PCMHs can have a positive influence on health care experience.3 A conceptual diagram (Figure 1) linked our predictor (PCMH) with our outcomes (child development and child prevention experience) and included the confounding and moderating variables tested in our models. Separate models were run for the 2 CAHPS composites for each research question: (1) NCQA score predicting CAHPS and (2) PCMH recognition predicting CAHPS, for a total of 4 models. Potential confounders included practice type (pediatric vs family medicine), patient age, patient sex, adult respondent's education (categorized as college graduate vs less), and an overall rating of patient health (on a 5-point scale, with the lowest value indicating excellent health and the highest value indicating poor health). Race was not included because our sample was 97% white. Practice type was tested as a moderator by including an interaction term between type of practice and (1) NCQA score or (2) PCMH recognition. Models were stratified by practice type when interaction terms were significant. The β coefficients from the adjusted regression outputs are interpreted as the mean difference in the composite scores as a percentage for a 1-unit change in each predictor variable, adjusting for all other variables in the model. A positive value indicates that respondents answered yes to more questions as the value of the predictor increased. Results were considered significant at P < .05.
This conceptual framework links the patient-centered medical home as the main predictor of child patient experience and includes important patient-level characteristics and practice type tested as both a confounder and as a moderator.
This study was approved by the institutional review board at the University of Vermont.
Results
NCQA Score
Among PCMH practices, every 10-point increase in NCQA score results in a 3.1% higher CAHPS child prevention composite score, controlling for potential confounders (95% confidence interval [CI], 1.0–5.1). Although in the same positive direction, an increase in NCQA score is not significantly associated with the CAHPS child development composite score when controlling for potential confounders (mean difference, 1.3%; 95% CI, −0.8 to 3.4). Type of practice is not a moderator in the association between NCQA score and CAHPS. Practice type, however, is significantly associated with CAHPS child composite measures: pediatric practices have 14.2% and 16.8% higher child development and child prevention composite scores, respectively, when controlling for NCQA score and other confounders (P < .0005).
PCMH Recognition
Practice and patient characteristics by PCMH recognition are summarized in Table 1. In simple comparisons, CAHPS composite scores are not different between PCMH and non-PCMH practices (P > .05). Our first set of models looks at PCMH recognition as the main predictor of CAHPS composite scores. There is no difference in the CAHPS child development or child prevention scores between our PCMH and non-PCMH groups when controlling for confounders (Table 2). Pediatric practices, older patient age, and college-educated parents are all significantly associated with higher CAHPS PCMH child development and child prevention scores, and better overall patient health is also significantly associated with higher CAHPS child prevention scores (Table 2). Pediatric practices have 9.8% and 11.5% higher child development and child prevention scores, respectively, compared with family medicine practices when controlling for PCMH recognition and other confounders (Table 2).
Practice type is found to be a significant moderator (P < .0005) of the association between PCMH recognition and CAHPS child development and child prevention composite scores. Analyses stratified by practice type reveal that, among pediatric practices only, PCMH recognition is associated with a >7% increase in CAHPS child development and child prevention scores, respectively, when controlling for confounders (Table 3). Among family medicine practices only, the associations between PCMH recognition and CAHPS child development and child prevention scores are also significant and of similar magnitude but in the opposite (negative) direction (Table 3).
Discussion
In this era of primary care transformation and patient-centered outcomes research, it is critical to measure patient experience. This study tests the association between NCQA score and patient experience, as well as PCMH recognition and patient experience; it does not test whether our measure for patient experience, differences in CAHPS composite scores, are specifically associated with differences in the quality of clinical care. In our first research question we found that a higher average NCQA score is associated with better patient experience with child prevention efforts, regardless of practice type. Although not significant, the CAHPS child development score is also trending in the positive direction, and with Vermont practices scoring higher on the second round of NCQA scoring, we expect patient experience to improve. For our second research question, we found that practice type is a significant confounder and a significant moderator of the association between PCMH recognition and patient experience. Within pediatric practices only, recognition as a PCMH is associated with better patient experience on CAHPS child composite measures.
Our results support study findings of a positive impact on patient experience among children with special health care needs who were part of a PCMH14,15 and other child patient samples.16 Together, these findings suggest that the PCMH model may hold promise for improving patient experience even if there is still a question of whether there are improvements in quality or cost over the short term.3 Surprising and a bit concerning is our finding that the experience of patients from family medicine PCMH practices is lower (worse) than that of patients from non-PCMH practices.
There are several external environment contextual factors17 related to practice transformation in Vermont. The study period encompasses the early stages of complex delivery system reforms across the state. Vermont's Blueprint for Health, a state-led program, supports PCMHs by direct investments in multidisciplinary community health teams, local leadership, and a locally organized transformation and self-management infrastructure.18,19 In each health service area, primary care providers work with other local providers to organize community health teams whose staff work in PCMHs to provide patients and families with better access to more complete health services (medical and nonmedical).
Integrating community health team staff into the practice workflow takes time, even with substantial support through insurer payments and the investment of state government in leadership, technology, and programmatic infrastructure supported by grants to each health service area. The measurement of patient experience occurs during the early stages of these structural, programmatic, and cultural changes. A longer observation period may help to more fully understand the relationship between practice transformation and patient experience, accounting for potentially important factors such as practice type as well as the influence of improved access to an extended network of community services.
There are also several practice-level contextual factors17 related to practice transformation in Vermont. In collaboration with these state-led practice transformation efforts, VCHIP also had a Children's Health Insurance Program Reauthorization Act quality demonstration grant to help accelerate the pediatric practices' transformation to PCMH. VCHIP funds (from the Children's Health Insurance Program Reauthorization Act grant) supported pediatric-informed facilitation to guide the pediatric practices through the NCQA PCMH recognition process. The pediatric-informed facilitators provided practices with suggestions for conditions to focus on and support during the NCQA PCMH recognition process. This knowledge of which child conditions work well within the NCQA process likely helped practices. While we expect that the pediatric-informed facilitators are a positive benefit, we are unable to directly measure the impact on patient experience other than the designation of practice type.
Another difference between the pediatric and family medicine practices is that family medicine practices focus primarily on adult conditions during the NCQA PCMH process. This choice may be related to the components of the NCQA process fitting well with managing adult chronic conditions. Given the fact that NCQA PCMH standards focused on conditions and processes that are less common among pediatric patients,10 and the Vermont Blueprint for Health practice payments were based on NCQA PCMH level attained, family medicine may have chosen to focus on adult conditions to receive higher payments. Another potential reason for this choice may be related to the national trend demonstrating that family physicians have been seeing fewer child patients over the past 3 decades.20⇓–22 This is of particular interest in Vermont given its large rural population and the usual reliance of such populations on family medicine. A study of family medicine practices focusing on child conditions compared with family practices focusing on adult conditions is needed to help tease apart this effect of practice type. Depending on workforce trends and results from future studies, the NCQA may consider requiring family medicine practices with a certain proportion of child patients to focus on at least 1 condition within their child population during their recognition process.
While our findings of better patient experience with PCMH recognition are encouraging, the observational nature of our data limits our ability to explore possible causal associations. We are limited to parent-reported experience since, for research purposes, most children seen in primary are too young to report their own experience.23 However, insofar as parents are the primary agents for their children's health care, it can be argued in any case that it is both legitimate and important to measure the parent experience. Another limitation is that 5 of our family medicine and 1 of our pediatric practices are level 1 for their first round of NCQA PCMH recognition. Although these are few compared with our total number of practices, other studies have received similar criticism.7
Looking further into our data, we are not able to test the influence of each NCQA level on our outcomes because we have only 1 level 2 practice, and, fortunately, the remaining are level 3. In Vermont, practice payments are linked to NCQA scores, with higher scores resulting in higher payments; although not reported in this manuscript, almost all level 1 and level 2 practices have moved to a higher NCQA level in the second round, and none have moved lower. Therefore, we expect to see further improvements in patient experience. While we were able to include many important confounders, we did not have data on other contextual factors, such as practice size, which may be associated with practices' struggles to meet the requirements for PCMH recognition.24 The generalizability of our Vermont findings to urban and nonwhite populations is questionable, yet they are generalizable to more rural areas of the country with similar populations and where pediatric and family medicine practices with a large mix of age ranges exist in sufficient numbers to provide a choice to patients and families.
Conclusions
In this study we assess the impact of higher NCQA scores and PCMH recognition on parent-reported child patient experience. Although we hypothesize that both will result in better patient experience, higher NCQA scores result in better experience related to only child prevention efforts, and PCMH recognition is associated with better experience for only pediatric practices. The fact that pediatric practices have the benefit of a pediatric-specific facilitator, and family medicine practices focus primarily on adult conditions during their PCMH transformation, has implications for other states, as well as for changes to the NCQA requirements for certification. Accreditors and states should consider the importance of quality improvement efforts focused on pediatric conditions at family medicine practices that see substantial numbers of children. Additional pediatric-informed resources, such as facilitators, need to be supported in practice-based quality improvement efforts in the hope of improving the experience of the child patients.
Notes
This article was externally peer reviewed.
Funding: This study was funded by a Children's Health Insurance Program Reauthorization Act (CHIPRA) Quality Demonstration Grant.
Conflict of interest: none declared.
- Received for publication February 15, 2015.
- Revision received July 29, 2015.
- Accepted for publication August 17, 2015.
