Lost In Translation: The Value of Qualitative Data

At first glance, it would be easy to relegate the study by Backer et al,1 published in this issue of the Journal, to the growing inventory of articles reporting less than fully successful attempts to improve the implementation of effective clinical services in practice. The major goal of the Every Woman Matters study—to increase documentation of mammograms and Papanicolaou smears among eligible women seen in primary care practice—was achieved in less than a third of the practices in which the chosen intervention was introduced, and the observed increases reached statistical significance (compared with baseline) in only a single practice. There are a number of reasons, however, why Backer’s work should not be so quickly dismissed.

At the top of the list is the rich qualitative data included in the report, which describes the process of change (or resistance to change) that occurred in each of the practices. Of particular interest are the insights this information provides into practices that were unable to accomplish the goals they had set for themselves. The descriptions of the initial conditions in these practices—including the values of the practice leaders, the relationships among clinicians and staff, and methods of responding to the often chaotic health care environment—adds greatly to our understanding of why practice change in this country is often so difficult. Such information is all too frequently not collected or not included in published reports of more quantitative research. One wonders what valuable new knowledge about practice change might have been lost over the past 2 decades by the numerous investigators who, in clinical trials, carefully observed whether a given intervention succeeded or failed in changing physician practices but apparently took little notice of how these results might have been affected by the unique situations faced by the physicians enrolled in the study. If the Every Woman Matters study can be criticized for anything, it is that the observations of contextual factors were not even more detailed. It would have been valuable to know, for example, the extent to which each practice had access to computerized patient data (eg, patient registries or reminders) and how comfortable physicians and staff felt in using computers in the everyday delivery of preventive care.

Backer et al had good reason to believe that, in most practice settings, their intervention would significantly increase the rates of up-to-date preventive services. Previous research has shown that interventions focused on the individual physician are unlikely to be effective, and that interventions that rely on more than one method appear to be the most successful.2 Their intervention was multifaceted and targeted the entire practice team. It was based on the GAPS model, which calls for providing practices with individualized assistance in identifying and reducing barriers to quality improvement. This method has in the past been shown effective in increasing rates of preventive services delivery.3 In addition, the Every Woman Matters intervention had much in common with the more recently described STEP-UP approach, in which tailored interventions are applied to meet each practice’s unique and dynamic needs. In a recent study, implementation of the STEP-UP approach was shown to result in sustained increases (over 24 months) in the rate of delivery of preventive services in primary care practices.4

The odds, however, were against the Every Woman Matters study producing similarly positive quantitative results. The STEP-UP study involved 79 practices, 39 of which were randomized to the intervention; the Every Woman Matters sample was limited to 7 practices. The larger sample size allowed the STEP-UP study to demonstrate overall effectiveness despite the fact that the intervention’s effect was inconsistent. The impact was large in some practices; in others, it resulted in no change or even a decrease in services delivered. The failure of Backer’s study to demonstrate similarly positive overall results could simply be due to an inadequate sample of practices. The less positive results could also be related to the fact that the study practices were randomly selected from a list of those participating in the state-run, federally funded Every Woman Matters program, whereas practices volunteered to participate in the STEP-UP study and thus may have entered that study with a greater readiness for change. Before it is written off, the Every Woman Matters intervention deserves to be tested in a larger number of primary care practices. To determine just how widely and easily the intervention (if proven effective) could be disseminated, the investigators should collect not only qualitative contextual data about the practices but also information about the cost of implementation.

A decade of studies on how to translate research into practice has taught us, if nothing else, that bringing about change in practice will not be simple. The primary care practice is a much more complex system than most of us originally imagined, and there appears to be great diversity in the specific features of individual practice settings.5 Future research must not only look for innovative ways of effecting change in a wide configuration of practices. It also needs to take an in-depth look at primary care practices over time and to account for the effects on practice of the rapidly evolving health care environment. Data, and particularly qualitative data, that helps us understand practice and find better ways of facilitating positive change in practice must not be lost in the process of studying translation but instead be routinely collected as part of the research effort. This information will inevitably provide important lessons for improving the quality of care provided to our patients.

• Received for publication July 1, 2005.