Abstract
Background: Practical studies in real-world settings may be particularly vulnerable to unintended effects on intervention outcomes, including what is commonly known as the Hawthorne Effect. This phenomenon suggests that study subjects' behavior or study results are altered by the subjects' awareness that they are being studied or that they received additional attention. This is especially a concern when subjects are not blinded to randomization or when they participate in studies with observational components. As part of a larger practical intervention designed to improve the clinical management of skin and soft tissue infections (SSTIs), we specifically examined the potential for a Hawthorne Effect from the extra attention some clinicians received when completing follow-up case reviews.
Methods: De-identified, electronic data from a larger practical intervention allowed for the comparison of the clinical management of SSTIs among 14 randomly selected clinicians who participated in follow-up case reviews versus 77 clinicians who did not.
Results: There were no differences in the management of SSTIs between the 2 groups of clinicians. No evidence of a Hawthorne Effect was observed in this quality-improvement intervention.
Conclusion: More extensive contact with the research team did not seem to have unintended effects on the outcomes of interest for the management of SSTIs. Further study in practice-based research settings could help to establish whether different types of studies and outcomes are more or less susceptible to the Hawthorne Effect.
- Community-Acquired Infections
- Hawthorne Effect
- Methicillin-Resistant Staphylococcus Aureus
- MRSA
- Practice-based Research
- Primary Health Care
- Soft Tissue Infections
Studies in practice-based research networks (PBRNs) often seek to understand how interventions perform in real-world settings, yet some common limitations may have unintended effects on intervention outcomes. Specifically, studies that are not blinded or studies that have observational components may be vulnerable to the Hawthorne Effect. This phenomenon suggests that study subjects' behavior or study results are altered by the subjects' awareness that they are being studied or if they received additional attention. Though originally observed in the 1920s and 1930s while studying worker productivity at the Hawthorne Works factory in Cicero, Illinois,1 this phenomenon has been extended to the realm of medical research2,3; however, it rarely is quantified as part of intervention research4,5 and is not commonly mentioned among the limitations of PBRN studies. As part of a larger practical intervention designed to improve the clinical management of skin and soft tissue infections (SSTIs), we specifically examined the potential for a Hawthorne Effect from the extra attention some clinicians received when completing follow-up case reviews with the research team.
Methods
From October 2009 to April 2010, a practical intervention was conducted in 16 primary care practices in 2 health care systems. Based on the Centers for Disease Control and Prevention guidelines for the treatment of SSTIs, the intervention consisted of a ready-made kit for incision & drainage (I&D) procedures, methicillin-resistant Staphylococcus aureus (MRSA) information for clinicians, a patient information handout, provider and staff education, and patient follow-up. The intervention materials were made available to all clinicians in the participating practices.
The primary aims were to increase drainage procedures, increase the rate at which abscesses were cultured, and increase antibiotics that covered MRSA when antibiotics were prescribed. A 12-month historic cohort of patients seen for SSTIs was compared with the patients seen in the same 16 practices during the intervention period. Based on International Classification of Diseases, Ninth Revision (ICD-9) coding, cases were divided into likely abscesses (680.x) and cellulitis without or without abscess (681.x and 682.x, respectively); most in this second grouping were cellulitis without abscess, for which a drainage procedure and culture cannot be performed. All outcome data were collected remotely by electronic data extraction and all intervention components were administered by the regular clinic staff.
Results of the intervention showed no significant improvement in the rate of I&D procedures or cultures of abscesses obtained, but results did show increased use of antibiotics overall and antibiotics that typically covered MRSA strains. (Detailed results of the intervention were reported previously.6)
As part of a quality improvement effort related to the intervention, before the start of the intervention, a random sample of 25% of clinicians from all the participating clinics were invited to participate in case reviews to gather more details about the management of specific skin infections (Figure 1).
Case selection for Hawthorne Effect analysis of comparison versus observed clinicians.
Assessment of Hawthorne Effect
The potential Hawthorne Effect on clinician management of SSTIs was assessed by comparing clinicians who participated in case reviews with those who did not, using the outcome data that were extracted from electronic records. In effect, clinicians who participated in the case reviews were being “observed” because they received additional attention and interaction with the research team. Clinicians who did not participate in the case reviews were considered “comparisons.” The case reviews for observed clinicians consisted of completion of a detailed report about a specific, recent case followed by telephone interviews conducted by family physicians on the research team to gather more contextual information and case management details. The clinical management of SSTIs of the 2 clinician groups were compared using χ2 and Fisher exact tests to test for statistical significance. Though the patient data were de-identified, a third party made it possible to compare patient outcome data from both sets of clinicians.
This study was reviewed for human subjects protections and approved by the Colorado Multiple Institutional Review Board and by the American Academy of Family Physicians Institutional Review Board.
Results
A total of 91 primary care clinicians were included from 2 health systems (one in Texas and one in North Carolina); each saw at least one patient with an SSTI during a 6- to 7-month intervention period.
Approximately 15% of the clinicians ultimately participated in the quality improvement follow-up reviews of selected SSTI cases, completing both follow-up case reports and telephone interviews (observed clinicians). The other 85% of clinicians were the comparison clinicians. Among the clinicians who saw patients for abscesses during the intervention period, we did not find evidence suggestive of a Hawthorne Effect related to the prescription of antibiotics or in the selection of antibiotics that cover MRSA (2 outcome measures in which a Hawthorne Effect might be especially expected; see Table 1).
Comparison of Clinical Management of Skin and Soft Tissue Infections Between Observed and Comparison Clinicians
Discussion
Clinicians who participated in the case reviews received extra attention from the research staff, who conducted interviews asking detailed questions to understand some of the clinical decision making that goes into the management of the SSTIs. The case reviews also asked specifically about key components of the intervention including the I&D kits, provider guidelines, and patient guidelines. Despite this level of direct contact with the research team, we observed no significantly higher rates of drainage procedures, cultured abscesses, and prescription of antibiotics that covered MRSA. It is possible that a Hawthorne Effect could have been missed because of the small number of cases in the sample. Among the larger number of cellulitis cases, we also observed no statistically significant differences in the prescribing of antibiotics. Though this was a limited examination of the Hawthorne Effect, primary care researchers engaged in similar work should find some encouragement that additional contact with researchers collecting observational data did not seem to influence significantly clinician behavior in their management of SSTIs.
Conclusion
For this practical trial in frontline primary care practices, more extensive contact with the research team did not seem to have unintended effects on the outcomes of interest for the management of SSTIs. Although it is difficult to generalize our findings to other PBRN projects, further study in practice-based research settings could help to establish whether different types of studies and outcomes are more or less susceptible to the Hawthorne Effect.
Notes
This article was externally peer reviewed.
Funding: This project was funded under contract no. HHSA290 2007 10008, task order no. 4 from the Agency for Healthcare Research and Quality, US Department of Health and Human Services, as part of the Primary Care Practice-Based Research Network Master Contract.
-
Conflict of interest: none declared.
-
Disclaimer: The authors of this report are responsible for its content. Statements in the report should not be construed as endorsement by the Agency for Healthcare Research and Quality or the US Department of Health and Human Services.
- Received for publication January 18, 2011.
- Revision received July 6, 2011.
- Accepted for publication July 12, 2011.
In this issue
Jump to section
Related Articles
Cited By...
- Effects of an mHealth intervention for community health workers on maternal and child nutrition and health service delivery in India: protocol for a quasi-experimental mixed-methods evaluation
- Dexamethasone plus Somatostatin-analog Manipulation as Bone Metastasis Microenvironment-targeting Therapy for the Treatment of Castration-resistant Prostate Cancer: A Meta-analysis of Uncontrolled Studies
- Answers to Common Clinical Questions