Abstract
Background: ClinfoTracker is a prompt and reminder system designed as a platform for the study of the cognitive aspects of prompting in primary care. We examined the impact of ClinfoTracker in three practices and its potential for studying primary care clinicians’ use of prompt and reminder systems.
Methods: We studied ClinfoTracker in three practices using an observational design. We measured performance rates of three services during the course of 5 years in one practice and prompt response rates in all three practices for 5 months after implementation. We performed qualitative cognitive task analyses of ClinfoTracker use.
Results: Five-year compliance rates increased for fecal occult blood testing (1996–2000, 22.5%, P = .0037; 1997–2000, 23.5%, P = .0024) and type 2 diabetes retinopathy testing (1997–2000, 15.7%, P = .0004; 1998–2000, 16.9%, P = .0001; 1999–2000, 12.4%, P = .0048). ClinfoTracker response rates from the three practices showed significant increases during 5 months (P < .0001). Significant differences seen between practice response rates indicate variable success in ClinfoTracker implementation. Cognitive task analyses performed at site 1 indicate differences in how ClinfoTracker prompts are incorporated into practice.
Conclusions: Improvements in preventive and disease management services after implementation show the potential of ClinfoTracker. Differences in implementation between practice sites and observations of individual clinicians show the promise ClinfoTracker holds for studying clinician use of prompt and reminder systems.
Continuity and comprehensiveness are emphasized in the definition and role of primary care as delineated in the Institute of Medicine report of 1996.1 Starfield and her colleagues2–4 have published on the positive effect of a healthy primary care sector on global measures of health at the national and state level. The continuity and comprehensiveness of primary care, however, are threatened in the United States, by calls to carve out management of certain chronic diseases and assign this function to specialty-based disease management programs. Studies of guideline adherence and the delivery of key evidence-based services (eg, β-blockers for patients after myocardial infarction, diagnosis and treatment of depression) regularly show poorer performance rates for primary care than for specialty-based disease management.5–7
Clearly, if the aspect of comprehensiveness is to remain at the core of primary care, methods must be found to address this issue of guideline adherence. Yet the very quality of comprehensiveness makes guideline adherence difficult in the primary care setting. Comprehensive care by definition means that no health care complaint is out of bounds in the primary care office and, therefore, sets up an environment of competing demands in which attention to any single disease-specific guideline can be superseded by an acute complaint or chronic problem exacerbation in another area.8,9 An illustrative example is the tendency of primary care clinicians to be distracted by issues of chronic illness or acute problems regardless of the severity of depression symptoms.10 The benefits of comprehensiveness and continuity could be maintained if the consistent performance of key services, ie, consistent application of guidelines, could be achieved in primary care.
Several groups have sought to apply prompt and reminder strategies to this problem,11,12 and there is enticing evidence that prompted general practice can achieve guideline adherence rates and results equal to those of specialty clinics.13 These strategies, however, have not achieved widespread use.
Two characteristics of previous primary care prompt and reminder systems have limited their dissemination. First, successful prompt and reminder systems have typically been built either around a specific electronic medical record infrastructure14,15 that was not widely adopted or around the logistics of a single practice or institution.16 Second, prompt and reminder systems are typically designed from an information systems rather than a human factors engineering or cognitive science perspective.
We believe that for a prompt and reminder system to support high-quality primary care, it must be built on a sound understanding of the cognitive tasks of preventive and disease management services in primary care. Unlike the cognitive work of specialists, where the content of each encounter is narrowly defined, in primary care not even a patient’s medical history suffices to define fully the potential content of any given encounter. An appointment originally scheduled for hypertension follow-up can easily become a visit for acute bronchitis. Because of its basis on a narrowly defined set of clinical tasks, an aircraft checklist approach to prompting is doomed to failure. In effect, primary care clinicians not only cannot predict the flying weather, they often do not know what plane they will be flying.
We have constructed a deliberately generic prompt and reminder system as an infrastructure for research on the cognitive tasks of preventive and disease management service delivery in primary care. This system is called ClinfoTracker and has been implemented in four primary care practices to date.
ClinfoTracker Characteristics
The core concepts that have guided the development of ClinfoTracker are (1) prompting only when appropriate and (2) integration of clinician input into the prompting process. Prompting only when appropriate requires that the ClinfoTracker prompts and reminders are driven by patient demographics, a clinician-verified and clinically relevant problem list, and a regularly updated database of services already performed. Prompting only when appropriate is essential to the long-term sustainability of any system. Prompts or reminders that are delivered when they are inappropriate create a level of noise that requires a busy clinician to work harder to distinguish the appropriate prompts. If this noise reaches a high enough level, clinicians begin to ignore all information provided by the system.
The integration of clinician input into the prompting process is achieved in several ways. The prompts and reminders that are offered by ClinfoTracker can be modified by each individual practice. Each practice, therefore, can set standards on which preventive or disease management services they wish to have ClinfoTracker monitor. Clinicians have a great deal of flexibility in their responses to the ClinfoTracker encounter forms. Clinicians can indicate when action on a prompt or reminder is inappropriate or should be deferred. Clinicians can also verify and update the ClinfoTracker problem lists each time they see a patient. This high level of clinician input into the prompt and reminder system helps to ensure that the clinicians are involved in the process of the prompt generation. Frame17 has emphasized the importance of process beyond the simple use of a tool in implementing prevention strategies in a practice. Clinician input serves to enhance the acceptability of the prompt and reminder system and promotes its sustainability.
To ensure that problem lists track clinical issues at an appropriate level of specificity, we use the International Classification of Primary Care (ICPC).18,19 ICPC contains a rich set of symptom-level terms as well as diagnostic terms that are common to primary care. As such, ICPC permits the recording and tracking of problems at a level of specificity appropriate for primary care and permits the tracking of episodes of care for individual problems.
As described above, a common limitation of computerized prompt and reminder systems is their design around a specific set of software. Recognizing this limitation, the goal behind the development of ClinfoTracker was to design the system around a set of concepts rather than as a software package. As such, ClinfoTracker is built with an open-systems design that can easily be implemented using any database compatible with structured query language (SQL). Clinicians can receive prompts from the system and give information back to the system using a single sheet of paper (Figure 1) or a Web form or direct SQL queries from an electronic record system. The system requires a single computer capable of running an SQL-compatible database, a means of inputting data from appointment and laboratory systems, and a means of printing encounter documents that contain the problem list and prompts specific to an encounter. As a result, ClinfoTracker can be implemented in a clinical environment working either beside an existing electronic record system or not.
ClinfoTracker sample encounter form.
This article describes our initial implementation of ClinfoTracker and the information we have already gleaned about the cognitive task of preventive and disease management services in primary care.
Methods
To date, ClinfoTracker has been implemented in three practices of the Department of Family Medicine at the University of Michigan and one internal medicine practice in a small community near Ann Arbor that is affiliated with the University of Michigan. We sought to evaluate our implementation of ClinfoTracker using two quantitative and one qualitative observational methods. First, we obtained chart review data from one practice on compliance with preventive services recommendations for preventive mammography, fecal occult blood testing, and referral for diabetic retinopathy screening from 1996 through the first half of 2000. Second, we examined monthly trends for prompts and responses across all practices where ClinfoTracker had been implemented. Third, we performed a set of cognitive task analyses to evaluate how clinicians were integrating the prompts from ClinfoTracker into their daily patient care routines.
Chart Reviews
Chart reviews were completed on 605 adult patient records, chosen because they were candidates for the prompted services at site 1. Site 1 is a faculty-only clinical practice site for the Department of Family Medicine at the University of Michigan. ClinfoTracker was implemented at this site in phases beginning in 1998, with all clinicians participating beginning in May of 2000. The records were chosen at random and were sampled from January 1996 through June 2000. For purposes of this comparison, only patients who had been with the practice for the entire study period were included, which left 470 patients eligible for analysis.
For each of the three preventive services examined (preventive mammography, fecal occult blood testing, and diabetic retinopathy screening), patients were considered to be in compliance if the preventive service had been completed within recommended intervals. For example, if a diabetic patient had retinopathy screening completed in 1996, 1997, 1998, and 2000, that patient would be considered to have an 80% compliance score. It should be noted that because chart review data were available only for the first half of 2000, compliance rates for 2000 were annualized. Repeated measures analysis of variance (ANOVA) with Bonferroni adjustment for multiple comparisons were used to evaluate differences between compliance rates during the 4-year period.
ClinfoTracker System Data
System logs were queried for encounter documents with prompts for preventive mammography, fecal occult blood testing, and diabetic retinopathy screening generated during the months of May through October of 2000 for three of the four practices in which ClinfoTracker was being implemented. We excluded one of the four practices because only one clinician at that site was using ClinfoTracker. As described above, site 1 is a University of Michigan faculty family practice in Ann Arbor with 17 clinicians. Site 2 is another University of Michigan faculty family practice in a small town outside Ann Arbor with 7 clinicians. Site 3 is a nonacademic internal medicine practice managed by the University of Michigan with 4 clinicians in a small community 15 miles north of Ann Arbor.
For purposes of our comparisons, we examined daily clinician response rates by rule and by clinic. A rule is the issuance of a prompt at a specific interval for patients meeting the candidacy conditions and not having already received the service within the interval. Because we were interested in the level of clinician’s use of the system, we counted responses not only when services were ordered, but also when clinicians marked a prompted service as deferred by the clinician, decided against by the patient, or already performed at a site not affiliated with the University of Michigan. Mean response rates were calculated for each month to allow repeated measures ANOVA with Bonferroni adjustments for multiple comparisons across all 8 months of the study period.
Cognitive Task Analysis
Cognitive task analysis20 is a highly structured qualitative interview method used in decision-making research, particularly in US Naval ship command, military aviation, firefighting, and similar tasks requiring high-level expertise and judgment under conditions of time pressure, uncertainty, and high stakes. It is a method designed specifically to gain an understanding of the decision making processes of experts in real-world production environments, which differ in fundamental ways from those of the more often studied novices (eg, residents and interns) and laboratory or paper-and-pencil decision tasks. Specifically, expert decision behavior is characterized by rapid and efficient selection among and application of well-encoded rules.21–23 Cognitive task analysis teases out these rules and the pattern recognition settings that trigger them. Cognitive task analyses for this project were conducted by a research assistant with an undergraduate degree in cognitive science who had completed the formal Applied Cognitive Task Analysis curriculum offered by the developers of the methodology (Klein Associates, http://www.decisionmaking.com).
Results
Chart Review Data
Results for preventive mammography, fecal occult blood testing, and diabetic retinopathy screening from 1996 through June of 2000 are displayed in Figure 2. Repeated measures ANOVA shows significant differences between rates of compliance for different rules (P < .0001), across time (P = .0291), and for interactions between rules and time (P = .0079). Specifically, significantly higher rates of compliance are seen for preventive mammography than fecal occult blood testing (56.1% vs 27.5%, P < .0001), or screening for type 2 diabetic retinopathy (56.1% vs 34.9%, P < .0001) when averaged over time. When the effect of time is considered, a significant increase in overall compliance is seen from 1997 to 2000 (29.6% vs 53.1%, P = .0041) and 1998 to 2000 (34.7% vs 53.1%, P = .0005). Finally, significant interactions between time and the type of preventive service were seen, with significant increases in compliance with fecal occult blood testing (1996–2000, 22.5% increase, P = .0037; 1997–2000, 23.5% increase, P = .0024) and screening for type 2 diabetic retinopathy (1997–2000, 15.7% increase, P = .0004; 1998–2000, 16.9% increase, P = .0001; and 1999–2000, 12.4% increase, P = .0048). In contrast, compliance with preventive mammography recommendations remained relatively flat.
Service compliance rates based on chart review from 1996 to 2000.
ClinfoTracker Data
We first analyzed the ClinfoTracker response summary data for all three sites and the three prompts being studied—preventive mammography, fecal occult blood testing, and screening for diabetic retinopathy. As shown in Figure 3, we saw significant increases in responses to prompts from May through July, after which response rates remained relatively flat (overall repeated measures ANOVA, P < .0001). At their highest point, in August, on average nearly one third of prompts were eliciting a response by the clinician.
Response rates to prompts: May − October 2000. Note: significant differences between response rates according to Bonferroni adjustments: May vs July, August, September, October (P < .001); June vs July, August, September, October (P < .001).
Next, we checked for an interaction effect from the site where the prompts were delivered. After entering that data into our repeated measures ANOVA, we saw significant differences overall between sites (P = .0360), but insignificant differences in the changes of response rates between sites with time (P = .2116). In other words, while sites responded to the prompts at different rates, there was no evidence that any one of the sites implemented the system more rapidly. Response rates separated by practice site are displayed in Figure 4.
Prompt response rates from May − October 2000 according to practice site. Note: significant differences between response rates across all months according to Bonferroni adjustment: site 1 vs site 2 (P = .0158).
Finally, we checked for interaction effects of the specific service being prompted. We saw no significant overall differences in response rates by service (P = .2546) and no differences between the services in how their responses changed with time (P = .0707). In other words, there was no evidence of a differential response by clinicians to the prompts based on the specific service. Response rates separated by specific service are displayed in Figure 5.
Prompt response rates from May − October 2000 according to service. Note: no significant differences seen between response rates across all months according to Bonferroni adjustment.
Cognitive Task Analysis Data
The 7-physician cognitive task analyses completed for this project uncovered a substantial degree of variation below the surface of what appears to be a relatively uniform task. Viewed from the traditional operational or health services research perspective, physicians in this practice were uniform in their patient care activity. At the start of each clinic session, they reviewed their schedules with their medical assistants. At the time of each visit they examined the patient’s chart immediately before entering the room, and they established the issues to be addressed with the patient at or very near the beginning of the physician-patient encounter. Cognitive task analysis showed, however, that the physicians varied greatly in how they approached and accomplished the decision tasks of those last two steps. The patterns of cognitive workload and framing constraints, important for any attempt to engineer change in the processes of care, emerged that were not consistent across physicians.
The overriding consistent message in the cognitive task analyses was the sense of time pressure among the physicians. Specifically, they believed that the number of worthy tasks to accomplish exceeded the time available, and that, of necessity, some would go unperformed in every encounter. To an extent that varied by physician, time pressure seemed to serve as a proxy for cognitive workload. Tasks with greater divergence from the primary visit agenda (eg, ensuring that diabetes care services were up to date when the patient seeks care for depression and multiple somatoform complaints) were perceived as imposing more on the physician’s time. (Cognitive workload and distraction are crucial variables affecting expert decision making in all paradigms studied, but to date they have been best studied in aviation.24)
Two styles of interaction with ClinfoTracker encounter forms emerged from the cognitive task analysis. One was a style that was open to agenda items for an encounter beyond those expressed by the patient, and the other was a style that was less receptive to these additional agenda items. The evidence for this difference was seen in the amount of time and energy invested in reviewing the medical record immediately before the encounter. For some physicians this step was almost cursory; those physicians tended strongly to defer health maintenance issues to scheduled physical examinations and chronic disease management services to appointments scheduled specifically to follow up on those conditions. Physicians adhering to this pattern of cognitive task structure perceived more cognitive workload burden from distracting tasks and tended either to actively ignore the prompts on the prompt and reminder system encounter document (ie, deliberate cognitive filtering), or to use them only as cues to tell the patient to schedule an appointment for a physical examination or diabetes visit. Physicians structuring the encounter task in this manner also tended to avoid defining the visit agenda until seeing the patient and generally structured the visit almost entirely on the patient’s complaint(s).
A different pattern was exhibited by physicians who scanned the medical record (and when provided, the ClinfoTracker encounter sheets) problem list before entering the examination room and used that information to plan the visit with the patient. These physicians tended to try to fit prevention or managing chronic diseases into acute visits, rather than to schedule return visits specifically for them. The physicians who tended to plan encounters before entering the examination room also used the prompts to save work. If ClinfoTracker did not flag the patient as needing a diabetic eye examination, for example, the physician would spare himself or herself the workload and distraction of searching through the chart to find the report from the last examination, trusting that if it were due, it would have been flagged.
Discussion
Our initial implementation of ClinfoTracker shows the ability of the system to serve as a platform to study prompted primary care practice and the beneficial impact of ClinfoTracker on preventive and disease management service delivery. During this initial implementation phase, we examined clinicians’ interactions with ClinfoTracker in three ways: chart review of documented preventive and disease management services delivery, analysis of clinicians’ response rates to ClinfoTracker-generated prompts, and a cognitive task analysis of how clinicians incorporate ClinfoTracker into their daily practice of medicine.
For two of the three services studied through chart review, we found significant improvements in compliance with recommendations after implementation of ClinfoTracker. Specifically, ClinfoTracker appeared to have a significant impact on the performance of diabetic retinopathy screening and fecal occult blood testing. Although our sampling design could not ensure that every patient visit had a ClinfoTracker encounter form, the introduction of ClinfoTracker clearly resulted in a change in a previously stable performance rate. No impact was noted on preventive mammography screening rates. The high base rate of mammography in this particular site (which has a relatively high socioeconomic status and easy availability of mammography services) might reflect a ceiling effect: all the patients receptive to mammography recommendations were already receiving mammograms.
The data that capture clinicians’ responses to ClinfoTracker prompts are especially interesting. A clear implementation phase that requires 2 months was observed for all practices. Response rates appear to plateau after that. This implementation phase is also consistent across the three services examined; therefore, there does not appear to be any one service for which prompts are implemented in a more rapid fashion. In addition, despite the similar implementation phase between sites, there were differences in overall responses to ClinfoTracker prompts by clinical site. The three sites studied are quite different in their location (suburban and rural), number of clinicians,4–15 and clinician specialty (family practice or internal medicine). It is tempting to conclude that location, specialty, and size differences are responsible for the differential response rates, but to do so could be quite misleading. We believe that specific factors responsible for these differences in response can be discovered by a cognitive task analysis performed across all practices using ClinfoTracker.
It can be argued that a response rate to prompts that peaks at 32% is quite low. It should be noted, however, that prompts are being delivered at all clinical encounters rather than encounters in which health maintenance issues were already part of the agenda, such as annual examinations. In this context, we view the ability of ClinfoTracker to get preventive and disease management services on the agenda as quite successful.
The cognitive task analyses performed at site 1 show the effectiveness of this technique in uncovering the different ways that ClinfoTracker prompts were utilized by individual clinicians. The cognitive task analysis results in this study are not sufficiently extensive to draw conclusions or design human factors engineering interventions, but the results begin to highlight the range of variation in how physicians structure the cognitive tasks of prevention and chronic disease management in primary care, as well as how much variation exists at the cognitive task level beneath the surface of what appears to be a consistent task at the operational level. Even this limited sample makes it clear that traditional quality improvement interventions conducted at the clinic operations level will have widely varying and unpredictable results.
The necessary next step is a set of cognitive task analyses and human factors engineering interviews of a broader, purposive sample of physicians, selected for maximum variation, as well as the office staff associated with them. The goal of that research project would be to achieve saturation, that is, continue sampling until new variations and patterns are no longer being uncovered. In that way we can catalog the range of patterns of cognitive task structuring, so that prompt and reminder interventions can be introduced with more successful and more consistent effects.
In summary, ClinfoTracker has begun to prove itself useful, not only for actively improving clinician compliance with preventive and disease management services, but also for the study of how to best integrate these prompts and reminders into primary care practice. We are actively pursuing further research to extend the results reported thus far. It is our belief that this research will add substantially to the theoretical basis for prompting primary care clinicians. Ultimately we hope to derive principles that will guide development of future prompting systems. In this way, we believe primary care can deliver on the promise of comprehensiveness and continuity while delivering state-of-the-art, evidence-based care.
Acknowledgments
The authors would like to acknowledge the contributions of Robert E. Chistensen.
Notes
Presented at the 28th Annual Meeting of the North American Primary Care Research Group in Amelia Island, Fla, November 6th, 2000.
- Received for publication November 6, 2001.
- Revision received November 6, 2001.
References
Jump to section
Related Articles
Cited By...
- Clinical Reminders Designed and Implemented Using Cognitive and Organizational Science Principles Decrease Reminder Fatigue
- Prompting Primary Care Providers about Increased Patient Risk As a Result of Family History: Does It Work?
- Impact of an Electronic Health Record (EHR) Reminder on Human Papillomavirus (HPV) Vaccine Initiation and Timely Completion
- Using Patient Monetary Incentives and Electronically Derived Patient Lists to Recruit Patients to a Clinical Trial
- Adherence to Maintenance-Phase Antidepressant Medication as a Function of Patient Beliefs About Medication