Rate of undesirable events at beginning of academic year: retrospective cohort study
BMJ 2009; 339 doi: https://doi.org/10.1136/bmj.b3974 (Published 14 October 2009) Cite this as: BMJ 2009;339:b3974
- Guy Haller, consultant14,
- Paul S Myles, professor and director2,
- Patrick Taffé, biostatistician3,
- Thomas V Perneger, professor and head of division4,
- Christopher L Wu, associate professor5
- 1Department of Anaesthesia, Pharmacology and Intensive Care, Geneva University Hospital, University of Geneva, 1211 Geneva, Switzerland
- 2Department of Anaesthesia and Perioperative Medicine, Alfred Hospital, Monash University, Prahran, Vic 3181, Australia
- 3Institute of Social and Preventive Medicine, 1005 Lausanne, Switzerland
- 4Division of Clinical Epidemiology, Geneva University Hospital
- 5Department of Anesthesiology and Critical Care Medicine, Johns Hopkins University, Baltimore 21205, MD, USA
- Correspondence to: Guy Haller Guy.Haller{at}hcuge.ch
- Accepted 4 August 2009
Abstract
Objective To determine whether an increase in the rate of undesirable events occurs after care provided by trainees at the beginning of the academic year.
Design Retrospective cohort study using administrative and patient record data.
Setting University affiliated hospital in Melbourne, Australia.
Participants 19 560 patients having an anaesthetic procedure carried out by first to fifth year trainees starting work for the first time at the hospital over a period of five years (1995-2000).
Main outcome measures Absolute event rates, absolute rate reduction, and rate ratios of undesirable events.
Results The rate of undesirable events was higher at the beginning of the academic year compared with the rest of the year (absolute event rate 137 v 107 per 1000 patient hours, relative rate reduction 28%, P<0.001). The overall adjusted rate ratio for undesirable events was 1.40, 95% confidence interval 1.24 to 1.58. This excess risk was seen for all residents, regardless of their level of seniority. The excess risk decreased progressively after the first month, and the trend disappeared fully after the fourth month of the year (rate ratio for fourth month 1.21, 0.93 to 1.57). The most important decreases were for central and peripheral nerve injuries (relative difference 82%), inadequate oxygenation of the patient (66%), vomiting/aspiration in theatre (53%), and technical failures of tracheal tube placement (49%).
Conclusions The rate of undesirable events was greater among trainees at the beginning of the academic year regardless of their level of clinical experience. This suggests that several additional factors, such as knowledge of the working environment, teamwork, and communication, may contribute to the increase.
Introduction
At the beginning of an academic year, teaching hospitals accept an influx of new trainees and fellows. In the United States alone, this represents more than 100 000 interns/residents; in Europe more than 32 000 new medical graduates start work each year.1 2 This transition phase is often considered to be the worst time of the year to be admitted to hospital; it is referred to as the “August killing season” in the United Kingdom and the “July phenomenon” in the United States.3 4 Whether this transition period is truly associated with an increase in adverse events or poor quality of care remains unclear,5 6 7 8 9 10 11 12 13 largely because previous studies have relied on relatively small samples,9 10 insufficient adjustments for differences in case mix,4 5 11 and administrative data to analyse rare outcomes such as in-hospital mortality.6 7 8 12 13
Anaesthetic care affords a unique opportunity to explore this phenomenon. The delivery of anaesthesia takes place in a complex and dynamic environment, where opportunities for undesirable events abound.14 15 Most quality assurance programmes for anaesthesia require these events to be systematically recorded and collected into large datasets that are used for teaching and quality assurance purposes.16 17 18 We did a retrospective cohort study based on administrative data and patients’ records to examine whether patients exposed to an anaesthetic procedure carried out by trainees at the beginning of the academic year had a higher rate of undesirable events than patients operated on later in the year. We also analysed the trend in the rate of undesirable events throughout the year.
Methods
Study design and setting
We did a retrospective cohort study using data collected during the period from 1 October 1995 to 31 December 2000 in the Department of Anaesthesia and Perioperative Medicine at the Alfred Hospital, a large university affiliated hospital in Melbourne, Australia. It serves as an accredited centre approved by the Australian and New Zealand College of Anaesthetists for both basic and advanced training in anaesthesia. The hospital accepts trainees from all levels (first to fifth year) after completion of at least 24 months of general hospital experience (equivalent to PGY1 and PGY2) in a programme approved by the Confederation of Postgraduate Medical Education Councils after graduation from medical school.
The training of new anaesthesia registrars usually begins in February, but a few trainees may join the programme later in the year. At the time of data acquisition the department was responsible for the anaesthetic care of around 15 000 adult surgical patients a year, including most types of specialised surgical services but excluding obstetrics and paediatrics.
Data source
For the purpose of the study, we retrieved and aggregated data from the anaesthesia department and hospital administrative databases and cross checked validity with handwritten medical records. Recorded data included patients’ demographics, comorbidities, the American Society of Anesthesiologists (ASA) physical status score,19 previous anaesthetic complications, comprehensive anaesthetic details, type of surgery, intraoperative adverse events, and procedure related information such as timing, duration, and sequence of procedure, emergency status, faculty number for registrars/consultants in charge, and level of supervision. The final dataset incorporated all information on in-hospital or ambulatory procedures carried out with an anaesthetist in attendance and classified according to the ICD-10 (international classification of diseases, 10th revision).20 Details of the methods used to prepare and validate the dataset can be found in additional publications.21 22
Participants
We included all patients who had an anaesthetic procedure carried out by first to fifth year trainees working for the first time at our hospital. If procedures were done by trainees who had already started their training at our institution before the beginning of the data collection process, we excluded the patients. If trainees stayed more than one year, we considered only procedures done during their initial training year at our hospital.
Primary outcome measure
Our primary outcome measure was the number of undesirable events occurring during an anaesthetic procedure. Events were measured by the electronic incident reporting system. This is integrated into the patient’s electronic record for anaesthesia, and completion is mandatory for any procedure carried out, even if no event occurs. All trainees are instructed in the collection of data and are provided with written instructions and definitions of items. The quality assurance officer or the consultant in charge of managing the database does a weekly check on the completion of these forms.
The form includes a list of 44 predefined incidents, one free text field, and a “no incident” category. It was developed after a consensus conference of five board certified anaesthetists and one quality assurance officer. They defined incidents as unintended events or outcomes that could have reduced or did reduce the safety margin for the patient.23 Thus, the definition included both injuries resulting from medical management (adverse events) and acts of commission or omission that could have harmed but did not cause harm (near misses).24 25 This reporting system has 80% sensitivity and 91% specificity for intraoperative incidents identified by peer reviewers in medical charts. Reporting practices are practically constant over time.21
Other variables
We used the following variables in our analysis: patient’s age, sex, ASA physical status score, comorbidities, type and duration of surgical and anaesthetic procedures, emergency status, time of day, trainee’s level of seniority, and type of supervision. All were considered as potential confounders or effect modifiers for the association between undesirable events during procedures carried out by trainees and the period of the year when these procedures were done. To describe undesirable events, we used standardised rates of resident associated events per 1000 patient hours of procedures (for each level of training). To account for the fact that some trainees had their first contact with the hospital several months later than the official beginning day of the academic year (1 February in Australia), we defined the beginning of the academic year as the first sequence of procedures carried out by trainees during their initial year of residency at the Alfred Hospital. Afterwards, we divided the academic year into 12 equal periods. To take into account the experience acquired by trainees both over time and with the number of procedures carried out, we defined periods, for each resident, as one twelfth of the total number of anaesthetic procedures they had done during the year. Thus, the true length of one period could be slightly shorter or longer than a calendar month.
We classified training and supervision level according to the specifications of the Australian and New Zealand College of Anaesthetists’ training programme. We divided the time of day of the procedures into in hours (7 00 am to 6 59 pm) and after hours (7 00 pm to 6 59 am). We summarised patients’ comorbidities by ICD-10-AM (Australian modification) diagnostic codes and surgical procedures by aggregation into 14 main groups of individual ICD-10-AM intervention codes.20
To assess consistency of record keeping, we created a new variable named “incident missing” to describe the non-use of the incident reporting system by trainees. We attributed the value 1 to the “incident missing” variable if neither the different categories of incidents nor the “no incident” variable was ticked, because incidents can only be present or absent. We attributed the value 0 in all other cases.
Statistical analysis
We categorised continuous covariables for simpler presentation. We used frequencies and percentages to describe data. We used SAS 9.1.3 and Stata version 10.1 for all the analyses.
Our primary analyses focused on the relations between undesirable events during procedures carried by anaesthesia trainees and the period of the year when these procedures were done. Firstly, we compared the initial period of the academic year (February in Australia) with the rest of the year. Then we assessed the evolution of the rate of undesirable events throughout the 12 periods of the year. We used cross tabulation and χ2 tests for these explorations. We developed multivariable models to account for differences in case mix of patients (age, sex, ASA physical status score, and comorbidities), type of surgery, and characteristics of anaesthetic procedure (type, duration, emergency status, time of day, level of training, and mode of supervision) during the different periods of the year. Because the data had a complex multilevel cross classified structure, with individual interventions (level 1) nested in groups of aggregated interventions (level 2) and in individual trainees (level 2’), they were not independent and the analyses accounted for the clustering. Therefore, we used generalised linear mixed models with the number of undesirable events per procedure as the outcome variable. Because over-dispersion is common with count data, we tested various multilevel count models for error distributions: Poisson, geometric, and negative binomial. Furthermore, as the probability of zero occurrence exceeded that assumed by the model, we also considered zero inflated and zero truncated versions of these models.26 The final model that proved to be the most appropriate was the geometric multilevel count model incorporating the two levels of nesting. We considered all covariables as potential confounders. We checked for co-linearity and tested potential interactions between periods and the degree of training.
We treated procedures as a fixed effects and individual trainees as a random effects, because each year new trainees are enrolled. For the estimation, we used maximum likelihood as well as a quasi-likelihood methods.27 A P value of less than 0.05 was considered to indicate statistical significance, and all tests were two sided. We evaluated goodness of fit of the models by using the Pearson χ2 dispersion statistic, graphs of mean predicted and observed probabilities of counts, and QQ plot of various residuals (Deviance, Anscombe).
In secondary analyses, we focused on specific undesirable events. We identified and classified all events according to their frequency and level of decrease between the first period and the rest of the academic year.
Results
Description of dataset
After extraction, the initial dataset included 47 809 patients with an anaesthetic procedure. After excluding interventions carried out entirely by consultants or by trainees already working at our hospital at the beginning of the study, we identified 19 560 patients with an anaesthetic procedure done by first to fifth year trainees during their initial year of residency at the Alfred Hospital.
The proportion of missing data for patients, procedures, or trainee related variables ranged between 1% and 2.2%. Most could be completed on the basis of information from medical charts.
Patients and procedures
During the study period, 19 560 anaesthetic procedures were done by 93 trainees during their initial year of residency at the Alfred Hospital. Most trainees (n=57) were in their first or second training year (basic training); the remainder (n=36) were in their third, fourth, or fifth year of training (advanced training). The trainees did on average 210 (interquartile range 109-300) interventions per academic year, of which on average 30 (18-38) were done during the first period after their arrival in the hospital. The median duration of interventions was 1.25 (0.75-2.33) hours.
Table 1⇓ shows the characteristics of the patients, and table 2⇓ shows the characteristics of the trainees and the system. On average, at the beginning of the academic year, patients were healthier and had a lower anaesthetic risk (ASA physical status score). They were also less likely to have emergency and high risk surgical procedures such as cardiothoracic surgery or neurosurgery. Trainees’ supervision by a consultant was also reinforced with a higher level of one to one supervision in the operating theatre. The overall reporting rate of undesirable events was very high, and the system was used in 94.04% of the procedures. Its use remained sensibly constant over all the different periods of the year (P=0.07).
Characteristics of patients. Values are numbers (percentages)
Characteristics of trainees and system. Values are numbers (percentages)
Undesirable events at beginning of academic year
During the study period, 2672 undesirable events occurred. These events occurred more often in the first month of the academic year than during the rest of the year (absolute event rate 137 v 107 per 1000 patient hours, relative rate reduction 28%, P<0.001). This increase in the first month occurred not only for first year trainees (crude rate ratio 1.21, 95% confidence interval 1.04 to 1.42) but also for trainees at more advanced levels, including fifth (last) year trainees (crude rate ratio 1.65, 1.31 to 2.07) (table 3⇓). The increase in the rate of adverse events early in the course of the academic year was even amplified after statistical adjustment for differences in patients’ comorbidities, surgical interventions, ASA scores, timing of procedures (night/day), emergency status, and reinforced supervision in February (table 3⇓), with an overall adjusted rate ratio for undesirable events of 1.40, 1.24 to 1.58 (P<0.001).
Crude and adjusted rate ratio for undesirable events between first period and rest of academic year, according to training level*
This increase in the rate ratio of undesirable events during the first month varied somewhat across training levels (table 3⇑). However, these differences were not statistically significant (P=0.33), indicating that the increase in risk was about the same regardless of the level of training.
Evolution of undesirable events over year
For all trainees, regardless of their training level, an increase in the rate of undesirable events occurred during the second, third, and fourth months of the year—adjusted rate ratios for undesirable events 1.26, (0.97 to 1.63), 1.29 (0.99 to 1.66), and 1.21 (0.93 to 1.57). This increase did not, however, reach statistical significance. After the fourth month, the rate of undesirable events reached steadiness (figure⇓).
Evolution of undesirable events over different periods of year (overall)
Type of undesirable events
The most common types of undesirable events occurring during anaesthetic procedures at the beginning of the academic year were severe uncontrolled hypotension (93.1 per 1000 procedures) and hypertension (15.6), technical failures of arterial line insertion (14.0), miscellaneous airways and haemodynamic related complications (12.5), accidental upper airway obstruction (10.5), and technical failures of end tracheal tube placement (9.7). Most of them decreased after the first month of the academic year. The biggest decreases were those for central and peripheral nerve injuries (82%), inadequate oxygenation of patient (66%), vomiting/aspiration in theatre (53%), and technical failures of tracheal tube placement (49%). Table 4⇓ shows undesirable events with the most variation at the beginning of the academic year compared with the rest of the academic year.
Undesirable events showing most important variation at beginning compared with rest of academic year
Discussion
We found an increase in the rate of undesirable events at the beginning of the academic year for anaesthesia trainees joining a hospital for the first time, regardless of their level of training. This phenomenon decreased progressively after the first month, and the trend fully disappeared after the fourth month. This decline was particularly large for undesirable events related to the performance of technical procedures and overall management of patients.
Comparison with other studies
A popular perception is that the introduction of new trainee physicians is associated with an increase in adverse outcomes in patients, but the evidence from previous studies in general medical, obstetric, or surgical patients does not support this hypothesis.4 5 6 8 9 10 11 12 13 No increase in morbidity and mortality was seen, even when new house officers cared for severely ill patients.4 8 Even in studies examining procedure oriented specialties, no increase in complications or mortality was seen at the beginning of the academic cycle.10 12 13 One recent study identified an increase in 30 day postoperative mortality in an early cohort (1 July-30 August) compared with a later one (15 April-15 June), but a spike in mortality also occurred in the middle of the academic year (December), and no significant difference in morbidity occurred throughout the year.7
Several reasons may explain why our results differ from those of previous negative studies. Firstly, our ongoing data collection process for quality assurance provides great detail on specific events in patients, which is different from studies that explored only administrative databases.6 7 8 12 13 Secondly, compared with previous studies examining actual patients’ data,4 10 our sample size is much larger, which results in greater statistical power. Finally, anaesthesia care is inherently riskier than other medical specialties, which facilitates the detection of a spike in adverse events at the beginning of the academic year.
Possible causes and implications of study findings
Previous studies found that the success rate of surgical, medical, or anaesthetic procedures was correlated with the number of procedures carried out.28 29 30 Our data confirm the presence of a learning curve. The rate of undesirable events was significantly higher in the early stages of a rotation, particularly for technical related skills, which also showed significant improvements throughout the year. Endotracheal tube failures decreased by 49% and arterial catheter placement failures by 27%. Central and peripheral nerve injuries related to patients’ position on the operating table, performance of peripheral nerve block, and brain tissue oxygenation of patients during anaesthesia decreased by up to 82%. This higher rate of undesirable events related to technical procedures was present despite higher one to one supervision (76%) for first year trainees at the beginning of the academic year. This suggests that supervision protects patients from serious errors and violations or helps to mitigate harm when it occurs but does not provide novices with the experience and technical success rate of someone more experienced.31 Training in technical procedures should be the focus of attention during the first period of contact with the clinical environment, particularly for junior trainees. This could be achieved through formalised training sessions including special sessions in simulators scheduled early in the academic year to develop and enhance trainees’ technical skills without risk of injuries to patients.32 33 34
Surprisingly, the rate of undesirable events did not differ significantly between the different training levels. This is in contradiction to previous studies showing that the success rate of surgical or medical procedures increases with clinical experience.35 36 37 38 For instance, Lesar et al found a higher rate of drug prescription errors among first year trainees compared with other classes of prescriber, and Cosgrove et al found evidence of better surgical outcome and fewer complications on lower limb amputation when procedures were done by senior surgeons rather than by junior trainees.35 37
Our finding that more experienced trainees have as many undesirable events as new trainees suggests that lack of technical skills is not the only mechanism explaining the occurrence of incidents or its increase at the beginning of the academic year. New trainees, regardless of their level of training, are unfamiliar with their new working environment (for example, hospital rules, location of medical charts and laboratory results, roles of other healthcare providers). As a result, breakdown in communication and poor interprofessional interactions, two well identified causes of errors and undesirable events, may result.39 40 Secondly, advanced trainees (unlike first year trainees) may be allocated to several locations and activities throughout the hospital within a few hours, and excessive cross coverage and poor handovers can lead to undesirable events.41 42 Finally, insufficient supervision of advanced trainees at the beginning of the academic year may be another explanation for our findings.43 Because advanced trainees, particularly fifth year trainees, are considered to be technically more competent they may be quickly involved in complex clinical cases even though they may still need direction and guidance about hospital routines, material, and organisation.
Thus, several factors related to knowledge of the working environment, teamwork, and communication may contribute to this increase in undesirable events at the beginning of the academic year. This challenges the common view that in the medical profession expertise relies solely on medical knowledge, technical skills, and clinical judgment.44 Adequate knowledge of the specificities of the working environment and awareness of teamwork related factors may also contribute strongly to professional expertise. These aspects should become a full part of the definition of academic training.45
Possible improvement strategies
Strategies to minimise the rate of undesirable events at the beginning of the academic year should look at improving trainees’ orientation and integration during the first weeks, by developing, for example, mandatory introductory courses, hospital settings’ visits, and interprofessional meetings and, beyond all the rest, by avoiding residents’ involvement in clinical tasks from the first day. The orientation period could also include close one to one supervision, particularly when cross cover work is done in different hospital settings. The systematic use of written documentation of standard working practices should be encouraged to minimise the loss of tacit knowledge associated with staff turnover.46 Crew resource management programmes such as those developed in aviation and now increasingly introduced in the healthcare environment could be used to improve team coordination and interprofessional collaboration.36 47 48 Finally, early training sessions in simulators could be scheduled to favour rapid improvements in junior trainees’ technical skills.
Strengths and limitations of study
Several limitations must be considered when interpreting the results of our study. Our evaluation of the phenomenon was limited to anaesthesia trainees of one university affiliated institution, and the results may lack generalisability. Despite its high sensitivity and specificity, the mandatory incident reporting system may have missed some events, possibly resulting in an underestimation of the extent of the phenomenon. Another possible concern is a seasonal fluctuation in the levels of anaesthetic risk, perhaps due to differences in case mix of patients, types of procedures, or staffing patterns. We tackled these problems by adjusting results for patients’ comorbidities, demographic characteristics, type and complexity of anaesthesia and surgical procedure, and night-time and emergency status. We also defined the beginning of the academic year by the first sequence of procedures carried out by trainees after their initial contact with the hospital rather than by a specific calendar month. This enabled us to account for the fact that some trainees had their first contact with the hospital several months after the others (holidays, sick leave) while minimising the impact of unmeasured confounding factors related to specific calendar months. We cannot totally exclude, however, that other relevant confounders may have been missing from this analysis.
Another limitation relates to the definition of undesirable events we used. It includes both injuries caused by medical management (adverse events) and acts of commission or omission that did not cause harm (near misses). Although both are relevant to patients’ safety, their respective impact on patients and the methods used to cope with them differ significantly. When we analysed individually the different types of undesirable events that occurred and decreased the most between the beginning and the rest of the academic year, most of them were not near misses but true adverse events (central and peripheral nerve injuries, vomiting/aspiration in theatre, respiratory arrest). This emphasises the relevance of our findings for patients’ safety and the importance of rapidly tackling this increased rate of undesirable events at the beginning of the academic year.
Finally, we limited the measurement of undesirable events to the intraoperative period. How many of these events led to adverse outcomes during the postoperative period is unclear. Future studies on this phenomenon should consider the measurement of undesirable events through direct observation, to increase the sensitivity of detection for undesirable events.49 50
Conclusions and policy implications
We identified an increased rate of undesirable events at the beginning of the academic year during procedures carried out by new anaesthesia trainees. Possible strategies to minimise this phenomenon include improving trainees’ orientation and integration in their first weeks of employment, increasing intensity of supervision in more advanced trainees, and developing early programmes aimed at improving technical and teamwork skills.
What is already known on this topic
Common wisdom suggests an increase in the rate of undesirable events at the beginning of the academic year in teaching hospitals, but available data on this phenomenon are unclear
Published evidence suggests no relation between the introduction of new trainees and an increase in the number of undesirable events
What this study adds
An increase in the rate of undesirable events occurred at the beginning of the academic year
This phenomenon was present regardless of trainees’ level of clinical experience; it decreased progressively after the first month to disappear fully after the fourth month
The beginning of the academic year is a risky period for patients; seniority of trainees does not protect patients from undesirable events
Notes
Cite this as: BMJ 2009;339:b3974
Footnotes
Contributors: GH, PSM, and CLW were responsible for study conception and design. GH and PSM acquired the data. All authors analysed and interpreted the data. GH and CLW drafted the manuscript, and all authors revised it critically for important intellectual content. GH and PT did the statistical analysis. GH obtained funding. DH, PSM, TVP, and CLW provided administrative, technical, or material support. All authors had full access to all of the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis. GH and CLW are the guarantors.
Funding: This research was supported by a University of Geneva (Switzerland) postgraduate research scholarship awarded to GH. The sponsor had no role in study design; data collection, analysis, and interpretation; or publication of results. The researchers were all independent from the funder.
Competing interests: None declared.
Ethical approval: The study involved no contact with patients. The Alfred Institutional Ethics Committee approved the use of data retrieved from patients’ medical charts and hospital databases.
Data sharing: No additional data available.
This is an open-access article distributed under the terms of the Creative Commons Attribution Non-commercial License, which permits use, distribution, and reproduction in any medium, provided the original work is properly cited, the use is non commercial and is otherwise in compliance with the license. See: http://creativecommons.org/licenses/by-nc/2.0/ and http://creativecommons.org/licenses/by-nc/2.0/legalcode.