ABSTRACT
OBJECTIVE
Valid teamwork assessment is imperative to determine physician competency and optimize patient outcomes. We systematically reviewed published instruments assessing teamwork in undergraduate, graduate, and continuing medical education in general internal medicine and all medical subspecialties.
DATA SOURCES
We searched MEDLINE, MEDLINE In-process, CINAHL and PsycINFO from January 1979 through October 2012, references of included articles, and abstracts from four professional meetings. Two content experts were queried for additional studies.
STUDY ELIGIBILITY
Included studies described quantitative tools measuring teamwork among medical students, residents, fellows, and practicing physicians on single or multi-professional (interprofessional) teams.
STUDY APPRAISAL AND SYNTHESIS METHODS
Instrument validity and study quality were extracted using established frameworks with existing validity evidence. Two authors independently abstracted 30 % of articles and agreement was calculated.
RESULTS
Of 12,922 citations, 178 articles describing 73 unique teamwork assessment tools met inclusion criteria. Interrater agreement was intraclass correlation coefficient 0.73 (95 % CI 0.63–0.81). Studies involved practicing physicians (142, 80 %), residents/fellows (70, 39 %), and medical students (11, 6 %). The majority (152, 85 %) assessed interprofessional teams. Studies were conducted in inpatient (77, 43 %), outpatient (42, 24 %), simulation (37, 21 %), and classroom (13, 7 %) settings. Validity evidence for the 73 tools included content (54, 74 %), internal structure (51, 70 %), relationships to other variables (25, 34 %), and response process (12, 16 %). Attitudes and opinions were the most frequently assessed outcomes. Relationships between teamwork scores and patient outcomes were directly examined for 13 (18 %) of tools. Scores from the Safety Attitudes Questionnaire and Team Climate Inventory have substantial validity evidence and have been associated with improved patient outcomes.
LIMITATIONS
Review is limited to quantitative assessments of teamwork in internal medicine.
CONCLUSIONS
There is strong validity evidence for several published tools assessing teamwork in internal medicine. However, few teamwork assessments have been directly linked to patient outcomes.
Similar content being viewed by others
Effective teamwork among health professionals improves patient safety.1 , 2 A substantial proportion of preventable errors in United States hospitals are attributable to teamwork and communication failures.3 , 4 Recognizing this, the Institute of Medicine, the Joint Commission, the Agency for Healthcare Research and Quality (AHRQ), and others have made teamwork a top priority in their recommendations for improving healthcare.5 – 9
Teamwork is also prominently positioned within the American Board of Internal Medicine (ABIM) requirements for maintenance of certification for internists,10 as well as the Accreditation Council for Graduate Medical Education’s core competencies,11 milestones,12 and medical student competencies.13 As such, every physician at the undergraduate, graduate, and continuing professional level must demonstrate competency in teamwork.
While there is broad agreement on the imperative to improve teamwork, there is little consensus regarding how to measure it. Internal medicine teams vary substantially in composition, setting, function and charge. The knowledge, skills, and attitudes required for optimal teamwork within an inpatient medical team may differ from those necessary for successful interprofessional collaboration among undergraduate students in a classroom.13 , 14 Additionally, there are numerous purposes for teamwork assessment, including determining individual physician competence as well as measuring the effectiveness of teams as a whole.15
Given the heterogeneity of healthcare teams within internal medicine, it is logical that no single teamwork measurement tool will suit all clinical and educational situations. Yet, any endeavor to measure teamwork is likely to be most successful if it is grounded in the literature, built upon prior work, reliable and valid.15 Prior reviews have examined teamwork training and interventions, as well as the outcomes of effective teams.1 , 16 – 24 These reviews have advanced the understanding of ‘what works’ to improve teamwork (i.e. curricula and interventions), but they do not fully answer the critical question of how teamwork is best measured in healthcare.
Therefore, the objective of this systematic review is to provide a synthesis of published instruments that have been used to assess teamwork in internal medicine. Given the breadth and marked heterogeneity of literature on teamwork assessment within healthcare as a whole, this review was limited to a synthesis of teamwork tools used in internal medicine. It encompasses all instruments used in undergraduate, graduate, and continuing medical education in general internal medicine and internal medicine subspecialties. To capture all published validity evidence for each tool, we also included articles from non-internal medicine specialties that reported additional validity evidence. This paper is intended to serve as a resource to help educators, clinicians, and other health professionals identify appropriate teamwork measurement tools to apply to their own internal medicine settings and teams.
METHODS
Although there are no standard reporting guidelines specific to systematic reviews of assessment tools, this review is reported according to applicable sections of the Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA) standards25 and similar reviews of assessment tools in medical education.26
Data Sources and Search Strategy
We searched MEDLINE, MEDLINE In-process, Cumulative Index to Nursing and Allied Health Literature (CINAHL), and PsycINFO for English language studies from January 1, 1979 through October 31, 2012. To identify studies related to teamwork, the term team was exploded to include all Medical Subject Headings (MeSH) and keywords containing “team” (e.g. team, teamwork, teamworking, team behavior, team climate, team culture, team collaboration, team effectiveness). Other MeSH terms and keywords related to teamwork included interprofessional relations, patient care team, cooperative behavior, crew resource, crisis resource and non-technical skills. These terms were combined with measurement terms including: scale, measure, inventory, questionnaire, tool, instrument, assessment, evaluation, profile, indicator, index and survey. Last, terms for teamwork and measurement were combined with terms pertaining to medical education and health professionals, including MeSH students, health occupations, health personnel, education professional, internship and residency, healthcare facilities, manpower and services, and exploded terms doctor, physician, nurse, student, intern, resident, registrar, house officer, medical, surgeon, operating, health, clinic, patient, interdiscliplinary, multidiscliplinary and interprofessional. An expert librarian with experience conducting literature searches for systematic reviews assisted in development and implementation of the search. The exact search strategies for each database are available from the authors.
The reference lists of all included articles were reviewed for additional studies. To identify in-press and unpublished studies, we searched scientific abstracts from national meetings between 2010 and 2012 of the Association of American Medical Colleges (AAMC), Association of Medical Educators of Europe (AMEE), Society of General Internal Medicine (SGIM), and the International Meeting on Simulation in Healthcare (IMSH). Authors of relevant abstracts were contacted for unpublished manuscripts. Finally, two experts who have published prior systematic reviews of teamwork or empiric studies of teamwork assessment reviewed the list of included articles to identify additional studies.
Study Selection
Articles were included if they were original research describing a quantitative tool designed for measuring teamwork within healthcare teams involving physicians and/or trainees in general internal medicine or an internal medicine subspecialty. To provide a comprehensive synthesis, we also included articles from non-internal medicine specialties that reported validity evidence for included tools. Studies of interprofessional teams (defined as two or more professions working together as a team)27 , 28 were included as long as internal medicine physicians (or medical students, residents, fellows) were one of the professions studied. Tools were considered measures of teamwork, based on authors’ descriptions of tools as measuring teamwork, collaboration, team process or function, team behavior, team effectiveness, team climate/environment, team culture, non-technical skills, or crew/crisis management. We excluded studies that measured just one specific aspect of team function, such as conflict, negotiation, leadership, communication, disruptive behavior and harassment. Studies of patient hand-over were excluded since recent reviews on this topic have been published.29 – 31
Title and Abstract Review
The search yielded 12,922 citations (Fig. 1). Each title/abstract was reviewed and we erred on the side of full article retrieval if titles/abstracts were insufficient to determine eligibility. A total of 892 articles was included for full article review. All uncertainties regarding inclusion were resolved by consensus.
Article search and selection.
Data Extraction
Data were entered into a structured extraction form that included information on articles (study location, design, participants, setting) and tool characteristics (content, validity, and outcomes). Five authors extracted data. These authors met weekly during the study period and uncertainties were resolved by consensus. Thirty percent of articles were independently extracted by two authors to verify consistency in coding and determine interrater agreement using an intra-class correlation coefficient (ICC). The remaining 70 % of articles were extracted by a single reviewer.
We used an established framework to categorize the validity of instruments32 – 34 that has been used in similar evaluations of assessment tools.26 This framework includes five categories of validity evidence: 1) content (the degree to which the tool content reflects the construct being measured); 2) response process (training of raters to use the tool); 3) internal structure (instrument reliability including internal consistency, interrater, intrarater, and test-retest reliability); 4) relationships to other variables (relationship between scores and other variables measuring the same construct); and 5) consequences (outcomes associated with tool scores). Kirkpatrick’s hierarchy was used to categorize outcomes as satisfaction/opinion, knowledge and skills, behaviors, and patient outcomes.35 Patient measures were recorded as outcomes only if the study reported a direct quantitative association between the teamwork assessment score and the patient outcome.
To evaluate the methodological quality of studies, we used criteria from the ten-item Medical Education Research Study Quality Instrument (MERSQI),36 which encompasses basic methodological components (e.g. study design, sampling, analysis). Validity evidence for the MERSQI includes content, interrater, intrarater, and internal consistency reliability, and relationships to other variables, including correlations between instrument items and journal impact factor, 3-year citation rate, and journal editors’ quality ratings,36 as well as predictive validity based on associations with editors’ decisions to accept or reject manuscripts for publication.37 We tallied the number of studies that fully, partially or failed to satisfy each of the ten quality criteria in the MERSQI.
Data Synthesis
Characteristics of studies and teamwork measurement tools were synthesized qualitatively and reported in evidence tables. Articles describing identical tools were grouped to enable examination and presentation of all validity evidence and outcomes for each unique tool. Frequencies and percentages were used to describe study and tool characteristics. Means and standard deviations were used to summarize quality scores. Meta-analysis was not possible nor logical, given that this was a review of assessment tools with obvious heterogeneity among instruments, study designs, and outcomes.
RESULTS
Of the 12, 922 citations, 12,629 were identified through the electronic database searches, 16 from reference lists of included articles, two from expert review, and 275 from relevant meeting abstracts. We identified 98 articles from non-internal medicine specialties that contained validity evidence for included tools. The total number of articles meeting inclusion criteria was 178 (Fig. 1). Interrater agreement for data extraction was very good (ICC = 0.73, 95 % CI: 0.63–0.81).
Table 1 shows the characteristics of the 178 included studies. Approximately half of studies were conducted in the U.S. and one-third in Europe. Most (142, 80 %) of studies included practicing physicians as participants, followed by residents (68, 38 %) and medical students (11, 6 %). The majority (152, 85 %) of studies also assessed non-physician professionals (e.g. nurses, pharmacists, mid-level providers, therapists, social workers, administrators) in interprofessional teams. Although most studies took place in actual inpatient or outpatient practice settings, 37 (21 %) of studies were simulation-based and 13 (7 %) took place in classrooms.
Study Quality
Figure 2 shows the proportion of studies satisfying the ten MERSQI quality criteria. Ten (6 %) studies fully satisfied, 59 (33 %) partially satisfied, and 109 (61 %) did not satisfy quality criteria for study design. The most frequent study design was single group cross-sectional (89 %). Ten studies were randomized controlled experiments.38 – 47 A majority (153, 86 %) of studies fully satisfied at least one validity criterion: 122 (69 %) studies reported content validity, 115 (65 %) reported internal structure, and 47 (26 %) described relationships to other variables. Twenty-nine (16 %) studies fully satisfied all three of these validity criteria. Most studies (140, 79 %) relied on subjective assessments by study participants for measuring teamwork.
Methodological quality of 178 studies describing teamwork assessment tools.
Description and Validity Evidence for Teamwork Assessment Tools
The 178 included articles described 73 unique tools designed to measure teamwork (Table 2). Of the 73 tools, 15 (21 %) measured the teamwork of individuals working within teams, 43 (60 %) measured the teamwork of teams as a whole, and 15 (21 %) assessed both individuals and teams.
Content validity was demonstrated for 54 (74 %) of tools (Table 2) and generally consisted of developing instrument content from expert panels, existing instruments, and literature review. The TeamSTEPPS Teamwork Attitudes Questionnaire80 is an example of an assessment tool with strong content validity designed to assess the teamwork attitudes, knowledge and skills of learners participating in the TeamSTEPPS curriculum. TeamSTEPPS is a training program developed by the United States Department of Defense and the AHRQ that encompasses leadership, situation monitoring, mutual support and communication.7 , 80 , 198 The TeamSTEPPS Teamwork Perception Questionnaire is a second instrument associated with this curriculum that measures individuals’ perceptions of organizational teamwork.216
Few tools (12, 16 %) reported response process, which included training raters to correctly use tools. The Multi-disciplinary Team Performance Assessment Tool160 , 161 is an observational teamwork assessment of cancer teams modified from an established teamwork assessment tool in the surgical literature (Observational Teamwork Assessment in Surgery).217 – 219 Assessors were trained in the use of this tool by an expert psychologist with experience using the tool.160
Reliability of tools was demonstrated by internal consistency (38, 52 %), interrater reliability (16, 22 %), intrarater reliability (1, 1 %), and test-retest reliability (2, 3 %). Reliability estimates for most tools were very good (> 0.7).220 The Physician/Pharmacist Collaboration Index is an example of a tool assessing interactions between internists and pharmacists that has extensive reliability evidence, including factor analysis, internal consistency (Crohnbach alpha 0.70-0.90) and interrater reliability (ICC 0.89).38 , 164 – 170 This tool measures the pharmacist’s view of collaboration among physicians and other health professionals in both inpatient and outpatient settings.
Relationships between teamwork scores and other variables reflecting the construct of teamwork were reported for 25 (34 %) tools (Table 2). Studies varied widely with regard to the specific variables reported. The Attitudes Toward Health Care Teams Scale (ATHCTS) has been used in ten studies measuring attitudes towards interprofessional collaboration in a variety of settings, most commonly interprofessional education.58 – 67 It consists of three subscales assessing attitudes about team value, team efficiency and the physician’s shared role on the team. ATHCTS scores have been shown to correlate with other measures of team process.65 The Ottawa Global Rating Scale has been used in multi-specialty education as an objective measurement of an individual’s crisis resource management skills in simulated scenarios.211 , 212 This tool has been shown to differentiate among residents’ level of training when applied in simulated medical crisis scenarios.211
Consequences validity refers to the outcomes associated with scores from teamwork tools. For many tools (35, 48 %), outcomes included satisfaction or opinion of participants (Table 2). Twelve (16 %) tools measured participants’ teamwork skills. Teamwork skills such as leadership, communication and crisis management were assessed through simulation;41 , 138 , 209 however, other tools involved direct observation of skills in actual practice settings, such as medical residents’ abilities to lead ward teams213 and palliative care physicians’ communication in team meetings.195 Behaviors of students, residents/fellows, or practicing physicians were reported outcomes for ten (14 %) tools.
Teamwork Tools Associated with Patient Outcomes
Relationships between teamwork scores and patient outcomes have been directly examined for 13 (18 %) of teamwork tools (Table 3). Teamwork tools by Baggs83 , 210 and Wheelan155 show inverse relationships between positive teamwork and mortality rates.
Of the tools shown to correlate with patient outcomes, the Safety Attitudes Questionnaire (SAQ)113 has the strongest validity evidence, and has been adapted for use across multiple settings and learner levels. The SAQ contains six domains, one of which is teamwork. Twenty-seven studies have reported validity evidence for the SAQ.45 , 46 , 113 – 137 SAQ scores have been correlated with reduced postoperative complications;116 , 117 however, studies have not shown associations between the SAQ and mortality or patient safety events.116 , 118
The Team Climate Inventory (TCI) has been used to assess teamwork among inpatient and outpatient interprofessional teams in 21 studies.172 – 192 The TCI has four subscales: vision, participative safety, task orientation, and support for innovation.190 A study by Bower et al. found that ratings on the Team Climate Inventory (TCI) were associated with better diabetes care,172 while another study showed no relationship between the TCI and diabetes management.173
The Intensity of Interprofessional Collaboration Questionnaire is an instrument that measures the nurse–physician collaboration in the inpatient setting. Patients cared for by teams with high intensity collaboration on this scale reported higher satisfaction, lower uncertainty, and better pain management.96 However, there was no relationship between collaboration and patient length of stay.96
DISCUSSION
Assessing teamwork is imperative for determining physician competency11 , 13 , 221 and ensuring patient safety.3 , 5 Valid and reliable measurement of teamwork is necessary to understand connections between teamwork and patient safety, and to maximize gains achieved through teamwork education.
Together, the 178 studies and 73 teamwork tools summarized in this review constitute a resource for internists who wish to apply teamwork assessment tools to their local settings and teams. Although there is considerable validity evidence for many of these teamwork tools, most assessments consisted of participants’ subjective reports of satisfaction, attitude, or opinion. A thorough understanding of attitudes is prerequisite to improving teamwork; yet, tools examining teamwork behaviors in actual practice provide scores that may be more readily linked to important patient safety outcomes.83 , 155 , 202 Unfortunately, these assessments often require extensive rater training to achieve adequate reliability,105 which can be time consuming and costly. Implementing existing tools, rather than creating new ones, should reduce the cost of tool development so that these resources can be allocated to rater training and implementation. Furthermore, the trustworthiness of validity information depends upon the methodological quality of studies from which it is derived. Based on MERSQI criteria, further studies should aim to improve rigor of study design and outcome assessment.
Evidence suggests that teamwork training should improve patient safety,1 , 2 yet our review indicates that most studies examining teamwork in internal medicine do not directly link teamwork measures to reported patient outcomes. Several studies in this review described concurrent changes in patient outcomes and teamwork scores (e.g. pre/post teamwork training), but did not actually examine relationships between outcomes and teamwork scores, thus making it difficult to attribute gains in patient safety to teamwork improvements. To advance the understanding of how to improve safety through collaboration, future studies should not only apply valid teamwork assessments, but should directly examine relationships between these assessments and patient outcomes. Robust teamwork assessments and appropriate conceptual frameworks are essential to meaningful evaluations of relationships between teamwork and patient outcomes.
The majority of teamwork tools in this review were applied to groups of individuals working together to achieve a common goal within traditional team structures (e.g. physically side by side/face to face).23 , 222 However, the concept of ‘team’ in healthcare is rapidly evolving to include a greater emphasis on interprofessional collaboration,223 as well as new team structures. With the advent of restricted duty hours,224 and frequent hand-offs,29 , 30 team members are often working in shifts225 – 227 and are also becoming dispersed geographically. The telemedicine intensive care unit is an example in which intensivists and nurses use telemetry and electronic medical records to provide care to patients hospitalized remotely.228 Teams dispersed over distance and/or time face unique teamwork challenges119 that may require new or adapted assessment tools.
There are limitations to this review. First, although our search was comprehensive, we may have failed to capture some non-indexed or unpublished studies. We attempted to limit this possibility by reviewing abstracts from four professional meetings that are likely to include teamwork content, reviewing reference lists of included articles, and by having two content experts examine our reference list. Also, our electronic search included terms such as “registrar” that should have helped capture studies across countries. Second, to make the scope of the review manageable, it was limited to tools published in the field of internal medicine. However, some validity evidence was obtained from studies conducted in other specialties such as surgery and anesthesia. Validity is not a property of an instrument itself; rather it is a property of inferences derived from implementation of the instrument within specific contexts.34 As such, the setting in which tools are applied influences the validity information acquired. When selecting a tool for use in a new setting, it is important to consider the degree to which existing validity evidence may apply to the new context.
Third, this review included only quantitative measurement tools; however, qualitative studies provide valuable frameworks for understanding team behaviors and processes28 , 229 , 230 that are essential to the development of meaningful assessment tools. A synthesis of findings from the qualitative literature on teamwork would be a useful next step. Fourth, although we used an extremely broad definition and search strategy for teamwork, we excluded studies that examined just one specific element of interpersonal interaction, such as disruptive behavior and harassment. These behaviors alone do not constitute teamwork; however, they certainly may influence team interactions.231 – 233 Finally, we used established frameworks for abstracting tool validity32 , 33 and study quality;36 however, these frameworks do not encompass every aspect of validity and/or quality present in studies.
In conclusion, this systematic review provides a synthesis of teamwork assessment tools in internal medicine that may serve as a resource for educators who wish to assess teamwork for various learner levels and settings. Valid teamwork assessment is essential to determine physician competency and to ensure patient safety. Future research should expand the validity evidence for existing tools and further explore relationships between teamwork assessment and important patient safety outcomes.
REFERENCES
Baker DP, Gustafson S, Beaubien J, Salas E, Barach P. Medical teamwork and patient safety: The evidence-based relation. AHRQ Publication No. 05–0053. Rockville: Agency for Healthcare Research and Quality; 2005.
Leonard M, Graham S, Bonacum D. The human factor: the critical importance of effective teamwork and communication in providing safe care. Qual Saf Health Care. 2004;13(Suppl 1):i85–i90.
The Joint Commission. Improving America's hospitals—The Joint Commission's annual report on quality and safety-2007. Available at: http://www.jointcommission.org/Improving_Americas_Hospitals_The_Joint_Commissions_Annual_Report_on_Quality_and_Safety_-_2007/. Accessed October 10, 2013.
The Joint Commission. Sentinel event data: Root causes by event type. Available at: http://www.jointcommission.org/Sentinel_Event_Statistics/. Accessed October 10, 2013.
Greiner AC, Knebel E eds. Health professions education: a bridge to quality. Institute of Medicine, 2003. Available at: http://www.iom.edu/Reports/2003/health-professions-education-a-bridge-to-quality.aspx. Accessed October 10, 2013.
The Joint Commission. Improving patient and worker safety: opportunities for synergy, collaboration and innovation. Available at: http://www.jointcommission.org/assets/1/18/TJC-ImprovingPatientAndWorkerSafety-Monograph.pdf. Accessed October 10, 2013.
Clancy CM, Tornberg DN. TeamSTEPPS: assuring optimal teamwork in clinical settings. Am J Med Qual. 2007;22(3):214–217.
Schyve PM. Teamwork—the changing nature of professional competence. Jt Comm J Qual Patient Saf. 2005;31(4):183–184.
Clancy CM. Ten years after To Err is Human. Am J Med Qual. 2009;24(6):525–528.
American Board of Internal Medicine. Teamwork Effectiveness Assessment Module (TEAM). Available at: http://team.abim.org/. Accessed October 10, 2013.
Accreditation Council for Graduate Medical Education. Summative evaluation of competencies for 12-month clinical phase residents. Available at: http://www.acgme.org/acgmeweb/Portals/0/PFAssets/ProgramResources/380_SummativeEvaluation_GPM_AA_04_10_2008.pdf. Accessed October 10, 2013.
American Board of Internal Medicine. Milestones framework. Available at: http://www.abim.org/milestones/public. Accessed October 10, 2013.
Interprofessional Education Collaborative. Core competencies for interprofessional collaborative practice. 2011. Available at: http://www.aacn.nche.edu/education-resources/ipecreport.pdf Accessed October 10, 2013.
Leggat SG. Effective healthcare teams require effective team members: defining teamwork competencies. BMC Health Serv Res. 2007;7:17.
Baker DP, Salas E, King H, Battles J, Barach P. The role of teamwork in the professional education of physicians: current status and assessment recommendations. Jt Comm J Qual Patient Saf. 2005;31(4):185–202.
O'Leary KJ, Sehgal NL, Terrell G, Williams MV. Interdisciplinary teamwork in hospitals: A review and practical recommendations for improvement. Journal of Hospital Medicine (Online). Oct 31 2011.
Chakraborti C, Boonyasai RT, Wright SM, Kern DE. A systematic review of teamwork training interventions in medical student and resident education. J Gen Intern Med. 2008;23(6):846–853.
Davidson M, Smith RA, Dodd KJ, Smith JS, O'Loughlan MJ. Interprofessional pre-qualification clinical education: a systematic review. Aust Health Rev. 2008;32(1):111–120.
Hammick M, Freeth D, Koppel I, Reeves S, Barr H. A best evidence systematic review of interprofessional education: BEME Guide no. 9. Med Teach. 2007;29(8):735–751.
Remington TL, Foulk MA, Williams BC. Evaluation of evidence for interprofessional education. Am J Pharm Educ. 2006;70(3):66.
Reeves S, Zwarenstein M, Goldman J, et al. The effectiveness of interprofessional education: key findings from a new systematic review. J Interprof Care. 2010;24(3):230–241.
Buljac-Samardzic M, Dekker-van Doorn CM, van Wijngaarden JD, van Wijk KP. Interventions to improve team effectiveness: a systematic review. Health Policy. 2010;94(3):183–195.
Lemieux-Charles L, McGuire WL. What do we know about health care team effectiveness? A review of the literature. Med Care Res Rev. 2006;63(3):263–300.
Weaver SJ, Lyons R, DiazGranados D, et al. The anatomy of health care team training and the state of practice: a critical review. Acad Med. 2010;85(11):1746–1760.
Moher D, Liberati A, Tetzlaff J, Altman DG. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. Ann Intern Med. 2009;151(4):264–269. W264.
Kogan JR, Holmboe ES, Hauer KE. Tools for direct observation and assessment of clinical skills of medical trainees: a systematic review. JAMA. 2009;302(12):1316–1326.
Hammick M, Olckers L, Campion-Smith C. Learning in interprofessional teams: AMEE Guide no 38. Med Teach. 2009;31(1):1–12.
Xyrichis A, Ream E. Teamwork: a concept analysis. J Adv Nurs. 2008;61(2):232–241.
Riesenberg LA, Leitzsch J, Massucci JL, et al. Residents' and attending physicians' handoffs: a systematic review of the literature. Acad Med. 2009;84(12):1775–1787.
Arora VM, Manjarrez E, Dressler DD, Basaviah P, Halasyamani L, Kripalani S. Hospitalist handoffs: a systematic review and task force recommendations. J Hosp Med Online. 2009;4(7):433–440.
Ong MS, Coiera E. A systematic review of failures in handoff communication during intrahospital transfers. Jt Comm J Qual Patient Saf. 2011;37(6):274–284.
Downing SM. Validity: on meaningful interpretation of assessment data. Med Educ. 2003;37(9):830–837.
Messick S. Standards of validity and the validity of standards in performance assessment. Educ Meas Issues Pract. 1995;14(1):5–8.
Cook DA, Beckman TJ. Current concepts in validity and reliability for psychometric instruments: theory and application. Am J Med. 2006;119(2):166 e167–116.
Kirkpatrick D. Evaluation of training. In: Craig R, Bittel I, eds. Training and Development Handbook. New York: McGraw-Hill; 1976.
Reed DA, Cook DA, Beckman TJ, Levine RB, Kern DE, Wright SM. Association between funding and quality of published medical education research. JAMA. 2007;298(9):1002–1009.
Reed DA, Beckman TJ, Wright SM, Levine RB, Kern DE, Cook DA. Predictive validity evidence for medical education research study quality instrument scores: quality of submissions to JGIM's Medical Education Special Issue. J Gen Intern Med. 2008;23(7):903–907.
Zillich AJ, Milchak JL, Carter BL, Doucette WR. Utility of a questionnaire to measure physician-pharmacist collaborative relationships. J Am Pharm Assoc. 2006;46(4):453–458.
Frengley RW, Weller JM, Torrie J, et al. The effect of a simulation-based training intervention on the performance of established critical care unit teams. Crit Care Med. 2011;39(12):2605–2611.
Willaert W, Aggarwal R, Harvey K, et al. Efficient implementation of patient-specific simulated rehearsal for the carotid artery stenting procedure: part-task rehearsal. Eur J Vasc Endovasc Surg. 2011;42(2):158–166.
Boet S, Bould MD, Bruppacher HR, Desjardins F, Chandra DB, Naik VN. Looking in the mirror: self-debriefing versus instructor debriefing for simulated crises. Crit Care Med. 2011;39(6):1377–1381.
Jankouskas TS, Haidet KK, Hupcey JE, Kolanowski A, Murray WB. Targeted crisis resource management training improves performance among randomized nursing and medical students. Simul Healthc: J Soc Med Simul. 2011;6(6):316–326.
Welke TM, LeBlanc VR, Savoldelli GL, et al. Personalized oral debriefing versus standardized multimedia instruction after patient crisis simulation. Anesth Analg. 2009;109(1):183–189.
Zausig YA, Grube C, Boeker-Blum T, et al. Inefficacy of simulator-based training on anaesthesiologists' non-technical skills. Acta Anaesthesiol Scand. 2009;53(5):611–619.
O'Leary KJ, Wayne DB, Haviley C, Slade ME, Lee J, Williams MV. Improving teamwork: impact of structured interdisciplinary rounds on a medical teaching unit. J Gen Intern Med. 2010;25(8):826–832.
Riley W, Davis S, Miller K, Hansen H, Sainfort F, Sweet R. Didactic and simulation nontechnical skills team training to improve perinatal patient outcomes in a community hospital. Joint Commission Journal on Quality & Patient Safety. 2011;37(8):357–364.
Neal JM, Hsiung RL, Mulroy MF, Halpern BB, Dragnich AD, Slee AE. ASRA checklist improves trainee performance during a simulated episode of local anesthetic systemic toxicity. Reg Anesth Pain Med. 2012;37(1):8–15.
Plant JL, van Schaik SM, Sliwka DC, Boscardin CK, O'Sullivan PS. Validation of a self-efficacy instrument and its relationship to performance of crisis resource management skills. Adv Health Sci Educ. 2011;16(5):579–590.
Jankouskas T, Bush MC, Murray B, et al. Crisis resource management: evaluating outcomes of a multidisciplinary team. Simul Healthc: J Soc Med Simul. 2007;2(2):96–101.
Yee B, Naik VN, Joo HS, et al. Nontechnical skills in anesthesia crisis management with repeated exposure to simulation-based education. Anesthesiology. 2005;103(2):241–248.
Westli HK, Johnsen BH, Eid J, Rasten I, Brattebo G. Teamwork skills, shared mental models, and performance in simulated trauma teams: an independent group design. Scand J Trauma Resuscitation Emerg Med. 2010;18:47.
Fletcher G, Flin R, McGeorge P, Glavin R, Maran N, Patey R. Anaesthetists' Non-Technical Skills (ANTS): evaluation of a behavioural marker system. Br J Anaesth. 2003;90(5):580–588.
Muller MP, Hansel M, Fichtner A, et al. Excellence in performance and stress reduction during two different full scale simulator training courses: a pilot study. Resuscitation. 2009;80(8):919–924.
Fletcher G. Rating non-technical skills: developing a behavioural marker system for use in anaesthesia. Cogn Tech Work. 2004;6:165–171.
Graham J, Hocking G, Giles E. Anaesthesia non-technical skills: can anaesthetists be trained to reliably use this behavioural marker system in 1 day? Br J Anaesth. 2010;104(4):440–445.
Mudumbai SC, Gaba DM, Boulet JR, Howard SK, Davies MF. External validation of simulation-based assessments with other performance measures of third-year anesthesiology residents. Simul Healthc. 2012;7(2):73–80.
Lee JY, Mucksavage P, Canales C, McDougall EM, Lin S. High fidelity simulation based team training in urology: a preliminary interdisciplinary study of technical and nontechnical skills in laparoscopic complications management. J Urol. 2012;187(4):1385–1391.
Fulmer T, Hyer K, Flaherty E, et al. Geriatric Interdisciplinary Team Training Program. J Aging Health. 2005;17(4):443–470.
Saini B, Shah S, Kearey P, Bosnic-Anticevich S, Grootjans J, Armour C. An interprofessional learning module on asthma health promotion. Am J Pharm Educ. 2011;75(2):30.
Wamsley M, Staves J, Kroon L, et al. The impact of an interprofessional standardized patient exercise on attitudes toward working in interprofessional teams. J Interprof Care. 2012;26(1):28–35.
Lennon-Dearing R, Lowry LW, Ross CW, Dyer AR. An interprofessional course in bioethics: training for real-world dilemmas. J Interprof Care. 2009;23(6):574–585.
Leipzig RM, Hyer K, Ek K, et al. Attitudes toward working on interdisciplinary healthcare teams: a comparison by discipline. J Am Geriatr Soc. 2002;50(6):1141–1148.
Brown GF, Chamberlin GD. Attitudes toward quality, costs, and physician centrality in healthcare teams. J Interprof Care. 1996;10(1):63–72.
Novy D, Hamid B, Driver L, et al. Preliminary evaluation of an educational model for promoting positive team attitudes and functioning among pain medicine fellows. Pain Med. 2010;11(6):841–846.
Heinemann GD, Schmitt MH, Farrell MP, Brallier SA. Development of an Attitudes Toward Health Care Teams Scale. Eval Health Prof. 1999;22(1):123–142.
Hyer K, Fairchild S, Abraham I, Mezey M, Fulmer T. Measuring attitudes related to interdisciplinary training: revisiting the Heinemann, Schmitt and Farrell 'attitudes toward health care teams' scale. J Interprof Care. 2000;14(3):249–258.
Braithwaite J, Westbrook M, Nugus P, et al. A four-year, systems-wide intervention promoting interprofessional collaboration. BMC Health Serv Res. 2012;12:99.
Dillon PM, Noble KA, Kaplan L. Simulation as a means to foster collaborative interdisciplinary education. Nurs Educ Perspect. 2009;30(2):87–90.
Thomson S. Nurse-physician collaboration: a comparison of the attitudes of nurses and physicians in the medical-surgical patient care setting. MEDSURG Nurs. 2007;16(2):87–91.
Ardahan M, Akcasu B, Engin E. Professional collaboration in students of Medicine Faculty and School of Nursing. Nurse Educ Today. 2010;30(4):350–354.
Jones TS, Fitzpatrick JJ. CRNA-physician collaboration in anesthesia. AANA J. 2009;77(6):431–436.
Garber JS, Madigan EA, Click ER, Fitzpatrick JJ. Attitudes towards collaboration and servant leadership among nurses, physicians and residents. J Interprof Care. 2009;23(4):331–340.
Hughes B, Fitzpatrick JJ. Nurse-physician collaboration in an acute care community hospital. J Interprof Care. 2010;24(6):625–632.
Ward J, Schaal M, Sullivan J, Bowen ME, Erdmann JB, Hojat M. The Jefferson Scale of Attitudes toward Physician-Nurse Collaboration: a study with undergraduate nursing students. J Interprof Care. 2008;22(4):375–386.
Sterchi LS. Perceptions that affect physician-nurse collaboration in the perioperative setting. AORN J. 2007;86(1):45–57.
Hansson A, Arvemo T, Marklund B, Gedda B, Mattsson B. Working together—primary care doctors' and nurses' attitudes to collaboration. Scand J Public Health. 2010;38(1):78–85.
Yildirim A, Akinci F, Ates M, et al. Turkish version of the Jefferson Scale of Attitudes Toward Physician-Nurse Collaboration: a preliminary study. Contemp Nurse. 2006;23(1):38–45.
Taylor CL. Attitudes toward physician-nurse collaboration in anesthesia. AANA J. 2009;77(5):343–348.
McCaffrey R, Hayes RM, Cassell A, Miller-Reyes S, Donaldson A, Ferrell C. The effect of an educational programme on attitudes of nurses and medical residents towards the benefits of positive communication and collaboration. J Adv Nurs. 2012;68(2):293–301.
Armour Forse R, Bramble JD, McQuillan R. Team training can improve operating room performance. Surgery. 2011;150(4):771–778.
Beebe P, Bawel-Brinkley K, O'Leary-Kelley C. Observed and self-perceived teamwork in a rapid response team. J Nurses Staff Dev. 2012;28(4):191–197.
Posmontier B, Montgomery K, Smith Glasgow ME, Montgomery OC, Morse K. Transdisciplinary teamwork simulation in obstetrics-gynecology health care education. J Nurs Educ. 2012;51(3):176–179.
Baggs JG, Schmitt MH, Mushlin AI, et al. Association between nurse–physician collaboration and patient outcomes in three intensive care units. Crit Care Med. 1999;27(9):1991–1998.
Baggs JG. Development of an instrument to measure collaboration and satisfaction about care decisions. J Adv Nurs. 1994;20(1):176–182.
Baggs JG, Schmitt MH, Mushlin AI, Eldredge DH, Oakes D, Hutson AD. Nurse–physician collaboration and satisfaction with the decision-making process in three critical care units. Am J Crit Care. 1997;6(5):393–399.
Maxson PM, Dozois EJ, Holubar SD, et al. Enhancing nurse and physician collaboration in clinical decision making through high-fidelity interdisciplinary simulation training. Mayo Clin Proc. 2011;86(1):31–36.
Nathanson BH, Henneman EA, Blonaisz ER, Doubleday ND, Lusardi P, Jodka PG. How much teamwork exists between nurses and junior doctors in the intensive care unit? J Adv Nurs. 2011;67(8):1817–1823.
Kemper PF, de Bruijne M, van Dyck C, Wagner C. Effectiveness of classroom based crew resource management training in the intensive care unit: study design of a controlled trial. BMC Health Serv Res. 2011;11:304.
Strasser DC, Falconer JA, Martino-Saltzmann D. The rehabilitation team: staff perceptions of the hospital environment, the interdisciplinary team environment, and interprofessional relations. Arch Phys Med Rehabil. 1994;75(2):177–182.
Halstead LS, Rintala DH, Kanellos M, et al. The innovative rehabilitation team: an experiment in team building. Arch Phys Med Rehabil. 1986;67(6):357–361.
Verhoef J, Toussaint P, Putter H, Zwetsloot-Schonk J, Vlieland T. The impact of introducing an ICF-based rehabilitation tool on staff satisfaction with multidisciplinary team care in rheumatology: an exploratory study. Clin Rehabil. 2008;22(1):23–37.
Boyle DK, Kochinda C. Enhancing collaborative communication of nurse and physician leadership in two intensive care units. J Nurs Adm. 2004;34(2):60–70.
Hamric AB, Blackhall LJ. Nurse-physician perspectives on the care of dying patients in intensive care units: collaboration, moral distress, and ethical climate. Crit Care Med. 2007;35(2):422–429.
Hansen HE, Biros MH, Delaney NM, Schug VL. Research utilization and interdisciplinary collaboration in emergency care. Acad Emerg Med. 1999;6(4):271–279.
Shortell SM, Rousseau DM, Gillies RR, Devers KJ, Simons TL. Organizational assessment in intensive care units (ICUs): construct development, reliability, and validity of the ICU nurse–physician questionnaire. Med Care. 1991;29(8):709–726.
San Martin-Rodriguez L, D'Amour D, Leduc N. Outcomes of interprofessional collaboration for hospitalized cancer patients. Cancer Nurs. 2008;31(2):E18–E27.
Reader TW, Flin R, Mearns K, Cuthbertson BH. Interdisciplinary communication in the intensive care unit. Br J Anaesth. 2007;98(3):347–352.
Thomas EJ, Sexton JB, Helmreich RL. Discrepant attitudes about teamwork among critical care nurses and physicians. Crit Care Med. 2003;31(3):956–959.
Sexton J, Thomas EJ, Helmreich RL. Error, stress, and teamwork in medicine and aviation: cross sectional surveys. J Hum Perform Extreme Environ. 2001;6(1):6–11.
Flin R, Yule S, McKenzie L, Paterson-Brown S, Maran N. Attitudes to teamwork and safety in the operating theatre. Surg J R Coll Surg Edinb Irel. 2006;4(3):145–151.
Fleming M, Smith S, Slaunwhite J, Sullivan J. Investigating interpersonal competencies of cardiac surgery teams. Can J Surg. 2006;49(1):22–30.
Sexton JB, Thomas EJ, Helmreich RL. Error, stress, and teamwork in medicine and aviation: cross sectional surveys. BMJ. 2000;320(7237):745–749.
Morgan PJ, Pittini R, Regehr G, Marrs C, Haley MF. Evaluating teamwork in a simulated obstetric environment. Anesthesiology. 2007;106(5):907–915.
Flin R, Fletcher G, McGeorge P, Sutherland A, Patey R. Anaesthetists' attitudes to teamwork and safety. Anaesthesia. 2003;58(3):233–242.
Yule S, Rowley D, Flin R, et al. Experience matters: comparing novice and expert ratings of non-technical skills using the NOTSS system. ANZ J Surg. 2009;79(3):154–160.
Beard JD, Marriott J, Purdie H, Crossley J. Assessing the surgical skills of trainees in the operating theatre: a prospective observational study of the methodology. Health Technol Assess (Winchester, Engl). 2011;15(1):i–xxi. 1–162.
Arora S, Miskovic D, Hull L, et al. Self vs expert assessment of technical and non-technical skills in high fidelity simulation. Am J Surg. 2011;202(4):500–506.
Crossley J, Marriott J, Purdie H, Beard JD. Prospective observational study to evaluate NOTSS (Non-Technical Skills for Surgeons) for assessing trainees' non-technical performance in the operating theatre. Br J Surg. 2011;98(7):1010–1020.
Yule S, Flin R, Maran N, et al. Debriefing surgeons on non-technical skills (NOTS). Cogn Tech Work. 2008;10:265–274.
Keenan GM, Cooke R, Hillis SL. Norms and nurse management of conflicts: keys to understanding nurse–physician collaboration. Res Nurs Health. 1998;21(1):59–72.
Tschannen D, Kalisch BJ. The impact of nurse/physician collaboration on patient length of stay. J Nurs Manag. 2009;17(7):796–803.
Lurie SJ, Schultz SH, Lamanna G. Assessing teamwork: a reliable five-question survey. Fam Med. 2011;43(10):731–734.
Sexton JB, Helmreich RL, Neilands TB, et al. The Safety Attitudes Questionnaire: psychometric properties, benchmarking data, and emerging research. BMC Health Serv Res. 2006;6:44.
O'Leary KJ, Haviley C, Slade ME, Shah HM, Lee J, Williams MV. Improving teamwork: impact of structured interdisciplinary rounds on a hospitalist unit. J Hosp Med Online. 2011;6(2):88–93.
Mishra A, Catchpole K, McCulloch P. The Oxford NOTECHS System: reliability and validity of a tool for measuring teamwork behaviour in the operating theatre. Qual Saf Health Care. 2009;18(2):104–108.
Davenport DL, Henderson WG, Mosca CL, Khuri SF, Mentzer RM Jr. Risk-adjusted morbidity in teaching hospitals correlates with reported levels of communication and collaboration on surgical teams but not with scale measures of teamwork climate, safety climate, or working conditions. J Am Coll Surg. 2007;205(6):778–784.
Haynes AB, Weiser TG, Berry WR, et al. Changes in safety attitude and relationship to decreased postoperative morbidity and mortality following implementation of a checklist-based surgical safety intervention. BMJ Qual Saf. 2011;20(1):102–107.
Poley MJ, van der Starre C, van den Bos A, van Dijk M, Tibboel D. Patient safety culture in a Dutch pediatric surgical intensive care unit: an evaluation using the Safety Attitudes Questionnaire. Pediatr Crit Care Med. 2011;12(6):e310–e316.
Chu-Weininger MYL, Wueste L, Lucke JF, Weavind L, Mazabob J, Thomas EJ. The impact of a tele-ICU on provider attitudes about teamwork and safety climate. Qual Saf Health Care. 2010;19(6):e39.
Catchpole KR, Dale TJ, Hirst DG, Smith JP, Giddings TAEB. A multicenter trial of aviation-style training for surgical teams. J Patient Saf. 2010;6(3):180–186.
Haller G, Garnerin P, Morales M-A, et al. Effect of crew resource management training in a multidisciplinary obstetrical setting. Int J Qual Health Care. 2008;20(4):254–263.
Siassakos D, Fox R, Hunt L, et al. Attitudes toward safety and teamwork in a maternity unit with embedded team training. Am J Med Qual. 2011;26(2):132–137.
Sexton JB, Holzmueller CG, Pronovost PJ, et al. Variation in caregiver perceptions of teamwork climate in labor and delivery units. J Perinatol. 2006;26(8):463–470.
Pettker CM, Thung SF, Raab CA, et al. A comprehensive obstetrics patient safety program improves safety climate and culture. Am J Obstet Gynecol. 2011;204(3):216.e211–216.
Makary MA, Sexton JB, Freischlag JA, et al. Operating room teamwork among physicians and nurses: teamwork in the eye of the beholder. J Am Coll Surg. 2006;202(5):746–752.
Wolf FA, Way LW, Stewart L. The efficacy of medical team training: improved team performance and decreased operating room delays: a detailed analysis of 4,863 cases. Ann Surg. 2010;252(3):477–483. discussion 483–475.
Bognar A, Barach P, Johnson JK, et al. Errors and the burden of errors: attitudes, perceptions, and the culture of safety in pediatric cardiac surgical teams. Ann Thorac Surg. 2008;85(4):1374–1381.
Bethune R, Sasirekha G, Sahu A, Cawthorn S, Pullyblank A. Use of briefings and debriefings as a tool in improving team work, efficiency, and communication in the operating theatre. Postgrad Med J. 2011;87(1027):331–334.
Sexton JB, Makary MA, Tersigni AR, et al. Teamwork in the operating room: frontline perspectives among hospitals and operating room personnel. Anesthesiology. 2006;105(5):877–884.
Carney BT, West P, Neily JB, Mills PD, Bagian JP. Improving perceptions of teamwork climate with the Veterans Health Administration medical team training program. Am J Med Qual. 2011;26(6):480–484.
Siassakos D, Draycott TJ, Crofts JF, Hunt LP, Winter C, Fox R. More to teamwork than knowledge, skill and attitude. BJOG: Int J Obstet Gynaecol. 2010;117(10):1262–1269.
Carney BT, West P, Neily J, Mills PD, Bagian JP. Differences in nurse and surgeon perceptions of teamwork: implications for use of a briefing checklist in the OR. AORN J. 2010;91(6):722–729.
Hamman WR, Beaudin-Seiler BM, Beaubien JM. Understanding interdisciplinary health care teams: using simulation design processes from the Air Carrier Advanced Qualification Program to identify and train critical teamwork skills. J Patient Saf. 2010;6(3):137–146.
Pronovost PJ, Berenholtz SM, Goeschel C, et al. Improving patient safety in intensive care units in Michigan. J Crit Care. 2008;23(2):207–221.
Tam S, Kwai L, Ping S, Ka Man H, Diana. Assessing the safety and teamwork climate of intensive care staff. CONNECT: World Crit Care Nurs. 2011;8(1):21–25.
Timmel J, Kent PS, Holzmueller CG, Paine L, Schulick RD, Pronovost PJ. Impact of the Comprehensive Unit-based Safety Program (CUSP) on safety culture in a surgical inpatient unit. Jt Comm J Qual Patient Saf. 2010;36(6):252–260.
Bleakley A, Allard J, Hobbs A. Towards culture change in the operating theatre: embedding a complex educational intervention to improve teamwork climate. Med Teach. 2012;34(9):e635–e640.
Reid J, Stone K, Brown J, et al. The Simulation Team Assessment Tool (STAT): development, reliability and validation. Resuscitation. 2012;83(7):879–886.
Stocker M, Allen M, Pool N, et al. Impact of an embedded simulation team training programme in a paediatric intensive care unit: a prospective, single-centre, longitudinal study. Intensive Care Med. 2012;38(1):99–104.
O'Leary KJ, Ritter CD, Wheeler H, Szekendi MK, Brinton TS, Williams MV. Teamwork on inpatient medical units: assessing attitudes and barriers. Qual Saf Health Care. 2010;19(2):117–121.
Gittell JH, Weinberg DB, Bennett AL, Miller JA. Is the doctor in? A relational approach to job design and the coordination of work. Hum Resour Manag. 2008;47(4):729–755.
Leipzig RM, Berkman CS, Ramirez-Coronado S, Pignotti M. Integrating housestaff into a geriatric inpatient interdisciplinary team. Gerontol Geriatr Educ. 2001;21(3):63–72.
Carbo AR, Tess AV, Roy C, Weingart SN. Developing a high-performance team training framework for internal medicine residents: the ABC'S of teamwork. J Patient Saf. 2011;7(2):72–76.
Weller J, Frengley R, Torrie J, et al. Evaluation of an instrument to measure teamwork in multidisciplinary critical care teams. BMJ Qual Saf. 2011;20(3):216–222.
Stahelski AJ, Tsukuda RA. Predictors of cooperation in health care teams. Small Group Res. 1990;21(2):220–233.
Chaboyer WP, Patterson E. Australian hospital generalist and critical care nurses' perceptions of doctor-nurse collaboration. Nurs Health Sci. 2001;3(2):73–79.
King L, Lee JL. Perceptions of collaborative practice between Navy nurses and physicians in the ICU setting. Am J Crit Care. 1994;3(5):331–336.
Baldwin A, Welches L, Walker DD, Eliastam M. Nurse self-esteem and collaboration with physicians. West J Nurs Res. 1987;9(1):107–114.
Weiss SJ, Davis HP. Validity and reliability of the Collaborative Practice Scales. Nurs Res. 1985;34(5):299–305.
Jones RA. Nurse–physician collaboration: a descriptive study. Holist Nurs Pract. 1994;8(3):38–53.
Torgersen KA, Chamings PA. Examining collaborative relationships between anesthesiologists and certified registered nurse anesthetists in nurse anesthesia educational programs. AANA J. 1994;62(2):139–148.
Maylone MM, Ranieri L, Quinn Griffin MT, McNulty R, Fitzpatrick JJ. Collaboration and autonomy: perceptions among nurse practitioners. J Am Acad Nurse Pract. 2011;23(1):51–57.
Berendsen AJ, Benneker WHGM, Groenier KH, Schuling J, Grol RPTM, Meyboom-de JB. DOC questionnaire: measuring how GPs and medical specialists rate collaboration. Int J Health Care Qual Assur. 2010;23(5):516–526.
Kunkler I, Fielding G, Macnab M, et al. Group dynamics in telemedicine-delivered and standard multidisciplinary team meetings: Results from the TELEMAM randomised trial. J Telemed Telecare. 2006;12(Suppl 3):55–58.
Wheelan SA, Burchill CN, Tilin F. The link between teamwork and patients' outcomes in intensive care units. Am J Crit Care. 2003;12(6):527–534.
Upenieks VV, Lee EA, Flanagan ME, Doebbeling BN. Healthcare Team Vitality Instrument (HTVI): developing a tool assessing healthcare team functioning. J Adv Nurs. 2010;66(1):168–176.
Grogan EL, Stiles RA, France DJ, et al. The impact of aviation-based teamwork training on the attitudes of health-care professionals. J Am Coll Surg. 2004;199(6):843–848.
Newton C, Wood V, Nasmith L. Building capacity for interprofessional practice. Clin Teach. 2012;9(2):94–98.
Jones RA, Marham S, Ricard RJ. Multidisciplinary collaboration: a method for measurement. Nurs Adm Q. 1999;23(4):86–90.
Lamb BW, Sevdalis N, Mostafid H, Vincent C, Green JSA. Quality improvement in multidisciplinary cancer teams: an investigation of teamwork and clinical decision-making and cross-validation of assessments. Ann Surg Oncol. 2011;18(13):3535–3543.
Lamb BW, Wong HWL, Vincent C, Green JSA, Sevdalis N. Teamwork and team performance in multidisciplinary cancer teams: development and evaluation of an observational assessment tool. BMJ Qual Saf. 2011;20(10):849–856.
Berkowitz RE, Schreiber R, Paasche-Orlow MK. Team improvement and patient safety conferences: culture change and slowing the revolving door between skilled nursing facility and the hospital. J Nurs Care Qual. 2012;27(3):258–265.
Walker S, Brett S, McKay A, Lambden S, Vincent C, Sevdalis N. Observational Skill-based Clinical Assessment tool for Resuscitation (OSCAR): development and validation. Resuscitation. 2011;82(7):835–844.
Zillich AJ, Doucette WR, Carter BL, Kreiter CD. Development and initial validation of an instrument to measure physician–pharmacist collaboration from the physician perspective. Value Health. 2005;8(1):59–66.
Doucette WR, Nevins J, McDonough RP. Factors affecting collaborative care between pharmacists and physicians. Res Soc Adm Pharm. 2005;1(4):565–578.
Snyder ME, Zillich AJ, Primack BA, et al. Exploring successful community pharmacist–physician collaborative working relationships using mixed methods. Res Soc Adm Pharm. 2010;6(4):307–323.
Brock KA, Doucette WR. Collaborative working relationships between pharmacists and physicians: an exploratory study. J Am Pharm Assoc: JAPhA. 2004;44(3):358–365.
Liu Y, Doucette WR. Exploring stages of pharmacist–physician collaboration using the model of collaborative working relationship. J Am Pharm Assoc: JAPhA. 2011;51(3):412–417.
Liu Y, Doucette WR, Farris KB. Examining the development of pharmacist–physician collaboration over 3 months. Res Soc Adm Pharm. 2010;6(4):324–333.
Zillich AJ, McDonough RP, Carter BL, Doucette WR. Influential characteristics of physician/pharmacist collaborative relationships. Ann Pharmacother. 2004;38(5):764–770.
Becker ER, Roblin DW. Translating primary care practice climate into patient activation: the role of patient trust in physician. Med Care. 2008;46(8):795–805.
Bower P, Campbell S, Bojke C, Sibbald B. Team structure, team climate and the quality of care in primary care: an observational study. Qual Saf Health Care. 2003;12(4):273–279.
Hann M, Bower P, Campbell S, Marshall M, Reeves D. The association between culture, climate and quality of care in primary health care teams. Fam Pract. 2007;24(4):323–329.
Proudfoot J, Jayasinghe UW, Holton C, et al. Team climate for innovation: what difference does it make in general practice? Int J Qual Health Care. 2007;19(3):164–169.
Gosling AS, Westbrook JI, Braithwaite J. Clinical team functioning and IT innovation: a study of the diffusion of a point-of-care online evidence system. J Am Med Inform Assoc. 2003;10(3):244–251.
Howard M, Brazil K, Akhtar-Danesh N, Agarwal G. Self-reported teamwork in family health team practices in Ontario: organizational and cultural predictors of team climate. Can Fam Physician. 2011;57(5):e185–e191.
Watts F, Lindqvist S, Pearce S, Drachler M, Richardson B. Introducing a post-registration interprofessional learning programme for healthcare teams. Med Teach. 2007;29(5):457–463.
Harris MF, Proudfoot JG, Jayasinghe UW, et al. Job satisfaction of staff and the team environment in Australian general practice. Med J Aust. 2007;186(11):570–573.
Watkin A, Lindqvist S, Black J, Watts F. Report on the implementation and evaluation of an interprofessional learning programme for inter-agency child protection teams. Child Abuse Rev. 2009;18(3):151–167.
West MA, Poulton BC. A failure of function: Teamwork in primary health care. J Interprof Care. 1997;11(2):205–216.
Carpenter J, Schneider J, Brandon T, Wooff D. Working in Multidisciplinary Community Mental Health Teams: The Impact on Social Workers and Health Professionals of Integrated Mental Health Care. Br J Soc Work. 2003;33(8):1081–1103.
Poulton BC, West MA. The determinants of effectiveness in primary health care teams. J Interprof Care. 1999;13(1):7–18.
Haward R, Amir Z, Borrill C, et al. Breast cancer teams: the impact of constitution, new cancer workload, and methods of operation on their effectiveness. Br J Cancer. 2003;89(1):15–22.
Kivimaki M, Kuk G, Elovainio M, Thomson L, Kalliomaki-Levanto T, Heikkila A. The Team Climate Inventory (TCI)—four or five factors? Testing the structure of TCI in samples of low and high complexity jobs. J Occup Organ Psychol. 1997;70(4):375–389.
Bosch M, Halfens RJG, van der Weijden T, Wensing M, Akkermans R, Grol R. Organizational culture, team climate, and quality management in an important patient safety issue: nosocomial pressure ulcers. Worldviews Evid-Based Nurs. 2011;8(1):4–14.
Elovainio M, Kivimaki M, Eccles M, Sinervo T. Team climate and procedural justice as predictors of occupational strain. J Appl Soc Psychol. 2002;32(2):359–374.
Goh TT, Eccles MP, Steen N. Factors predicting team climate, and its relationship with quality of care in general practice. BMC Health Serv Res. 2009;9:138.
Gibbon B, Watkins C, Barer D, et al. Can staff attitudes to team working in stroke care be improved? J Adv Nurs. 2002;40(1):105–111.
Kivimaki M, Vanhala A, Pentti J, et al. Team climate, intention to leave and turnover among hospital employees: prospective cohort study. BMC Health Serv Res. 2007;7:170.
Anderson NR, West MA. Measuring climate for work group innovation: development and validation of the team climate inventory. J Organ Behav. 1998;19(3):235–258.
Ouwens M, Hulscher M, Akkermans R, Hermens R, Grol R, Wollersheim H. The Team Climate Inventory: application in hospital teams and methodological considerations. Qual Saf Health Care. 2008;17(4):275–280.
Kivimaki M, Elovainio M. A shorter version of the Team Climate Inventory: Development and psychometric properties. J Occup Organ Psychol. 1999;72(2):241–246.
Cooper S, Cant R, Porter J, et al. Rating medical emergency teamwork performance: development of the Team Emergency Assessment Measure (TEAM). Resuscitation. 2010;81(4):446–452.
Sutton G, Liao J, Jimmieson NLD. Measuring Multidisciplinary Team Effectiveness in a Ward-Based Healthcare Setting: Development of the Team Functioning Assessment Tool. J Healthc Qual: Promot Excell Healthc. 2011;33(3):10–24.
Wittenberg-Lyles E, Oliver DP, Demiris G, Burt S, Regehr K. Inviting the absent members: examining how caregivers' participation affects hospice team communication. Palliat Med. 2010;24(2):192–195.
Cole KD, Waite MS, Nichols LO. Organizational structure, team process, and future directions of interprofessional health care teams. Gerontol Geriatr Educ. 2003;24(2):35–49.
Baker DP, Amodeo AM, Krokos KJ, Slonim A, Herrera H. Assessing teamwork attitudes in healthcare: development of the TeamSTEPPS teamwork attitudes questionnaire. Qual Saf Health Care. 2010;19(6):e49.
Mayer CM, Cluff L, Lin W-T, et al. Evaluating efforts to optimize TeamSTEPPS implementation in surgical and pediatric intensive care units. Jt Comm J Qual Patient Saf. 2011;37(8):365–374.
Lown BA, Manning CF. The Schwartz Center Rounds: evaluation of an interdisciplinary approach to enhancing patient-centered communication, teamwork, and provider support. Acad Med. 2010;85(6):1073–1081.
Hyer K, Skinner JH, Kane RL, et al. Using scripted video to assess interdisciplinary team effectiveness training outcomes. Gerontol Geriatr Educ. 2003;24(2):75–91.
Lukas CV, Mohr DC, Meterko M. Team effectiveness and organizational context in the implementation of a clinical innovation. Qual Manag Health Care. 2009;18(1):25–39.
Strasser DC, Falconer JA, Herrin JS, Bowen SE, Stevens AB, Uomoto J. Team functioning and patient outcomes in stroke rehabilitation. Arch Phys Med Rehabil. 2005;86(3):403–409.
Krogstad U, Hofoss D, Hjortdahl P. Doctor and nurse perception of inter-professional co-operation in hospitals. Int J Qual Health Care. 2004;16(6):491–497.
McLaughlin T, Hennecke P, Garraway NR, et al. A predeployment trauma team training course creates confidence in teamwork and clinical skills: a post-Afghanistan deployment validation study of Canadian Forces healthcare personnel. J Trauma-Inj Infect Crit Care. 2011;71(5 Suppl 1):S487–S493.
Roblin DW, Howard DH, Junling R, Becker ER. An evaluation of the influence of primary care team functioning on the health of Medicare beneficiaries. Med Care Res Rev. 2011;68(2):177–201.
Kydona CK, Malamis G, Giasnetsova T, Tsiora V, Gritsi-Gerogianni N. The level of teamwork as an index of quality in ICU performance. Hippokratia. 2010;14(2):94–97.
Auerbach AD, Sehgal NL, Blegen MA, et al. Effects of a multicentre teamwork and communication programme on patient outcomes: results from the Triad for Optimal Patient Safety (TOPS) project. BMJ Qual Saf. 2012;21(2):118–126.
Aboul-Fotouh AM, Ismail NA, Ez Elarab HS, Wassif GO. Assessment of patient safety culture among healthcare providers at a teaching hospital in Cairo. Egypt East Mediterr Health J. 2012;18(4):372–377.
Curran VR, Mugford JG, Law RMT, MacDonald S. Influence of an interprofessional HIV/AIDS education program on role perception, attitudes and teamwork skills of undergraduate health sciences students. Educ Health. 2005;18(1):32–44.
Baggs JG, Ryan SA, Phelps CE, Richeson JF, Johnson JE. The association between interdisciplinary collaboration and patient outcomes in a medical intensive care unit. Heart Lung. 1992;21(1):18–24.
Kim J, Neilipovitz D, Cardinal P, Chiu M. A comparison of global rating scale and checklist scores in the validation of an evaluation tool to assess performance in the resuscitation of critically ill patients during simulated emergencies (abbreviated as "CRM simulator study IB"). Simul Healthc: J Soc Med Simul. 2009;4(1):6–16.
Balki M, Cooke ME, Dunington S, Salman A, Goldszmidt E. Unanticipated difficult airway in obstetric patients: development of a new algorithm for formative assessment in high-fidelity simulation. Anesthesiology. 2012;117(4):883–897.
Orlander JD, Wipf JE, Lew RA. Development of a tool to assess the team leadership skills of medical residents. Med Educ Online. 2006;11:1–6.
Adler-Milstein J, Neal K, Howell MD. Residents' and nurses' perceptions of team function in the medical intensive care unit. J Crit Care. 2011;26(1):104.e107–115.
Hansel M, Winkelmann AM, Hardt F, et al. Impact of simulator training and crew resource management training on final-year medical students' performance in sepsis resuscitation: a randomized trial. Minerva Anestesiol. 2012;78(8):901–909.
Agency for Healthcare Research and Quality. TeamSTEPPS: National implementation. Available at: http://teamstepps.ahrq.gov/. Accessed October 10, 2013.
Undre S, Healey AN, Darzi A, Vincent CA. Observational assessment of surgical teamwork: a feasibility study. World J Surg. 2006;30(10):1774–1783.
Undre S, Sevdalis N, Healey AN, Darzi A, Vincent CA. Observational teamwork assessment for surgery (OTAS): refinement and application in urological surgery. World J Surg. 2007;31(7):1373–1381.
Hull L, Arora S, Kassab E, Kneebone R, Sevdalis N. Observational teamwork assessment for surgery: content validation and tool refinement. J Am Coll Surg. 2011;212(2):234–243. e231-235.
Landis JRKG. The measure of observer agreement for categorical data. Biometrics. 1977;33(1):159–174.
American Board of Medical Specialties. Leading the National Movement for Healthcare Quality. The ABMS Patient Safety Improvement Program. Available at: http://www.abms.org/who_we_help/physicians/improving_quality.aspx. Accessed October 10, 2013.
Salas E, DiazGranados D, Weaver SJ, King H. Does team training work? Principles for health care. Acad Emerg Med. 2008;15(11):1002–1009.
Barrow M. Conflict in context: designing authentic teamwork education. Med Educ. 2012;46(10):926–927.
Nasca TJ, Day SH, Amis ES Jr. The new recommendations on duty hours from the ACGME Task Force. N Engl J Med. 2010;363(2):e3.
Arora VM, Farnan JM, Humphrey HJ. Professionalism in the era of duty hours:time for a shift change? JAMA. 2012;308(21):2195–2196.
Arora VM, Reed DA, Fletcher KE. Building continuity in handovers with shorter residency duty hours. BMC Med Educ. 2013 In Press.
Reed DA, Fletcher KE, Arora VM. Systematic review: association of shift length, protected sleep time, and night float with patient care, residents' health, and education. Ann Intern Med. 2010;153(12):829–842.
Young LB, Chan PS, Lu X, Nallamothu BK, Sasson C, Cram PM. Impact of telemedicine intensive care unit coverage on patient outcomes: a systematic review and meta-analysis. Arch Intern Med. 2011;171(6):498–506.
Reeves S, Russell A, Zwarenstein M, et al. Structuring communication relationships for interprofessional teamwork (SCRIPT): a Canadian initiative aimed at improving patient-centred care. J Interprof Care. 2007;21(1):111–114.
Kyrkjebo JM, Brattebo G, Smith-Strom H. Improving patient safety by using interprofessional simulation training in health professional education. J Interprof Care. 2006;20(5):507–516.
Saxton R, Hines T, Enriquez M. The negative impact of nurse–physician disruptive behavior on patient safety: a review of the literature. J Patient Saf. 2009;5(3):180–183.
Uhari M, Kokkonen J, Nuutinen M, et al. Medical student abuse: an international phenomenon. JAMA. 1994;271(13):1049–1051.
Nagata-Kobayashi S, Maeno T, Yoshizu M, Shimbo T. Universal problems during residency: abuse and harassment. Med Educ. 2009;43(7):628–636.
Acknowledgements
Contributors
Dr. Paul Haidet of Pennsylvania State Hershey College of Medicine and Dr. Chayan Chakraborti, MD FACP of Tulane University School of Medicine conducted expert reviews of the reference list. Karen Larsen of Mayo Clinic assisted with the electronic searches. Kathy Thompson of Mayo Clinic assisted with article retrieval.
Funding /Support
This study was supported in part by the Mayo Clinic Program in Professionalism and Ethics and the Mayo Clinic-Rochester Internal Medicine Residency Office of Educational Innovations as part of the ACGME Educational Innovations Project.
Role of the Sponsor
The funding sources had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; or preparation, review, or approval of the manuscript.
Prior Presentations
This review was presented in an oral session at the Society of General Internal Medicine Annual Meeting in Denver, Colorado in April 2013.
Conflict of Interest
The authors declare that they do not have a conflict of interest.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Havyer, R.D.A., Wingo, M.T., Comfere, N.I. et al. Teamwork Assessment in Internal Medicine: A Systematic Review of Validity Evidence and Outcomes. J GEN INTERN MED 29, 894–910 (2014). https://doi.org/10.1007/s11606-013-2686-8
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11606-013-2686-8