Reliability of Peer Review of Abstracts Submitted to Academic Family Medicine Meetings

Joshua J. Fenton; Hazel Tapp; Netra M. Thakur; Andrea L. Pfeifle

doi:10.3122/jabfm.2020.06.200123

Article Figures & Data

Tables

View popup

Table 1.

Reliabilities of Abstract Reviews by Family Medicine Meeting, Year, and Submission Category

Submission by Year, Meeting, Category	N, Submissions	Number of Reviews, Mean	Intracluster Correlation Coefficients (95% CI)		Reliability of Within-Abstract Mean Score (95% CI)
Submission by Year, Meeting, Category	N, Submissions	Number of Reviews, Mean	Abstract-Level	Reviewer-Level	Reliability of Within-Abstract Mean Score (95% CI)
North American Primary Care Research Group
2016 oral presentation on completed research	392	2.0	0.18 (0.09-0.26)	0.22 (0.15-0.30)	0.30 (0.17-0.41)
2017 oral presentation on completed research	400	2.0	0.15 (0.08-0.24)	0.25 (0.17-0.33)	0.26 (0.15-0.39)
2018 oral presentation on completed research	388	2.0	0.17 (0.08-0.26)	0.15 (0.08-0.26)	0.29 (0.15-0.41)
2019 oral presentation on completed research	374	3.0	0.10 (0.03-0.16)	—*	0.24 (0.08-0.36)
2019 poster on completed research	139	2.0	0.03 (0.00-0.21)	—*	0.07 (0.00-0.34)
2019 poster on research in progress	238	2.0	0.11 (0.00-0.24)	—*	0.18 (0.00-0.38)
2019 workshop	41	2.9	0.00 (0.00-0.18)	—*	0.00 (0.00-0.39)
Society of Teachers of Family Medicine, 2018 Spring Meeting
Completed research poster	30	3.0	0.33 (0.11-0.55)	0.11 (0.00-0.27)	0.60 (0.27-0.79)
Completed research project	75	3.0	0.50 (0.37-0.63)	0.08 (0.00-0.16)	0.75 (0.64-0.84)
Completed scholarly project	37	2.9	0.21 (0.02-0.41)	0.34 (0.00-0.72)	0.44 (0.06-0.67)
Completed scholarly project poster	49	2.5	0.18 (0.00-0.40)	0.15 (0.00-0.35)	0.35 (0.00-0.63)
Developing scholarly project poster	112	2.3	0.13 (0.02-0.24)	0.37 (0.16-0.58)	0.25 (0.04-0.42)
Trainee work-in-progress poster	244	2.0	0.27 (0.16-0.38)	0.22 (0.04-0.40)	0.42 (0.28-0.55)
Lecture discussion	238	2.9	0.17 (0.10-0.25)	0.22 (0.06-0.38)	0.37 (0.24-0.49)
Panel discussion	20	3.0	0.20 (0.00-0.44)	0.19 (0.00-0.48)	0.42 (0.00-0.70)
Pre-conference workshop	14	7.0	0.34 (0.12-0.56)	0.16 (0.00-0.35)	0.78 (0.49-0.90)
Scholarly topic roundtable discussion	89	2.9	0.12 (0.01-0.24)	0.16 (0.01-0.31)	0.28 (0.03-0.48)
Seminar	180	3.0	0.16 (0.07-0.24)	0.15 (0.00-0.29)	0.36 (0.18-0.49)
Workshop	57	3.0	0.33 (0.17-0.49)	0.10 (0.00-0.26)	0.60 (0.38-0.74)

↵* Reviewer identifiers were not available for the 2019 North American Primary Care Research Group meeting, preventing estimation of reviewer-level ICCs.
CI, confidence interval; ICCs, intracluster correlation coefficients.