Reliability of Peer Review of Abstracts Submitted to Academic Family Medicine Meetings

References

1. 1.↵
   1. Kemper KJ,
   2. McCarthy PL,
   3. Cicchetti DV

   . Improving participation and interrater agreement in scoring Ambulatory Pediatric Association abstracts. How well have we succeeded? Arch Pediatr Adolesc Med 1996;150:380–3.

   OpenUrlCrossRefPubMedWeb of Science
2. 2.↵
   1. Montgomery AA,
   2. Graham A,
   3. Evans PH,
   4. Fahey T

   . Inter-rater agreement in the scoring of abstracts submitted to a primary care research conference. BMC Health Serv Res 2002;2:8.

   OpenUrlCrossRefPubMed
3. 3.↵
   1. Poolman RW,
   2. Keijser LC,
   3. de Waal Malefijt MC,
   4. Blankevoort L,
   5. Farrokhyar F,
   6. Bhandari M

   . Reviewer agreement in scoring 419 abstracts for scientific orthopedics meetings. Acta Orthop 2007;78:278–84.

   OpenUrlCrossRefPubMed
4. 4.↵
   1. Rowe BH,
   2. Strome TL,
   3. Spooner C,
   4. Blitz S,
   5. Grafstein E,
   6. Worster A

   . Reviewer agreement trends from four years of electronic submissions of conference abstract. BMC Med Res Methodol 2006;6:14.

   OpenUrlCrossRefPubMed
5. 5.↵
   1. Rubin HR,
   2. Redelmeier DA,
   3. Wu AW,
   4. Steinberg EP

   . How reliable is peer review of scientific abstracts? Looking back at the 1991 Annual Meeting of the Society of General Internal Medicine. J Gen Intern Med 1993;8:255–8.

   OpenUrlCrossRefPubMedWeb of Science
6. 6.↵
   1. Ingelfinger FJ

   . Peer review in biomedical publication. Am J Med 1974;56:686–92.

   OpenUrlCrossRefPubMedWeb of Science
7. 7.↵
   1. Kravitz RL,
   2. Franks P,
   3. Feldman MD,
   4. Gerrity M,
   5. Byrne C,
   6. Tierney WM

   . Editorial peer reviewers’ recommendations at a general medical journal: are they reliable and do editors care? PloS one 2010;5:e10072.

   OpenUrlCrossRefPubMed
8. 8.↵
   1. Fogelholm M,
   2. Leppinen S,
   3. Auvinen A,
   4. Raitanen J,
   5. Nuutinen A,
   6. Vaananen K

   . Panel discussion does not improve reliability of peer review for medical research grant proposals. J Clin Epidemiol 2012;65:47–52.

   OpenUrlCrossRefPubMed
9. 9.↵
   1. Pier EL,
   2. Brauer M,
   3. Filut A,
   4. et al

   . Low agreement among reviewers evaluating the same NIH grant applications. Proc Natl Acad Sci USA 2018;115:2952–7.

   OpenUrlAbstract/FREE Full Text
10. 10.↵
    1. Reinhart M

    . Peer review of grant applications in biology and medicine. Reliability, fairness, and validity. Scientometrics 2009;81:789–809.

    OpenUrlCrossRef
11. 11.↵
    1. Mayo NE,
    2. Brophy J,
    3. Goldberg MS,
    4. et al

    . Peering at peer review revealed high degree of chance associated with funding of grant applications. J Clin Epidemiol 2006;59:842–8.

    OpenUrlCrossRefPubMed
12. 12.↵
    1. Carrasco JL,
    2. Jover L

    . Estimating the generalized concordance correlation coefficient through variance components. Biometrics 2003;59:849–58.

    OpenUrlCrossRefPubMedWeb of Science
13. 13.↵
    1. Kraemer HC

    . Correlation coefficients in medical research: from product moment correlation to the odds ratio. Stat Methods Med Res 2006;15:525–45.

    OpenUrlCrossRefPubMedWeb of Science
14. 14.↵
    1. Shrout PE,
    2. Fleiss JL

    . Intraclass correlations: uses in assessing rater reliability. Psychol Bull 1979;86:420–8.

    OpenUrlCrossRefPubMedWeb of Science
15. 15.↵
    1. Carmines EG,
    2. Zeller RA

    , eds. Reliability and validity assessment. Newbury Park, CA: Sage Publications; 1979.
16. 16.↵
    1. McDowell I

    . Measuring health: a guide to rating scales and questionnaires. 3rd ed. New York, NY: Oxford University Press; 2006.
17. 17.↵
    1. Nelson EC,
    2. Gentry MA,
    3. Mook KH,
    4. Spritzer KL,
    5. Higgins JH,
    6. Hays RD

    . How many patients are needed to provide reliable evaluations of individual clinicians? Med Care 2004;42:259–66.

    OpenUrlCrossRefPubMedWeb of Science
18. 18.↵
    1. Sattler DN,
    2. McKnight PE,
    3. Naney L,
    4. Mathis R

    . Grant peer review: improving inter-rater reliability with training. PloS One 2015;10:e0130450.

    OpenUrl
19. 19.↵
    1. Kaplan D,
    2. Lacetera N,
    3. Kaplan C

    . Sample size and precision in NIH peer review. PloS One 2008;3:e2761.

    OpenUrlCrossRefPubMed