Skip to main content
SpringerLink
Log in

How reliable is peer review of scientific abstracts?

Looking back at the 1991 annual meeting of the society of general internal medicine

  • Original Articles
  • Published:
Journal of General Internal Medicine Aims and scope Submit manuscript

Abstract

Objective: To evaluate the interrater reproducibility of scientific abstract review.

Design: Retrospective analysis.

Setting: Review for the 1991 Society of General Internal Medicine (SGIM) annual meeting.

Subjects: 426 abstracts in seven topic categories evaluated by 55 reviewers.

Measurements: Reviewers rated abstracts from 1 (poor) to 5 (excellent), globally and on three specific dimensions: interest to the SGIM audience, quality of methods, and quality of presentation. Each abstract was reviewed by five to seven reviewers. Each reviewer’s ratings of the three dimensions were added to compute that reviewer’ssummary score for a given abstract. The mean of all reviewers’ summary scores for an abstract, thefinal score, was used by SGIM to select abstracts for the meeting.

Results: Final scores ranged from 4.6 to 136 (mean=9.9). Although 222 abstracts (5296) were accepted for publication, the 95% confidence interval around the final score of 300 (70.4%) of the 426 abstracts overlapped with the threshold for acceptance of an abstract. Thus, these abstracts were potentially misclassified. Only 36% of the variance in summary scores was associated with an abstract’s identity, 12% with the reviewer’s identity, and the remainder with idiosyncratic reviews of abstracts. Global ratings were more reproducible than summary scores.

Conclusion: Reviewers disagreed substantially when evaluating the same abstracts. Future meeting organizers may wish to rank abstracts using global ratings, and to experiment with structured review criteria and other ways to improve raters’ agreement.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Fletcher CM, Oldham PD. Bibliography on observer error and variation. In: Witt LJ (ed). Medical surveys and clinical trials, 2nd ed. New York: Oxford University Press, 1964.

    Google Scholar 

  2. Koran LM. The reliability of clinical methods, data and judgments. N Engl J Med. 1975;293:642–6, 695–701.

    Article  PubMed  CAS  Google Scholar 

  3. Cole S, Cole JR, Simon GA. Chance and consensus in peer review. Science. 1981;214:881–6.

    Article  PubMed  CAS  Google Scholar 

  4. Feinstein AR. A bibliography of publications on observer variability. J Chron Dis. 1985;38:619–32.

    Article  PubMed  CAS  Google Scholar 

  5. Smith R. Problems with peer review and alternatives. BMJ. 1988;296:774–7.

    PubMed  CAS  Google Scholar 

  6. Green JG, Calhoun F, Nierzwicki L, Brackett J, Meier P. Rating intervals: an experiment in peer review. FASEB J. 1989;3:2987–92.

    Google Scholar 

  7. McNutt RA, Evans AT, Fletcher RH, Fletcher SW. The effects of blinding on the quality of peer review: a randomized trial. JAMA. 1990;263:1371–6.

    Article  PubMed  CAS  Google Scholar 

  8. Cicchetti DV, Conn HO. A statistical analysis of reviewer agreement and bias in evaluating medical abstracts. Yale J Biol Med. 1976;49:373–83.

    PubMed  CAS  Google Scholar 

  9. Bennett KJ, Sackett DL, Haynes RB, Neufeld VR, Tugwell P, Roberts R. A controlled trial of teaching critical appraisal of the clinical literature to medical students. JAMA. 1987;257:2451–4.

    Article  PubMed  CAS  Google Scholar 

  10. Haynes RB, McKibbon KA, Fitzgerald D, Guyatt GH, Walker CJ, Sackett DL. How to keep up with the medical literature: why try to keep up and how to get started. Ann Intern Med. 1986;105:149–53.

    PubMed  CAS  Google Scholar 

  11. Haynes RB, McKibbon KA, Fitzgerald D, Guyatt GH, Walker CJ, Sackett DL. How to keep up with the medical literature: II. Deciding which journals to read regularly. Ann Intern Med. 1986;105:309–12.

    PubMed  CAS  Google Scholar 

  12. Haynes RB, McKibbon KA, Fitzgerald D, Guyatt GH, Walker CJ, Sackett DL. How to keep up with the medical literature: IV. Using the literature to solve clinical problems. Ann Intern Med. 1986;105:636–40.

    PubMed  CAS  Google Scholar 

  13. Huth EJ. Structured abstracts for papers reporting clinical trials. Ann Intern Med. 1987;106:626–7.

    PubMed  CAS  Google Scholar 

  14. Haynes RB, Mulrow CD, Huth EJ, Altman DG, Gardner MJ. More informative abstracts revisited. Ann Intern Med. 1990;113:69–76.

    PubMed  CAS  Google Scholar 

  15. Rennie D, Glass RM. Structured abstracts to make them more informative. JAMA. 1991;266:116–20.

    Article  PubMed  CAS  Google Scholar 

  16. Fleiss JL. Statistical methods for rates and proportions, 2nd edition. New York: Wiley, 1981.

    Google Scholar 

  17. Winslow CM, Solomon DH, Chassin MR, Kosecoff J, Merrick NJ, Brook RH. The appropriateness of carotid endarterectomy. N Engl J Med. 1988;318:721–7.

    Article  PubMed  CAS  Google Scholar 

  18. Chassin MR, Kosecoff J, Solomon DH, Brook RH. How coronary angiography is used. Clinical determinants of appropriateness. JAMA. 1987;258:2543–7.

    Article  PubMed  CAS  Google Scholar 

  19. Rubenstein LV, Kahn KL, Reinisch EJ, Sherwood M, Rogers WH, Brook RH. Structured implicit review: analysis of the method and quality of care results. Santa Monica, CA: The Rand Corporation, 1991;N-3033-HCFA.

    Google Scholar 

  20. Goldman L, Loscalzo A. Fate of cardiology research originally published in abstract form. N Engl J Med. 1980;303:255–9.

    Article  PubMed  CAS  Google Scholar 

  21. Gorman RL, Odera GM. Publication of presented abstracts at annual scientific meetings: a measure of quality? Vet Hum Toxicol. 1990;32:470–2.

    PubMed  CAS  Google Scholar 

  22. Koren G. A simple way to improve the chances for acceptance of your scientific paper. N Engl J Med. 1986;315:1298.

    PubMed  CAS  Google Scholar 

  23. Chesney R, Boxer IA. Acceptance of abstracts—a rebuttal. N Engl J Med. 1987;316:1279.

    PubMed  CAS  Google Scholar 

  24. Lock S. A difficult balance: editorial peer review in medicine. Philadelphia: ISI Press, 1986.

    Google Scholar 

  25. Garfunkel JM, Ulshen MH, Hamrick HJ, Lawson EE. Problems identified by secondary review of accepted manuscripts. JAMA. 1990;263:1369–71.

    Article  PubMed  CAS  Google Scholar 

  26. Rosenthal R. Retrieving and assessing research results. In: Meta-analytic procedures for social research. Newbury Park, CA: Sage Publications, 1991;36–58.

    Google Scholar 

  27. Cicchetti DV (and commentaries). The reliability of peer review for manuscript and grant submissions: a cross-disciplinary investigation. Behav Brain. 1991;14:119–86.

    Google Scholar 

  28. Dawes RM, Faust D, Meehl PE. Clinical versus actuarial judgment. Science. 1989;243:1668–74.

    Article  PubMed  CAS  Google Scholar 

  29. Comroe JH. Retrospectoscope: insights into medical discovery. Menlo Park, CA: Von Gehr Press, 1977.

    Google Scholar 

  30. Criteria for review and selection for abstracting. ACP J Club. 1991;Sept–Oct:A13–A14.

  31. Gardner MJ, Machin D, Campbell MJ. Use of check lists in assessing the statistical content of medical studies. BMJ. 1986;292:810–2.

    Article  PubMed  CAS  Google Scholar 

  32. Oxman AD, Guyatt GH, Singer J, et al. Agreement among reviewers of review articles. J Clin Epidemiol. 1991;44:91–8.

    Article  PubMed  CAS  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. the Division of Internal Medicine, the Program for Medical Technology and Practice Assessment, and the Department of Health Policy and Management, The Johns Hopkins University, Baltimore, Maryland

    Haya R. Rubin MD, PhD, Albert W. Wu MD, MPH & Earl P. Steinberg MD, MPP

  2. the Department of Medicine and the Division of Clinical Epidemiology, Wellesley Hospital Research Institute, University of Toronto, Toronto, Ontario, Canada

    Donald A. Redelmeier MD, MS

Authors
  1. Haya R. Rubin MD, PhD
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Donald A. Redelmeier MD, MS
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Albert W. Wu MD, MPH
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Earl P. Steinberg MD, MPP
    View author publications

    You can also search for this author in PubMed Google Scholar

Additional information

Drs. Steinberg and Rubin co-chaired the Abstract Selection Committee for the SGIM’s 14th annual meeting.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Rubin, H.R., Redelmeier, D.A., Wu, A.W. et al. How reliable is peer review of scientific abstracts?. J Gen Intern Med 8, 255–258 (1993). https://doi.org/10.1007/BF02600092

Download citation

  • Issue Date:

  • DOI: https://doi.org/10.1007/BF02600092

Key words

Search

Navigation