Abstract
Introduction Clinicians require concise, accurate summaries of new research to inform practice. Patient-Oriented Evidence that Matters (POEMs), published in American Family Physician, are a benchmark for summarizing primary literature in family medicine, while large language models (LLMs) offer scalable summarization but require rigorous evaluation. The objective of this study was to evaluate the accuracy and quality of summaries generated by large language models compared with expert-authored POEMs.
Methods In this study, we compared LLM-generated summaries (Microsoft Copilot, GPT-4o class) with 24 recent matched POEMs using a standardized prompt. Two trained raters independently scored each summary with a 13-item tool (score range 0-13), cataloged errors, recorded word counts, and indicated preferences on a 5-point scale.
Results LLM summaries outperformed POEMs in total score (mean 12.1 vs 10.6; mean difference 1.5, 95% CI 1.1-2.0; P < 0.001), with similar lengths (328 vs 353 words; P = 0.23). Errors occurred in fewer LLM-DOCSs (2/24) than POEMs (9/24), with a mean error score difference of 20% (95% CI 7% -33%; P < 0.001). POEMs most often missed in the categories Contextual Background and Limitations; both approaches frequently missed in Clinical Applicability. Reviewer preference favored LLM-DOCS (mean 2.44 on a 1-5 scale; 95% CI 2.1-2.8).
Conclusions An enterprise LLM, prompted in POEM style, produced accurate, low-error clinical summaries that matched or exceeded expert-edited POEMs and were generally preferred by reviewers, though further research is needed to assess broader applicability and impact. Findings support pragmatic LLM-assisted summarization and highlight the need for standardized evaluation tools and explicit prompts for clinical applicability.
- Clinical Decision-Making
- Clinical Decision Support
- Evidence-Based Medicine
- Family Medicine
- Large Language Models
- Natural Language Processing
Introduction
Critically appraising research literature, a core competency for physicians, is essential for staying current with medical literature. The accelerating pace of new clinical evidence can overwhelm effective translation of research into practice, particularly for primary care given its broad scope.1,2 Effective translation of research depends on systems that continuously identify, appraise, and distill key findings.
POEMs (Patient-Oriented Evidence that Matters) are published online in Essential Evidence Plus with a selection featured each month in American Family Physician.3–29 POEMs are concise summaries (approximately 400 words) with an informative title, clinical question, and bottom-line conclusion. They also assign level-of-evidence, summarize the study design, and include relevant statistics with sufficient detail for readers to judge how findings can be applied in practice. These summaries serve as a recognized model for summarizing clinical medical journal articles.
Large Language Models (LLMs) offer a rapid new approach to article summarization, though significant concerns exist about their abilities and reliability. It remains unclear whether LLMs produce significant errors (hallucinations) when summarizing individual studies, or whether they can accurately identify key elements and generate critical appraisals of medical literature to support clinical decision-making.30–37
Recent studies have examined the capabilities and limitations of large language models in clinical research. ChatGPT-3.5 outperformed experienced critical care specialists in writing background sections, with 80 evaluators often unable to distinguish its output from human-written content.30 GPT-4 demonstrated strong article screening performance, matching human reviewers with 91% agreement for excluding irrelevant papers and 76% for including relevant ones, while four other LLMs performed well but imperfectly.31,32 By contrast, earlier models, including ChatGPT-3.5, produced error-prone results with hallucinations and difficulty applying search filters.33 In extracting evidence from full-text documents, LLMs generally avoided fabrication but often missed relevant details.34
Benchmarking studies have evaluated LLMs across various domains of evidence-based medicine. One study compared seven LLMs on comprehensiveness, correctness, and readability, relying on expert opinion as the gold standard and highlighting the need for more objective evaluation metrics.35 A scoping review further categorized methods for assessing LLMs in medicine, identified common error types, and emphasized the need for standardized evaluation frameworks.36 Subsequently, a joint editorial by family medicine journal editors discussed the ethical and practical implications of LLMs in scientific publishing, advocating for transparency and recognizing their potential to enhance research productivity.37
While criteria exist to evaluate evidence-based medicine studies including randomized controlled trials and meta-analyses,38–41 no standardized tool exists for assessing medical journal article summaries like POEMs. Similarly, criteria for evaluating LLM performance is largely derived from computer science literature.30–36 To address these gaps, we developed, validated, and tested a new evaluation tool. The objective of this study was to use this tool to evaluate and compare the accuracy and quality of LLM-generated summaries (LLM-DOCs) with physician-authored POEMs. We hypothesized that LLMs could accurately summarize medical journal articles when assessed using this tool.
Methods
We conducted a head-to-head comparative analysis of LLM-DOCS and POEMs.
Large Language Model–Doctor-Oriented Clinical Summaries (LLM-DOCS)
The LLM-DOCS were generated using the Advocate Health Care enterprise version of Copilot in July 2025. Microsoft Copilot (enterprise, GPT-4o-class; “ChatGPT-turbo” build), according to Microsoft Corporation, is most like ChatGPT-4o. Once generated, unaltered LLM-DOCS were provided to the evaluators electronically as Word documents.
Data Safeguards for Content Use and AI Processing
A limited number of full-text articles, accessed through institutional licenses, were processed in a closed large language model environment (Microsoft Copilot, Advocate Health Care/Wake Forest University account) solely for academic research. The principal investigator was the only team member who generated summaries. The LLM did not store or learn from the content, and outputs were restricted to summary comparisons for evaluation purposes; no duplication, redistribution, or commercial use occurred. This approach was equivalent to a researcher reading articles to write summaries and was consistent with fair-use provisions or text- and data-mining exceptions for noncommercial academic research. The study was reviewed by the Advocate Health Care/Wake Forest University and Medical College of Wisconsin Institutional Review Boards and was determined to be nonhuman subject research.
POEMs and Original Articles
The 24 POEMs published in American Family Physician6–29 from February 2025 through July 2025 were used as the comparator for the LLM-DOCS. The original articles, used to create the POEMs served as the source articles.42–65 PDF versions of both the POEMs and their corresponding original journal articles were provided electronically to the reviewers. No supplemental medical articles were provided to the reviewers nor the LLM, but neither one was restricted from seeking them out.
LLM Prompt
LLM-DOCs were generated in Microsoft Copilot using a fixed prompt designed to produce POEM-style, doctor-oriented clinical summaries. Evidence-based medicine principles, critical appraisal tools, as well as elements of POEMs informed the LLM-DOCS’ prompt. The LLM was instructed to ignore prior chat history. The verbatim prompt, as entered into the LLM, is reproduced in Table 1.
Evaluation Tool
Because no validated tool exists to evaluate doctor-oriented clinical summaries of medical journal articles, we identified key elements for their critical appraisal, drawing on concepts from POEMs and Good Evidence Matters (or GEMS, a series of family medicine single-article summaries), as well as published literature on critical appraisal of articles.3,4,38–41,66,67
These evidence-based medicine and critical appraisal principles informed the development of the evaluation tool, and the resulting categories, listed in Table 2, reflect components needed to evaluate a medical article.38–41,67 Supplementing standard appraisal categories, the tool incorporated two extra domains. Errors in the LLM summaries were considered critical to this investigation, so a dedicated category was included. Clinical applicability was also included to assess humans’ ability to identify the practical relevance of the original article.
POEM and LLM-DOCS Scoring
Two trained evaluators independently scored each POEM and LLM-DOCS paired summaries. The four study authors functioned as the evaluators using a balanced incomplete block design in which each person evaluates 12 of the 24 articles, with every pair of reviewers sharing exactly four articles in common. Blinding was not possible because the evaluators were already familiar with the published POEMs. Each summary was assessed across 13 categories; a score of 1 indicated the category was present, 0.5 indicated it was insufficiently addressed, and 0 indicated it was absent. The errors category, defined as overt mistakes or misstatements, could receive full credit even if other categories scored zero. A perfect score of 13 reflected fulfillment of all categories with no errors. Evaluators were also asked to provide comments for all scores of 0 or 0.5, as well as additional feedback wherever relevant.
Evaluation Tool Validation
The evaluation tool was beta tested over five weeks using five sets of original source medical journal articles, POEMs, and LLM-DOCS. During this period, the four reviewers independently completed their assessments, and scores as well as comments were compiled. The group then met weekly to review mismatched answers and refine the tool. An iterative process examined validity and reliability by assessing whether the tool measured its intended constructs, provided clear category definitions, produced consistent responses across reviewers and articles, captured all relevant content, and was practical to use and interpret. The evaluation tool was adjusted as needed during this process and was finalized after the five-week pilot.
Interrater Reliability
After the 24 article sets were scored, Cohen’s kappa (κ) interrater reliability testing was calculated for frequency of the evaluators’ agreement in each category. Cohen’s kappa (κ) of 0.41 with a 95% CI of 0.27-0.55 achieved the lower end of moderate agreement and indicated meaningful reliability.
Post Scoring Adjudication
To improve scoring accuracy, the two primary reviewers for each set of articles met to adjudicate discrepancies between the two reviewers, reviewing and discussing any categories where their independent scores did not align. In this second step of the evaluation, reviewers referred to and checked original articles, LLM-DOCS, and POEMs to ensure accuracy. Each reviewer retained the option to revise their own scores, and any changes were recorded. Following each review session, the full group of four raters met to refine shared principles for scoring and clarifying category definitions. Across six one-hour adjudication meetings, we documented and recorded all changes, including their rationale. Interrater reliability was subsequently calculated as Cohen’s κ = 0.93, 95% CI 0.89-0.97.
Final Scores
If consensus was not reached in the adjudication meeting, the reviewers’ final scores retained discrepancies between the two evaluators. Combining the reviewers’ final category scores by averaging the two evaluators’ scores yielded possible category values of 0, 0.25, 0.5, 0.75, 1. The category averages were then totaled to produce a final score ranging from 0 to 13 per summary. Categories not achieving a “1” were tabulated, allowing analysis of which items and with what frequency each category were not meeting full expectations.
Reviewer Preferences
Reviewers recorded their preference between the two summaries for each individual category as well as an overall preference, using a 5-point scale, including:
Strongly Favor LLM
Favor LLM
Indistinguishable or no preference
Favor POEM
Strongly Favor POEM
Word Counts
After the completion of the reviewers’ evaluations, the word counts of both summaries were tabulated.
Results
The LLMs and the POEMs summarized the same 24 medical journal articles: 11 randomized controlled trials, 1 cohort study, 8 meta-analyses, and 4 systematic reviews (2 of which were guidelines).
Average summary length did not differ significantly between the LLM-DOCS and POEMs (328 vs 353 words, respectively; P = 0.23). The LLM-DOCS ranged from 280 to 400 words (range 120), whereas POEMs varied more widely from 198 to 551 words (range 353).
Both POEMs and LLM summaries used sub-sections; however, the LLM outputs tended to adopt a structured, list-like format, whereas the POEMs were written with more prose.
Primary Outcome
The LLM-DOCS mean score was higher than the POEMs on the 0 to13–point evaluation tool (mean points 12.1 versus 10.6, a 1.5-point advantage with a 95% CI 1.1-2.0; P < 0.001) (Figure 1) The LLM-DOCS scored higher on 20 of the 24 summaries (Figure 2).
* Range 0 to 13
† Mean difference: 1.5-points, 95% CI 1.1-2.0; P < 0.001
Abbreviations: LLM-DOCS = Large Language Model–Doctor-Oriented Clinical Summaries, POEMs = Patient-Oriented Evidence that Matters
Abbreviations: LLM-DOCS = Large Language Model–Doctor-Oriented Clinical Summaries, POEMs = Patient-Oriented Evidence that Matters
Secondary Outcomes
Table 3 shows scores by category. The POEMs’ lower scores clustered in Contextual Background (18 POEMs), Limitations and Biases (13 POEMs), and Clinical Applicability (12 POEMs), with additional deductions in Errors (9 POEMs) and Results (7 POEMs). In contrast, LLM-DOCS showed the most frequent deductions in Clinical Applicability (12 LLM-DOCS) and Results (6 LLM-DOCS), with moderate losses in Level of Evidence (3 LLM-DOCS), Description of Population (4 LLM-DOCS), and Errors (2 LLM-DOCS). The LLM-DOCS demonstrated no deductions in Contextual Background, Clarity/Brevity, or Methodology/Design, and only minor ones in Title, Intervention/Comparison, and Outcomes (each affecting one LLM-DOCS).
Errors (or Hallucinations)
Across the 24 articles, the LLM-DOCS produced fewer errors than the POEMs. Errors appeared in 9 of 24 POEMs (37.5%), including five partial point– and four full point–deductions. In contrast, LLM-DOCS contained errors in only two of 24 summaries (8.3%), one partial point– and one full point–deduction. A full point–deduction means that both evaluators scored the category as a zero, but does not indicate the severity or significance of the error. Error rates did not vary by study type. The mean “no-error” scores, which favor LLM-DOCS (mean difference 0.20, 95% CI 0.07 to 0.33, P = 0.01), are shown in Figure 3. Table 4 describes the errors by LLM-DOCS and POEMs.
*Mean error score where 1 equals no errors
† Mean difference 0.20, 95% CI 0.07 to 0.33, P = 0.01
Abbreviations: LLM-DOCS = Large Language Model–Doctor-Oriented Clinical Summaries, POEMs = Patient-Oriented Evidence that Matters
Contextual Background
This category showed the largest difference in scores between LLM-DOCS and POEMs. The POEMs missed points in 18 of 24 summaries, while the LLM-DOCS missed none.
Limitations and Biases
This category accounted for the second biggest point discrepancy with mean scores of 0.96 and 0.55 (mean difference 0.41, P-value 0.0002). The LLM-DOCS consistently itemized important evidence-based critiques.
Clinical Applicability
Both LLM and POEM summaries scored poorly in this category, each missing points in 12 of 24 summaries. It was the lowest performing category for the LLM-DOCS.
Reviewer Preferences
Overall and category-composite preference scores (Figure 4) were strongly correlated (Pearson’s r = 0.86, 95% CI 0.70 to 0.94), indicating close agreement between the two measures. Across 24 article summaries, overall preference scores leaned toward LLM-DOCS (mean = 2.44 on a 1-5 scale where 1 = LLM and 5 = POEM; 95% CI 2.1-2.8). Reviewers favored LLMs on 14 of 24 (58%) summaries, no preference on 5 of 24 (21%) summaries, and favored POEMs on 5 of 24 (21%) summaries. The averaged category-composite preference scores averaged 2.73, also favoring the LLM-DOCS.
Abbreviations: LLM-DOCS = Large Language Model–Doctor-Oriented Clinical Summaries, POEMs = Patient-Oriented Evidence that Matters
Discussion
The LLM produced accurate, low-error summaries that equaled or exceeded expert-edited POEMs and were generally preferred by reviewers. Given the recent advent of LLMs, this early research into LLMs’ ability to accurately summarize the medical literature for clinicians raises additional questions and possibilities. The LLMs could potentially improve translation of medical research into clinical practice. The speed of generating the LLM summaries, while not explicitly measured here, would likely be measured in seconds to minutes, suggesting the potential to summarize, sort, and rank large volumes of research for clinical use.
To better understand LLM errors, study authors created a dedicated category to identify and itemize mistakes. We found that errors were not a shortcoming of the LLMs; in fact, the LLM generated significantly fewer errors than the human-written POEMs. The mean difference in error scores (0.20, 95% CI 0.07-0.33, P = 0.01) corresponds to 20% fewer errors. This lower error rate suggests that LLM-generated summaries could offer a reliable alternative to human-generated summaries, potentially reducing the risk of misinformation in clinical decision-making and supporting more consistent, evidence-based practice.
The LLM prompt strongly influences the LLM-DOCS and altering the prompt warrants further investigation. Our prompt was created to simulate the POEMs’ format and to provide information that aids clinician scientists in interpreting original research. We explicitly included a “Contextual Background” evaluation prompt to ensure each summary addressed why the study was conducted and how it contributed to existing knowledge. Consequently, all 24 LLM-DOCS included this information, whereas the POEMs did not consistently do so. Study authors agreed that this category is critical both to prevent misapplication of evidence and to inform readers who may lack baseline knowledge on the subject. Notably, this category accounted for the greatest scoring discrepancy in our evaluation. While POEMs are intended to provide clinically relevant context,4 the extent to which background information is explicitly required versus assumed is not clearly specified in publicly available guidance. These design choices likely contributed to higher scores for contextual background among LLM-DOCS and should be considered when interpreting comparative performance across domains. These observations are presented solely to contextualize the comparative findings and are not intended as recommendations or critiques of POEM editorial practices.
These summaries, LLM-DOCS and POEMs, should bridge the gap between evidence and action, enabling clinicians to make informed, patient-centered decisions. In “Introducing POEMS,”4 Shaughnessy and Siwek state that POEMs “provide practical pointers for day-to-day practice” and are “always accompanied by an editorial commentary,” which informed the inclusion of “Clinical Applicability” as a category in our evaluation tool. However, these pointers or commentaries were often absent in prior published POEMs, and therefore “clinical applicability” was not used as an explicit LLM prompt. Our results show that neither summary type consistently fulfilled this criterion. As a result, it did not significantly contribute to the score differential between LLM-DOCs and POEMs (0.68 vs 0.64, P 0.21). Some original source articles did not meet the “POEM” criteria of affecting or changing practice; therefore, no clinical application or recommendation might be expected.
The limitations and biases category accounted for the second-largest discrepancy between the LLM-DOCS and POEM scores. The LLM summary consistently reported study limitations, whereas the POEMs often addressed them inadequately. Several other categories contributed minor differences to the scoring, including “Accuracy of key takeaway,” “description of population,” and “intervention and comparison,” which accounted for 6% to 9% of the total score differential. While these scoring differentials were small, these categories are fundamental to evidence-based medicine reviews and further highlight the effectiveness of LLM-DOCS.
The “Level of Evidence” category showed shortcomings for both summary types. The POEMs occasionally misapplied an outdated Oxford Centre for Evidence-Based Medicine nomenclature.68 The LLM-DOCS did not explicitly reference any formal nomenclature, but it frequently described the level of evidence in plain language (eg, “Level 1, RCT”) that did not correspond to any widely recognized system.
The POEM titles often revealed the key takeaway and occasionally provided clinical applicability information. In contrast, the LLM prompt of a “short catchy title” generated clever and playful titles that were rarely clinically useful.
This study has several limitations. The number and types of articles reviewed were limited, and all articles were selected by POEM authors, which may introduce selection bias and limit generalizability. Evaluators were not blinded to the summary type, and the small, relatively homogenous group of four reviewers may not reflect broader clinician perspectives. The LLM-DOCS outputs were generated using a single prompt and one model (Microsoft Copilot, GPT-4 variant), so results may differ with other prompts or models. While the evaluation tool was carefully developed and validated internally, it has not yet been tested externally. Additionally, the study focused on summary quality and did not assess real-world clinical impact or patient outcomes. Finally, differences in presentation style, such as bullet points in LLM-DOCS versus prose in POEMs, may have influenced reviewer preferences and scoring. Despite these limitations, the findings provide insight into the comparative performance of LLM-generated and human-curated clinical summaries.
Future research could explore several areas to build on these findings. Studies might examine the reproducibility of LLM-DOCS outputs, test newer LLM versions beyond ChatGPT-4, and evaluate a broader range of prompts. Further refinement and validation of the evaluation tool could enhance its reliability, and comparisons with traditional article abstracts may provide additional context for clinical use. A particularly important area for investigation is modifying prompts to explicitly elicit clinical applicability. While this could improve the relevance of summaries for patient care, it may also increase the risk of errors, as the LLM would need to interpret and apply research findings in the context of current standards of care rather than simply summarizing the study. Deeper investigation into this area, along with alignment to evidence-based principles, may help optimize LLM-generated summaries for safe and effective clinical decision-making.
Conclusion
LLM-DOCS provide a useful method to identify, appraise, distill, and translate key medical journal article findings into clinical practice.
Conflicts of Interest
None.
Corresponding Author
Richard Guthmann, MD, MPH, Advocate Illinois Masonic Family Medicine Residency, University of Illinois at Chicago, RickGuthmann{at}gmail.com;Rick.Guthmann{at}aah.org
This article was externally peer reviewed.
Acknowledgment
The authors thank the Family Medicine Midwest Scholarly Activity Collaborative for their support, and the American Board of Family Medicine Foundation Family Medicine Residency Learning Community Planning Grant for the funding.
- Received for publication October 14, 2025.
- Accepted for publication February 2, 2026.
References
Jump to section
Related Articles
Cited By...
- No citing articles found.