Editorial Spotlight: Implications of Using Artificial Intelligence as a Tool for Summarizing New Evidence

Re: Large Language Model versus Clinician-Written Summaries of Research Papers

Staying current with medical evidence is a critical skill for family physicians. As the pace of emerging medical evidence accelerates, brief summaries of impactful articles are an essential tool for busy family physicians to engage with research. A highly utilized tool is POEMs (patient-oriented evidence that matters), a product published by John Wiley & Sons through Essential Evidence Plus.^1,2 POEMs are clinically focused takeaways emphasizing outcomes that matter to patients, written by evidence-based medicine experts.

In their JABFM article, Guthmann et al reexamined research articles from previous POEMs with large language models (LLMs), comparing 24 POEMs published in the 2025 volume of American Family Physician with LLM-generated summaries of the same articles the POEMs summarized. Their findings show how generative artificial intelligence (AI) can distill research for practicing physicians. They also raise questions about the role of humans in appraising evidence for practice. In line with other perspectives on the debates around the use of LLMs in medicine, a human perspective is still critical to this process.³ From an editorial perspective, their findings also raise questions about the boundaries and tradeoffs of using LLMs for this purpose. For readers, how could clinical practice be impacted by using LLMs to summarize medical evidence for family physicians? For POEM authors, the ability to offload a task that may take hours to complete onto an LLM that can perform the task within minutes or seconds is appealing. But what do the discipline of family medicine and individual POEM authors lose by relinquishing this task to LLMs?

Family physicians must be timely consumers and translators of medical science. They make dozens of medical decisions per day, and medical science is rapidly changing. This includes a recent shift that we have observed in our own experiences: a shift away from interacting with primary research studies and a reliance on expert-written tertiary content like UpToDate, Dynamed, or Essential Evidence. A second shift has already begun: away from these sources toward AI-generated products (among them, medically-targeted products like DoxGPT and Open Evidence, in addition to broad LLMs like Gemini, chatGPT, or Claude).⁴ Little research has investigated the reasons for this shift. One reason may be that physicians prefer the writing that LLMs produce, which could potentially lead readers to conflate writing style with information quality. For example, one study found that healthcare professionals viewed AI-generated responses to patient questions as higher quality and more empathetic.⁵ Does this preference translate to the context of evidence-based summaries as well? Guthmann et al found that study evaluators who scored the POEMs and LLM summaries identified fewer errors in the LLM summaries and preferred the LLM versions overall. Yet, this finding does not necessarily preference LLMs above humans as synthesizers or authors. Rather, as the authors note, one reason underlying this preference was that the LLMs generated a structured summary that reflected the instructions provided in the prompt. If POEM authors strictly adhered to an empirically-driven information template, how would they compare?

If POEM authors had received the same instructions as the LLM regarding summary structure, it is possible that POEM authors might produce summaries with clearer clinical application—a noted low point in both LLM summaries and POEMs in Guthman et al. It is noteworthy that LLMs did a poor job of describing clinical applications of the research even though it received explicit instructions to include this information. Although LLMs may be better suited to summarizing information that is specifically reported in an article without errors, they remain limited to publicly accessible information and therefore cannot connect findings to other pertinent clinical knowledge about patients’ experiences. Physicians, on the other hand, possess privileged information about patients and patient care that will likely never be entered into an LLM. An LLM isn’t aware of patient histories, preferences, or norms that may help contextualize the clinical applicability of research findings, which makes physicians uniquely qualified for this aspect of writing research summaries.

The findings from Guthmann et al also point to areas for future research that should be studied to understand the future of LLMs. For example, study evaluators who scored the LLM summaries and POEMs only wrote comments to explain items that were either missing or insufficient, but what about items that were deemed complete? Justifications for what counts as sufficient would be equally valuable, and may be important for developing protocols around routinely implementing LLMs in research evaluation and dissemination to physicians. Additionally, the study included the most recent POEMs, which reflected the most current research well but did not include a representative sample of different study designs. Future research could explore how LLM summaries compare across study designs, beyond RCTs and meta analyses. Finally, a limitation of this study was that members of the research team were already familiar with the published POEMs and therefore were unable to conduct a blinded evaluation of the summaries. Given that previous research found that patients only preferred AI-generated responses when they didn’t know that the message was generated by AI,⁶ it is interesting that the authors preferred the LLM-generated summaries even though they were aware of the source. This highlights the need to further explore the relationship between perceptions of AI itself and views of the clinical content it produces. Future work could examine whether the preference for LLM summaries holds up when reviewed by the broader audience of clinicians that POEMs intend to serve.

POEMs may be perceived as an unbiased source of information since they are not written by the authors of the original article.⁷ LLM-generated summaries might seem even more unbiased since they are not written by humans who inherently possess their own experiences and values that shape the perceived meaning of the research they are summarizing. But the unique perspectives of real, human physicians who have cared for patients is precisely what adds value to these summaries, and this is something LLMs cannot provide. Guthmann et al report that even though the overall scores for LLMs were rated higher than POEMs, the overall preferences for one or the other weren’t as universal. This suggests that overall preference may go beyond the scoring system the authors developed, and future work should explore this further.

As the discipline of family medicine moves forward with new ways to apply LLMs to support disseminating evidence,^8–10 we must balance what we gain from these tools with what could be lost by accepting them too readily. The American Board of Family Medicine (ABFM) invests tremendous resources on expert physician-generated content for education and assessment. Efforts like the National Journal Club help family physicians become better consumers and appraisers of primary literature—although opinions on that approach are mixed.^10,11 Expedient access to accurate, clinically relevant summaries of the latest evidence is important for physicians working to provide high quality, evidence-based care. Using LLMs to summarize the evidence from research articles could help minimize errors in reporting research findings, which could then enable experts to instead focus their time on expanding the clinical application and interpretation of evidence.

The domains included in the evaluation tool that Guthmann et al developed calls attention to aspects of research summaries that physicians (and the research team) value, including clarity, brevity, presence or absence of errors, and clinical applicability. But what does the discipline of family medicine value? Learning to critically appraise an article and generate a targeted summary with a human perspective and lens is a skill in and of itself. If LLMs become the standard summary generator, who will be prepared to critically apply the evidence in the next generation?