United States

&lt;p&gt;M. Janina Sarol,&lt;sup&gt;1&lt;/sup&gt; Jodi Schneider,&lt;sup&gt;1,2&lt;/sup&gt; Halil Kilicoglu&lt;sup&gt;1&lt;/sup&gt;&lt;/p&gt;
&lt;h4&gt;Objective &lt;/h4&gt;
&lt;p&gt;An estimated 25% of citations in medical journals are inaccurate.&lt;sup&gt;1&lt;/sup&gt; If not corrected, these citation quotation errors can propagate misinformation and establish unfounded claims.&lt;sup&gt;2&lt;/sup&gt; Correcting them before publication is vital, yet citation accuracy is overlooked in the peer review process. The prevalence of citation quotation errors in the biomedical literature also remains unclear, because existing studies rely on manual evaluation on a limited set of articles.&lt;/p&gt;
&lt;h4&gt;Design &lt;/h4&gt;
&lt;p&gt;We explored the use of large language models, specifically Gemini 1.5 Pro, GPT-4o (gpt-4o-2024-08-06), and LLaMA-3.1 (8B and 70B), to automatically identify citation quotation errors. We compared 2 approaches. In the first, we supplied the prompt with the citation in question, the paragraph containing the citation, and the full reference article. The second approach consists of a 3-step procedure and mimicked how a human reviewer might perform the task manually by first identifying the citation context, then locating relevant sentences in the reference article, and finally assessing citation accuracy. Specifically, the procedure consists of (1) extracting the citation sentence, (2) retrieving the 50 most relevant sentences using the MedCPT retriever model and reranking them with a fine-tuned MonoT5 model to select the top 5 sentences, and (3) verifying the citation accuracy with the large language model (&lt;b&gt;Box 25-1015&lt;/b&gt;). Both approaches were evaluated from November 2024 to January 2025 using a citation quotation error dataset,&lt;sup&gt;3&lt;/sup&gt; the first publicly available corpus of its kind (Cohen κ for interrater agreement: 0.96 for citation context and 0.31 for accuracy detection). The better-performing approach was applied to assess 2898 citations from 2000 articles to the 100 most cited PubMed Central Open Access (PMC-OA) articles (24 different journals).&lt;/p&gt;

![](https://assets.underline.io/markdown_image/1/image/d650d5b842c02ea3ae5d4d9c28d01a15.png)

![](https://assets.underline.io/markdown_image/1/image/e80718c94a9aa238f09a1d74fc02135d.png)

&lt;h4&gt;Results &lt;/h4&gt;
&lt;p&gt;Gemini 1.5 Pro yielded best performance, obtaining an accuracy of 69% with both approaches. It outperformed GPT-4o slightly, while LLaMA-3 failed to generate meaningful results with the first approach and performed poorly with the second. Because the first approach only identified 25% (55 of 220) of the erroneous citations and the 3-step procedure identified 51% (113 of 220), the second approach was applied to the subset of PMC-OA articles. In the PMC-OA subset, 34% (981 of 2898) of the citation instances were assessed as erroneous. A total of 37% (742 of 2000) of citing articles were found to contain citation quotation errors; 98% (98 of 100) of reference articles were cited incorrectly at least once.&lt;/p&gt;
&lt;h4&gt;Conclusions &lt;/h4&gt;
&lt;p&gt;Our preliminary results confirm the high prevalence of citation quotation errors in the medical literature. We will present a more comprehensive analysis of citation quotation errors in the PMC-OA subset at the conference. Automated verification of citations could help journals and peer reviewers identify questionable citation practices and reduce propagation of misinformation, improving the trustworthiness of scientific evidence.&lt;/p&gt;
&lt;h4&gt;References&lt;/h4&gt;
&lt;p&gt;1. Jergas H, Baethge C. Quotation accuracy in medical journal articles: a systematic review and meta-analysis. &lt;i&gt;PeerJ&lt;/i&gt;. 2015;3:e1364. doi:10.7717/peerj.1364&lt;/p&gt;
&lt;p&gt;2. Greenberg SA. How citation distortions create unfounded authority: analysis of a citation network. &lt;i&gt;BMJ&lt;/i&gt;. 2009;339:b2680. doi:10.1136/bmj.b2680&lt;/p&gt;
&lt;p&gt;3. Sarol MJ, Ming S, Radhakrishna S, Schneider J, Kilicoglu H. Assessing citation integrity in biomedical publications: corpus annotation and NLP models. &lt;i&gt;Bioinformatics&lt;/i&gt;. 2024;40(7):btae420. doi:10.1093/bioinformatics/btae420&lt;/p&gt;

&lt;sup&gt;1&lt;/sup&gt;University of Illinois at Urbana-Champaign, IL, US, janinasarol@gmail.com; &lt;sup&gt;2&lt;/sup&gt;Harvard Radcliffe Institute for Advanced Study, Cambridge, MA, US.
&lt;h4&gt;Conflict of Interest&lt;/h4&gt; Disclosures Jodi Schneider declares nonfinancial associations with Crossref; Committee on Publication Ethics; International Association of Scientific, Technical and Medical Publishers; the European Association of Science Editors; the International Society of Managing and Technical Editors; the Institute of Electrical and Electronics Engineers; the National Information Standards Organization; and the Center for Scientific Integrity (parent organization of Retraction Watch) and has received data-in-kind from Retraction Watch and Scite and usability testing compensation from the Institute of Electrical and Electronics Engineers. The National Information Standards Organization is a subawardee on her Alfred P. Sloan Foundation grant (G-2022-19409). No other disclosures were reported.
&lt;h4&gt;Funding/Support&lt;/h4&gt; This study was supported by the Office of Research Integrity (ORI) of the US Department of Health and Human Services (HHS) (ORIIR220073). The contents are those of the authors and do not necessarily represent the official views of, nor an endorsement by, the ORI, Office of the Assistant Secretary of Health, HHS, or the US government. Jodi Schneider was supported in part as the 2024-2025 Perrin Moorhead Grayson and Bruns Grayson Fellow, Harvard Radcliffe Institute for Advanced Study and by US National Science Foundation’s CAREER grant (2046454).
&lt;h4&gt;Role of the Funder/Sponsor&lt;/h4&gt; The funders had no role in considering the study design or in the collection, analysis, interpretation of data, writing of the report, or decision to submit the article for publication.
&lt;h4&gt;Acknowledgment&lt;/h4&gt; We thank Shruthan Radhakrishna for his contributions to fine-tuning the MonoT5 model.
&lt;h4&gt;Additional Information&lt;/h4&gt; Halil Kilicoglu is a co–corresponding author (halil@illinois.edu).




The 10th Peer Review Congress

Leveraging Large Language Models for Detecting Citation Quotation Errors in Biomedical Literature

artificial intelligence and large language models

quality of the literature

research integrity

The Tenth Peer Review Congress featured 3 days for presentations of new research into peer review and all aspects of scientific publication, bias, quality of reporting, and information access and dissemination.

This year’s meeting was hybrid, and all plenary sessions were livestreamed with opportunities to view presentations and engage with participants in person and virtually during and after the meeting. All recordings are now available on Underline for viewing.

<div style="position:relative;padding-top:max(60%,324px);width:100%;height:0;"><iframe style="position:absolute;border:none;width:100%;height:100%;left:0;top:0;" src="https://online.fliphtml5.com/ebtyf/mget/"  seamless="seamless" scrolling="no" frameborder="0" allowtransparency="true" allowfullscreen="true" ></iframe></div>


#### Why? 
Our aim is to encourage research into the quality and credibility of peer review and scientific publication, to establish the evidence base on which scientists can improve the conduct, reporting, and dissemination of scientific research.

#### Who? 
Participants include editors and publishers of scientific peer-reviewed journals, researchers, funders, bibliometric and informatics experts, information innovators, librarians, journalists, policymakers, ethicists, scientific information disseminators, and anyone interested in the progress of the scientific information enterprise and the quality of scientific evidence.

The Congress embraces a wide range of disciplines, including biomedicine, health and life sciences, applied sciences, basic sciences, physical and chemical sciences, mathematics, computer sciences, engineering, economics, and social sciences as well as new and emerging disciplines.

#### What? 
The Congress provides a forum for the presentation and discussion of new research into the quality and credibility of editorial and funding peer review, scientific and scholarly publication, research methods and reporting practices, identification and management of biases, transparency, reproducibility, access, publication and dissemination metrics, identification and management of threats to the integrity of science, and advances in information exchange.

**Follow us**

[![](https://assets.underline.io/markdown_image/1/image/662a22766b9788ed3766cb23cd156e32.png)](https://x.com/peerrevcongress)  [![](https://assets.underline.io/markdown_image/1/image/bb105afb1773a01fa169aa97ca1af49a.png)](https://www.youtube.com/channel/UCsG_0nKi3_Zfyv0yRPdxjWQ)  [![](https://assets.underline.io/markdown_image/1/image/034e1e70a194a9a4178fdf470ef796db.png)](https://bsky.app/profile/peerreviewcongress.bsky.social)



You need to log in with the email address you registered with. 



Please log in to explore this event.

Registration is required to attend this event, please [here](https://peerreviewcongress.org/registration-information/) to secure your ticket. Please check your spam/promotional inbox too.



Please register!

The Tenth Peer Review Congress features 3 days for presentations of new research into peer review and all aspects of scientific publication, bias, quality of reporting, and information access and dissemination.

<p>M. Janina Sarol,<sup>1</sup> Jodi Schneider,<sup>1,2</sup> Halil Kilicoglu<sup>1</sup></p>
<h4>Objective </h4>
<p>An estimated 25% of citations in medical journals are inaccurate.<sup>1</sup> If not corrected, these citation quotation errors can propagate misinformation and establish unfounded claims.<sup>2</sup> Correcting them before publication is vital, yet citation accuracy is overlooked in the peer review process. The prevalence of citation quotation errors in the biomedical literature also remains unclear, because existing studies rely on manual evaluation on a limited set of articles.</p>
<h4>Design </h4>
<p>We explored the use of large language models, specifically Gemini 1.5 Pro, GPT-4o (gpt-4o-2024-08-06), and LLaMA-3.1 (8B and 70B), to automatically identify citation quotation errors. We compared 2 approaches. In the first, we supplied the prompt with the citation in question, the paragraph containing the citation, and the full reference article. The second approach consists of a 3-step procedure and mimicked how a human reviewer might perform the task manually by first identifying the citation context, then locating relevant sentences in the reference article, and finally assessing citation accuracy. Specifically, the procedure consists of (1) extracting the citation sentence, (2) retrieving the 50 most relevant sentences using the MedCPT retriever model and reranking them with a fine-tuned MonoT5 model to select the top 5 sentences, and (3) verifying the citation accuracy with the large language model (<b>Box 25-1015</b>). Both approaches were evaluated from November 2024 to January 2025 using a citation quotation error dataset,<sup>3</sup> the first publicly available corpus of its kind (Cohen κ for interrater agreement: 0.96 for citation context and 0.31 for accuracy detection). The better-performing approach was applied to assess 2898 citations from 2000 articles to the 100 most cited PubMed Central Open Access (PMC-OA) articles (24 different journals).</p>

![](https://assets.underline.io/markdown_image/1/image/d650d5b842c02ea3ae5d4d9c28d01a15.png)

![](https://assets.underline.io/markdown_image/1/image/e80718c94a9aa238f09a1d74fc02135d.png)

<h4>Results </h4>
<p>Gemini 1.5 Pro yielded best performance, obtaining an accuracy of 69% with both approaches. It outperformed GPT-4o slightly, while LLaMA-3 failed to generate meaningful results with the first approach and performed poorly with the second. Because the first approach only identified 25% (55 of 220) of the erroneous citations and the 3-step procedure identified 51% (113 of 220), the second approach was applied to the subset of PMC-OA articles. In the PMC-OA subset, 34% (981 of 2898) of the citation instances were assessed as erroneous. A total of 37% (742 of 2000) of citing articles were found to contain citation quotation errors; 98% (98 of 100) of reference articles were cited incorrectly at least once.</p>
<h4>Conclusions </h4>
<p>Our preliminary results confirm the high prevalence of citation quotation errors in the medical literature. We will present a more comprehensive analysis of citation quotation errors in the PMC-OA subset at the conference. Automated verification of citations could help journals and peer reviewers identify questionable citation practices and reduce propagation of misinformation, improving the trustworthiness of scientific evidence.</p>
<h4>References</h4>
<p>1. Jergas H, Baethge C. Quotation accuracy in medical journal articles: a systematic review and meta-analysis. <i>PeerJ</i>. 2015;3:e1364. doi:10.7717/peerj.1364</p>
<p>2. Greenberg SA. How citation distortions create unfounded authority: analysis of a citation network. <i>BMJ</i>. 2009;339:b2680. doi:10.1136/bmj.b2680</p>
<p>3. Sarol MJ, Ming S, Radhakrishna S, Schneider J, Kilicoglu H. Assessing citation integrity in biomedical publications: corpus annotation and NLP models. <i>Bioinformatics</i>. 2024;40(7):btae420. doi:10.1093/bioinformatics/btae420</p>

<sup>1</sup>University of Illinois at Urbana-Champaign, IL, US, janinasarol@gmail.com; <sup>2</sup>Harvard Radcliffe Institute for Advanced Study, Cambridge, MA, US.
<h4>Conflict of Interest</h4> Disclosures Jodi Schneider declares nonfinancial associations with Crossref; Committee on Publication Ethics; International Association of Scientific, Technical and Medical Publishers; the European Association of Science Editors; the International Society of Managing and Technical Editors; the Institute of Electrical and Electronics Engineers; the National Information Standards Organization; and the Center for Scientific Integrity (parent organization of Retraction Watch) and has received data-in-kind from Retraction Watch and Scite and usability testing compensation from the Institute of Electrical and Electronics Engineers. The National Information Standards Organization is a subawardee on her Alfred P. Sloan Foundation grant (G-2022-19409). No other disclosures were reported.
<h4>Funding/Support</h4> This study was supported by the Office of Research Integrity (ORI) of the US Department of Health and Human Services (HHS) (ORIIR220073). The contents are those of the authors and do not necessarily represent the official views of, nor an endorsement by, the ORI, Office of the Assistant Secretary of Health, HHS, or the US government. Jodi Schneider was supported in part as the 2024-2025 Perrin Moorhead Grayson and Bruns Grayson Fellow, Harvard Radcliffe Institute for Advanced Study and by US National Science Foundation’s CAREER grant (2046454).
<h4>Role of the Funder/Sponsor</h4> The funders had no role in considering the study design or in the collection, analysis, interpretation of data, writing of the report, or decision to submit the article for publication.
<h4>Acknowledgment</h4> We thank Shruthan Radhakrishna for his contributions to fine-tuning the MonoT5 model.
<h4>Additional Information</h4> Halil Kilicoglu is a co–corresponding author (halil@illinois.edu).




technical paper

<p>Ana Marušić<sup>1</sup></p>
<h4>Importance </h4>
<p>Authorship is the currency of academia and research, but it is not only about recognition. It brings accountability and transparency and can have ethical and legal implications.</p>
<h4>Observations </h4>
<p>Starting from the call for contributors to replace authors by Drummond Rennie and his colleagues almost 30 years ago,<sup>1</sup> the evolution of scientific authorship and contributorship models will be presented. Current challenges to authorship will also be explored, from paper mills to artificial intelligence.</p>
<h4>Conclusions </h4>
<p>Authorship has not been replaced by contributorship and will need a systemic approach to balance scientific credit, career advancement, and ethical responsibility.</p>
<h4>Reference</h4>
<p>1. Rennie D, Yank V, Emanuel L. When authorship fails: a proposal to make contributors accountable. <i>JAMA</i>. 1997;278(7):579-585. doi:10.1001/jama.278.7.579</p>

<p><sup>1</sup>University of Split School of Medicine, Split, Croatia; ana.marusic@mefst.hr.</p>
<h4>Conflict of Interest Disclosures</h4> Ana Marušić is a member of the Peer Review Congress Advisory Board but was not involved in the review or decision for this abstract.


Forward to the Past - Making Contributors Accountable

<p>Isamme AlFayyad,<sup>1</sup> Maurice Zeegers,<sup>2</sup> Lex Bouter,<sup>3</sup> Helen Macdonald,<sup>4</sup> Sara Schroter<sup>4</sup></p>
<h4>Objective </h4>
<p>Recent surveys report that a high percentage (50%-76%) of researchers use artificial intelligence (AI) in their research.<sup>1,2</sup> In April 2024, BMJ Group mandated submitting authors to disclose and describe their use of AI.<sup>3</sup> We analyzed the frequency of self-disclosed use of AI in research manuscripts submitted to 49 BMJ Group biomedical journals and identified types of AI tools used and the tasks they assisted with. We also compared the characteristics of manuscripts disclosing AI use compared with those not disclosing it.</p>
<h4>Design </h4>
<p>A cross-sectional analysis of original research submitted between April and November 2024 was conducted and reported using the STROBE reporting guideline. Main study outcomes were frequency of self-disclosed use of AI, types of AI tools used, and tasks they assisted with. Characteristics of submissions (region of submitting author, number of authors, acceptance rate, impact factor, general medical/specialty journal, peer review model) were extracted from journals’ manuscript tracking systems and websites. χ² test was used to compare proportions of these characteristics between submissions disclosing and not disclosing AI use. Factors associated with disclosing AI use were assessed using binomial logistic regression. A <i>P</i> value threshold for significance was set at .05.</p>
<h4>Results </h4>
<p>There were 25,114 eligible submissions from Asia (13,505 [53.8%]), Europe (6523 [26.0%]), North America (2795 [11.1%]), Africa (1196 [4.8%]), Oceania (708 [2.8%]), and South America (387 [1.5%]). A total of 1431 of 25,114 submissions (5.7%) disclosed AI use. The most common types of AI tools used were generative AI chatbots (812/1431 [56.7%]), writing assistant tools (182 [12.7%]), and visual/image processing tools (38 [2.7%]) (<b>Table 25-1013</b>). The majority of submitting authors who disclosed AI use reported using it to improve the quality of their writing (1248/1431 [87%]), and 28% (n = 399) reported using it for other purposes, including translation (107 [7.5%]), analyzing or collecting data (44 [3.1%]), generating data and output (87 [6.1%]), code writing (15 [1.0%]), image processing (36 [2.5%]), literature searches (49 [3.4%]), and managing references (8 [0.6%]). The percentage of submissions with disclosed AI use varied significantly by region (highest in Europe: 31.0%; lowest in Oceania: 1.0%; <i>P</i> &lt; .001). Submissions disclosing AI use had a lower mean number of authors (8.39 vs 8.90; <i>P</i> = .005). Authors from Europe (odds ratio [OR], 3.61 [95% CI, 2.11-6.18]), North America (OR, 2.30 [95% CI, 1.32-4.02]), South America (OR, 4.93 [95% CI, 2.62-9.28]), Asia (OR, 2.87 [95% CI, 1.68-4.89]), and Africa (OR, 3.34 [95% CI, 1.92-6.08]) were significantly more likely to disclose AI use than those from Oceania (<i>P</i> &lt; .05 for all). Conversely, each additional author reduced disclosure odds by 1% (OR, 0.99 [95% CI, 0.97-0.99]). Other characteristics of the journal (acceptance rate, impact factor, general medical/specialty journal, and peer review model) were not associated with AI use disclosure.</p>

![](https://assets.underline.io/markdown_image/1/image/4350638ef812c7db40bddf7b0b71c93c.png)

<h4>Conclusions </h4>
<p>The percentage of submitted articles with self-disclosed AI use was significantly lower in this study than what has been reported in recent surveys of researchers about their general use. Submitting authors may be underreporting their use of AI. </p>
<h4>References</h4>
<p>1. How are researchers responding to AI? <span class="text-roman">Oxford University Press. </span>May 23, 2024. Accessed September 1, 2024. <a href="https://corp.oup.com/news/how-are-researchers-responding-to-ai/">https://corp.oup.com/news/how-are-researchers-responding-to-ai/</a></p>
<p>2. <span class="text-roman">Insights 2024: Attitudes Toward AI.</span> Elsevier. 2024. Accessed September 1, 2024. <a href="https://www.elsevier.com/en-gb/insights/attitudes-toward-ai?utm_source=scalene-peer-review.beehiiv.com&amp;utm_medium=newsletter&amp;utm_campaign=scalene-8-chatgpt-as-reviewer-peer-structured-reviews">https://www.elsevier.com/en-gb/insights/attitudes-toward-ai</a></p>
<p>3. Macdonald H, Abbasi K. Riding the whirlwind: <i>BMJ</i>’s policy on artificial intelligence in scientific publishing. <i>BMJ</i>. 2023;382:1923. doi:10.1136/bmj.p1923</p>
<p><sup>1</sup>Research Center, King Fahad Medical City, Saudi Arabia, ialfayyad@gmail.com; <sup>2</sup>Department of Epidemiology, Care and Public Health Research Institute, Maastricht University, the Netherlands; <sup>3</sup>Department of Epidemiology and Data Science, Amsterdam University Medical Centers, Department of Philosophy, Faculty of Humanities, Vrije Universiteit Amsterdam, the Netherlands; <sup>4</sup><i>BMJ</i>, London, UK. </p>
<h4>Conflict of Interest Disclosures</h4> This research is part of an ongoing PhD collaboration between The BMJ and the team at Meta-Research at Maastricht University (UM) on the responsible conduct of publishing scientific research. The BMJ is published by BMJ Group, a wholly owned subsidiary of the British Medical Association. UM is a public legal entity in the Netherlands. This study is part of Isamme AlFayyad’s self-funded BMJ/UM PhD. No exchange of funds has taken place for this research project. All authors express their own opinions and not necessarily that of their employers. Helen Macdonald and Sara Schroter are full-time employees at BMJ Publishing Group. No other disclosures were reported. 
<h4>Additional Information</h4> The protocol of this research study was registered in Open Science Framework (osf.io/uvqms).



Authors Self-Disclosed Use of Artificial Intelligence in Research Submissions to 49 BMJ Group Biomedical Journals

<p>Roy H. Perlis,<sup>1,2</sup> Annette Flanagin,<sup>3</sup> Jacob Kendall-Taylor,<sup>3</sup> Michael Berkwits,<sup>4</sup> Kirsten Bibbins-Domingo<sup>3</sup></p>
<h4>Objective </h4>
<p>The scientific community is divided about the appropriate role of artificial intelligence (AI) in manuscript preparation.<sup>1</sup> In 2023, JAMA Network journals began requiring authors and reviewers to answer questions about use of AI to create or assist in the creation or editing of submitted manuscripts or with preparation of reviews.<sup>2</sup> This study was conducted to assess author and reviewer declarations of such use.</p>
<h4>Design </h4>
<p>Study of manuscripts and peer reviews submitted to 13 JAMA Network journals from September 2023 to May 2025. Numbers and proportions of manuscripts with corresponding authors and peer reviewers declaring use of AI were examined, along with factors associated with use (eg, journal, country, article type). For authors’ use, reasons, type of AI, and editorial decisions were also assessed; for reviewers’ use, turnaround time and ratings of reviews were assessed. Where sample sizes were sufficient, logistic regression models were run, with use of AI as the dependent variable and article type, journal, and review outcome as independent variables. A 2-sided uncorrected <i>P</i> value of &lt;.05 was considered the threshold for statistical significance. This study was deemed exempt from ethical approval by the Mass General Brigham Institutional Review Board.</p>
<h4>Results </h4>
<p>Of 82,829 manuscripts submitted during the study period, 2257 (2.7%) declared AI use. Authors’ use of AI increased from 1.6% in September 2023 to 4.2% in May 2025 (<i>r</i><sup>2</sup> = .34; <i>P</i> &lt; .01). <b>Table 25-1115</b> contrasts manuscripts with or without disclosed AI use by author country, journal, manuscript type, study type, editorial decision, AI use categories, and common AI models used. Author AI use was highest in <i>JAMA Network Open</i> (4.1%) and <i>JAMA</i> (3.1%) and lowest in <i>JAMA Dermatology</i> (1.6%) and was highest among authors from Taiwan (7.2%). Letters to the Editor (odds ratio [OR] 1.63, 95% CI 1.37-1.94) and Viewpoints (OR 1.44, 95% CI 1.21-1.72) were more likely to involve AI use than Original Investigations. The most common reported reasons for AI use by authors were for language and editing (50.0%), data analysis and statistics (11.8%), and content generation and drafting (8.2%). ChatGPT was the most common AI model used by authors (62.9%). Mean (SD) reviewer turnaround times were 11.8 (8.5) days for those who did not disclose AI use and 12.6 (7.6) days for reviewers who disclosed AI use. Reviewer use of AI was highest for <i>JAMA Network Open</i> (0.9%) and <i>JAMA Psychiatry</i> (0.8%) and among reviewers from the UK (2.7%).</p>

![](https://assets.underline.io/markdown_image/1/image/549162fd8a7ceeca532967bc4026042c.png)

<h4>Conclusions </h4>
<p>Among manuscripts submitted to 13 major medical journals, author declaration of AI use to create or edit content was low but more than doubled during the study period and varied by journal and article type. Very few reviewers disclosed use of AI, possibly because of guidance forbidding such use.<sup>3</sup> Limitations include small numbers of reported AI use, which limited comparisons for some factors, and the likelihood of underreporting of AI use.</p>
<h4>References</h4>
<p>1. Kwon D. Is it OK for AI to write science papers? <i>Nature</i> survey shows researchers are split. <i>Nature</i>. 2025;641:574-578. doi:10.1038/d41586-025-01463-8</p>
<p>2. Flanagin A, Pirracchio R, Khera R, Berkwits M, Hswen Y, Bibbins-Domingo K. Reporting use of AI in research and scholarly publication—JAMA Network guidance. <i>JAMA.</i> 2024;331(13):1096-1098. doi:10.1001/jama.2024.3471</p>
<p>3. Flanagin A, Kendall-Taylor J, Bibbins-Domingo K. Guidance for authors, peer reviewers, and editors on use of AI, language models, and chatbots. <i>JAMA</i>. 2023;330(8):702-703. doi:10.1001/jama.2023.12500</p>
<sup>1</sup>JAMA+ AI and <i>JAMA Network Open</i>, Chicago, IL, rperlis@mgh. harvard.edu; <sup>2</sup>Department of Psychiatry, Mass General Brigham, Boston, MA, US; <sup>3</sup><em>JAMA</em> and the JAMA Network, Chicago, IL, US; <sup>4</sup>Centers for Disease Control and Prevention, Atlanta, GA, US. 
<h4>Conflict of Interest Disclosures</h4> Roy H. Perlis receives payment for service as a scientific advisor to Circular Genomics, Genomind,and Alkermes. Annette Flanagin, Michael Berkwits, and Kirsten Bibbins-Domingo are members of the Peer Review Congress Advisory Board but were not involved in the review or decision for this abstract. 
<h4>Additional Information</h4>Llama 3.3 (Ollama) was used on June 13, 2025, to identify categories of author-declared use of artificial intelligence. Roy H. Perlis takes responsibility for the integrity of the content generated. 


Factors Associated With Author and Reviewer Declared Use of AI in Medical Journals

<p>Daniel S. Evanko,<sup>1</sup> Michael Di Natale<sup>1</sup></p>
<h4>Objective</h4>
<p> This study assessed the ability to reliably detect the use of generative artificial intelligence (genAI) by authors and reviewers in the field of cancer research, quantified this usage over time, and measured the impact of a change in policy on use of genAI by reviewers.</p>
<h4>Design</h4>
<p> In this cross-sectional study, original research manuscripts and reviewer comments submitted to 10 journals of the American Association for Cancer Research from 2021 through 2024 were analyzed for the presence of AI-generated text. The abstract, methods section, and reviewer comments were extracted from the submission system along with author and reviewer metadata. These sections were chosen because (1) the plain text was stored in the submission system, thus eliminating the need for extraction from files, and (2) we hypothesized that authors would most likely use genAI in the abstract and had observed that methods can be prone to false positives. Text was analyzed using an AI detection tool and classified by average AI likelihood score. Discrimination between full de novo generation of text, language translation, and editing was not possible.</p>
<h4>Results</h4>
<p> We analyzed 46,500 abstracts, 46,021 methods sections, and 29,544 reviewer comments. In 18,467 manuscripts submitted from 2021 through Q3 2022, only 7 abstracts—and no methods or reviewer comment text—were scored as highly likely to contain AI-generated text (AI likelihood score, 0.9 to 1). There were very few false positives (abstract: 0.22%, 40/18,467; methods: 0.07%, 13/18,303; reviewer comments: 0.03%, 4/12,908). Detection of AI-generated text in all 3 sources started increasing dramatically beginning in Q1 2023 after the public release of ChatGPT in November 2022 (<b>Figure 25-1180</b>). Detections in reviewer comments dropped by &gt;50% (from 46 to 34) in Q4 2023 after the journals implemented a policy prohibiting use of genAI for peer review before resuming a linear upward trend. In aggregate, the presence of AI-generated text in 2024 varied greatly between abstracts (23%, 2749/11,959), methods text (10%, 1209/11,875), and reviewer comments (4.8%, 348/7211). There was also substantial journal-dependent variation, from 12% to 15% of abstracts at 3 journals (183/1464, 29/214, 32/214) to 28% to 33% of abstracts at 2 journals (203/716, 438/1342). Authors affiliated with institutions in English-speaking countries were less than half as likely to use genAI compared with other authors. The presence of AI-generated text was associated with a substantially higher rejection rate before peer review regardless of institution location. There was a slight association of AI-generated text with lower-rated reviewer comments. </p>

![](https://assets.underline.io/markdown_image/1/image/0a82456056f57c54deb1f9fd0a3d35b4.png)

<h4>Conclusions</h4>
<p> AI-generated text can be detected in manuscripts and reviewer comments with virtually no false positives. GenAI usage is increasing rapidly, but journal policies to regulate usage can have a limited impact. Usage varies considerably between journals and the type of text and appears to be associated with different editorial outcomes.</p>
<sup>1</sup>American Association for Cancer Research, Philadelphia, PA, US, daniel.evanko@aacr.org. 
<h4>Conflict of Interest Disclosures</h4>
Daniel S. Evanko is a member of the Peer Review Congress Advisory Board but was not involved in the review or decision for this abstract. 
<h4>Acknowledgment</h4> 
We thank John Ho for his assistance collecting the data and entering it into the AI detection tool to obtain the AI likelihood scores.


Quantifying and Assessing the Use of Generative AI by Authors and Reviewers in the Cancer Research Field

<p>Ana-Catarina Pinho-Gomes,<sup>1,2</sup> Carinna Hockham,<sup>1</sup> Mark Woodward<sup>1,3</sup></p>
<h4>Objective</h4>
<p> Women are underrepresented among authors of scientific papers. Although the number of retractions has been rising over the past few decades,<sup>1</sup> gender differences among authors of retracted papers remain poorly understood. Therefore, this study investigated gender differences in authorship of retracted articles available on RetractionWatch.</p>
<h4>Design</h4>
<p> RetractionWatch (<a href="http://retractiondatabase.org/">http://retractiondatabase.org/</a>) provided us a dataset comprising 64,658 articles published between January 1971 and June 2025. This is an update of a previous study.<sup>2</sup> No articles were excluded. We extracted data for the first and last author of each paper and inferred their gender based on their first names using Gender-API software. We accepted gender predictions when the accuracy was estimated to be at least 80%. We estimated the percentage of retractions that could be attributed to women and men first authors and last authors overall and by reason. We used binomial distributions to estimate percentages retracted by women authors, with 95% CIs, and tested whether these percentages were significantly different from 50%. We compared women’s representation as authors of retractions vs their representation as authors of articles overall (40% for first authors and 30% for last authors<sup>3</sup>).</p>
<h4>Results</h4>
<p> Among 64,658 articles retracted between 1971 and 2025 that included 64,659 first authors and last authors, women accounted for 26.4% (95% CI, 25.8%-26.9%) of first authors and 23.7% (95% CI, 23.2%-24.2%) of last authors. There were significant differences in gender of authors in retracted articles by reason, and women’s representation appeared to be lower than their representation as authors of scientific articles (<span class="CharOverride-4"><b>Figure 25-0862</b></span>). The lowest representation of women was found for ethics and legal issues (16.6% [95% CI, 14.2%-19.2%] for first authors and 19.4% [95% CI, 17.1%-21.8%] for last authors). For first authors, women’s representation was the highest for errors caused by editors and publishers (33.8% [95% CI, 31.4%-36.4%]). For last authors, the highest representation of women was for plagiarism (28.9% [95% CI, 27.4%-30.6%]). Most retractions (60.9%) had men as first and last authors.</p>

![](https://assets.underline.io/markdown_image/1/image/6e7f48f38cfc6dda1a863b051bc8705f.png)

<h4>Conclusions</h4>
<p> Women’s representation among authors of retracted articles seemed to be slightly lower than women’s representation as authors of scientific papers overall. Women’s underrepresentation as authors of retractions due to fraud and misconduct as well as ethical concerns suggests that gender equality might enhance research integrity and, hence, reduce the negative impact that retractions have on population health and trust in science.</p>
<h4>References</h4>
<p>1. Gaudino M, Robinson NB, Audisio K, et al. Trends and characteristics of retracted articles in the biomedical literature, 1971 to 2020. <i>JAMA Intern Med. </i>2021;181(8):1118-1121. doi:10.1001/jamainternmed.2021.1807</p>
<p>2. Pinho-Gomes A-C, Hockham C, Woodward M. Women’s representation as authors of retracted papers in the biomedical sciences. <i>PLoS ONE</i>. 2023;18(5):e0284403. doi:10.1371/journal.pone.0284403</p>
<p>3. Squazzoni F, Bravo G, Farjam M, et al. Peer review and gender bias: a study on 145 scholarly journals. <i>Science Advances</i>. 2021;7(2):eabd0299. doi:10.1126/sciadv.abd0299</p>
<h4>Affiliations</h4> <sup>1</sup>The George Institute for Global Health, Imperial College London, London, UK, a.pinho-gomes@imperial.ac.uk; <sup>2</sup>Institute for Global Health, University College London, London, UK; <sup>3</sup>The George Institute for Global Health, University of New South Wales, Sydney, New South Wales, Australia. <h4>Conflict of Interest Disclosures</h4> Mark Woodward has been a consultant for Amgen and Freeline. No other disclosures were reported. <h4>Funding/Support</h4> Ana-Catarina Pinho-Gomes is funded by an Academic Clinical Lectureship by the National Institute for Health and Care Research (NIHR). Mark Woodward is supported by the National Health and Medical Research Council of Australia grant APP1174120. <h4>Role of Funder/Sponsor</h4> The funders had no influence in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the abstract; and decision to submit the abstract for presentation. <h4>Disclaimer</h4> The views expressed are those of the authors and not necessarily those of the NIHR or the Department of Health and Social Care.



Comparison of Reasons for Retraction of Biomedical Articles by Women and Men Authors

<p>Tim Kersjes<sup>1</sup></p>
<h4>Objective </h4>
<p>The rise of paper mills presents a significant problem for academic publishers. It is suspected that paper mill actors not only pollute the literature by publishing low-quality or even fake articles; they also try and enhance the credibility of their product by citing their own paper mill articles. This reinforces their credibility and attractiveness for paying customers. Papers mills also attempt to subvert the peer review process by suggesting their own actors as peer reviewers. This study is an analysis of how one successful paper mill used journals that rely on author-suggested reviewers to increase their output and credibility through the creation and use of fake authors and reviewers. The objective was to analyze how these fake personas amplified this paper mill’s success.</p>
<h4>Design </h4>
<p>Following a research integrity investigation by publisher Springer Nature, 55 articles in 4 of their journals (<i>Boundary Value Problems</i>, <i>Journal of Inequalities and Applications</i>, <i>Fixed Point Theory and Applications</i>, and <i>Advances in Difference Equations</i>) were identified to be compromised by a single paper mill, probably operating out of China. These articles were published between September 2013 and August 2020 in journals that asked submitting authors to suggest reviewers. In total, 26 fake personas were identified by confirming with their listed affiliations that these authors were in fact not with these institutions. Network mapping identified the relations between the fake personas and how these aided their publications, as well as citation patterns between articles authored and reviewed by these fake personas.</p>
<h4>Results </h4>
<p>The analysis demonstrated that this paper mill successfully introduced 2 fake personas early on by managing to get these personas as co-authors on published articles. This allowed the paper mill to submit an increasing number of articles (starting with 2 submissions in 2013 to a maximum of 12 submissions in 2016) with new fake personas, with previously published fake personas as suggested reviewers. Because these fake personas infiltrated the peer review process, more fake authors ended up in the published literature, allowing the paper mill to increase its publication output. Of the 26 fake personas, 13 managed to end up as published authors. For these 55 articles, 13 fake personas that are published authors were invited to review for a total of 68 times. In contrast, 13 fake personas without a publication record were only invited 25 times in total. Additionally, 48 of the 55 paper mill articles published by these fake personas cited 2 or more articles by other fake personas in an apparent attempt to further build credibility.</p>
<h4>Conclusions </h4>
<p>Fake personas can serve as both authors and reviewers. Once the paper mill has established a fake persona as credible, it can be used to amplify the credibility of additional fake personas, allowing a paper mill to build a network of fake authors and reviewers to subvert the peer review process. Identity verification and citation analysis may prevent paper mills from gaining success in this way.</p>
<sup>1</sup>Springer Nature, Dordrecht, the Netherlands, tim.kersjes@ springernature.com. 
<h4>Conflict of Interest Disclosures</h4> 
Tim Kersjes is a full-time employee of Springer Nature and owns Springer Nature stock. He also serves on the trustee board of the Committee on Publication Ethics.


Paper Mill Use of Fake Personas to Manipulate the Peer Review Process

<p>Elena Álvarez-García,<sup>1</sup> Daniel Garcia-Costa,<sup>1</sup> Flaminio Squazzoni,<sup>2</sup> Mario Malički,<sup>3,4,5</sup> Bahar Mehmani,<sup>6</sup> Francisco Grimaldo<sup>1</sup></p>
<h4>Objective </h4>
<p>While the publication of review reports increases the transparency of peer review, little is known about the effect it has on the content or quality of reports.<sup>1</sup> The objective of our study was to assess differences in length, information content, and similarity of open vs unpublished reports. </p>
<h4>Design </h4>
<p>This was a cross-sectional study following STROBE guidelines for reporting<sup>2</sup> that compared open with unpublished reports from 233 medical journals from Elsevier and Springer Nature, submitted from 2016 to 2021. Reports were obtained through confidential agreements with the publishers. Content of reports was compared using number of sentences, previously validated models for classification of 8 categories of content,<sup>3</sup> information content score (mean cumulative distribution function values of each category’s sentence count, based on a zipfian distribution, with scores ranging from 0 to 1, where 1 indicates the highest informativeness), and information score dispersion measured by the Gini index (measure of similarity between reports for the same manuscript, where 0 indicates total similarity and 1 indicates total dissimilarity). Number of sentences, information score, and dispersion were compared using Mann-Whitney <i>U</i> tests. The association of information score and Gini index with peer review type, journal quartile of impact factor, and reviewer gender, seniority, and region were explored using generalized linear models. To adjust for multiple comparisons, we considered <i>P</i> &lt; .001 as statistically significant.</p>
<h4>Results</h4>
<p> Compared with unpublished reports (n = 117,250), open peer review reports (n = 40,844) were longer (median [IQR] length, 22 [12-37] sentences vs 17 [10-28] sentences; <i>P</i> &lt; .001) than unpublished reports and had a higher informative content (median [IQR] score, 0.52 [0.35-0.70] vs 0.46 [0.30-0.65]; <i>P</i> &lt; .001), with the largest difference found in the number of suggestion and solution sentences (Figure 25-1081</b>). Women’s reports had a higher information score than men’s reports (difference, 6.3%), and reviewers from non-Western institutions had lower scores than those from Western institutions (difference, −6.0%). Open peer review reports were also more similar to each other (median [IQR] Gini index, 0.19 [0.09-0.33] vs 0.22 [0.11-0.35]; <i>P</i> &lt; .001).</p>

![](https://assets.underline.io/markdown_image/1/image/6f7beaaf8f16b79237fbcb5ba46ff349.png)

<h4>Conclusions </h4>
<p>Our study showed that open peer review reports were longer than traditional unpublished reports, with the greatest differences found in the number of suggestion and solution sentences. These results suggest that increasing the transparency of peer review could lead to more detailed reports that focus on manuscript improvement. <br />A limitation of our study is that we did not have access to published manuscripts and were unable to determine the impact of initial quality of manuscripts beyond journal ranking, nor the impact peer review reports had on manuscript improvement.</p>
<h4>References</h4>
<p>1. Ross-Hellauer T, Horbach SP. Additional experiments required: a scoping review of recent evidence on key aspects of open peer review. <i>Res Eva</i>. Published online February 8, 2024. doi:10.1093/reseval/rvae004</p>
<p>2. von Elm E, Altman DG, Egger M, Pocock SJ, Gøtzsche PC, Vandenbroucke JP; STROBE Initiative. The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) Statement: guidelines for reporting observational studies. <i>Int J Surg</i>. 2014;12(12):1495-1499. doi:10.1016/j.ijsu.2014.07.013</p>
<p>3. Severin A, Strinzel M, Egger M, et al. Relationship between journal impact factor and the thoroughness and helpfulness of peer reviews. <i>PLoS Biol</i>. 2023;21(8):e3002238. doi:10.1371/journal.pbio.3002238</p>
<sup>1</sup>Department of Computer Science, University of Valencia, Valencia, Spain; <sup>2</sup>Department of Social and Political Sciences, University of Milan, Milan, Italy; <sup>3</sup>Stanford Program on Research Rigor and Reproducibility, Stanford University, Stanford, CA, US, mmalicki@ stanford.edu; <sup>4</sup>Department of Epidemiology and Population Health, Stanford University, Stanford, CA, US; <sup>5</sup>Meta-Research Innovation Center at Stanford, Stanford University, Stanford, CA, US; <sup>6</sup>STM Journals, Elsevier, Amsterdam, the Netherlands. <h4>Conflict of Interest Disclosures</h4> Mario Malički was an Editor in Chief of Research Integrity and Peer Review, which is published by Springer Nature, provider of the part of the dataset. Bahar Mehmani is an employee of Elsevier, provider of the other part of the dataset. Bahar Mehmani is a member of the Peer Review Congress Advisory Board but was not involved in the review or decision for this abstract. No other disclosures were reported. <h4>Funding/Support</h4> Elena Álvarez-García, Daniel GarciaCosta, and Francisco Grimaldo have been partially supported by the Regional Ministry of Education, Culture, Universities and Employment of the Generalitat Valenciana under project CIAICO/2022/154. Flaminio Squazzoni was supported by a grant from the Progetti di Rilevante Interesse Nazionale—Italian Ministry of University and Research (grant 202297CKET_00 “ALGOLIT”). <h4>Role of the Funder/Sponsor</h4> The funder was not involved in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the abstract; and decision to submit the abstract for presentation. <h4>Acknowledgment</h4> We gratefully acknowledge the support of Springer Nature, which provided the open peer review reports for our study, specifically Suzuki Limbu, Attilia Czikmantori, and their production team. We also thank all reviewers of Springer Nature journals that participated in signing or allowing their review reports to be published and all journals and editors facilitating these processes. We acknowledge the support on data extraction from the IT staff of Elsevier, specifically Ramsundhar Baskaravelu and his team. This work uses Scopus data provided by Elsevier through the Peer Review Workbench. We also thank Dave Santucci from the Elsevier Scopus API team and Kristy James from the International Center for the Study of Research for their support on data enrichment about authors and reviewers. We thank Josep Monclús for his help on data extraction and preparation during the initial phase of this study.



Comparison of Content in Published and Unpublished Peer Review Reports

<p>Bojan Batalo,<sup>1</sup> Erica K. Shimomoto,<sup>1</sup> Neil Millar<sup>2</sup></p>
<h4>Objective </h4>
<p>Promotional language (hype) in biomedical research writing has increased significantly over the past 40 years and has potential to influence readers’ perceptions and evaluations of evidence.<sup>1-3</sup> Automatic systems capable of detecting and providing feedback on hype may offer a means to foster more objective reporting. However, the absence of formal guidelines for identifying hype represents a barrier for human annotators and the development of such systems. This pilot study develops formal guidelines for classifying hype and evaluates the application of annotated data to automate hype detection via machine learning.</p>
<h4>Design </h4>
<p>Annotation guidelines were developed to classify 11 adjectives commonly associated with novelty and potential hype (eg, <i>creative</i>, <i>first</i>, <i>groundbreaking</i>, <i>innovative</i>). Guidelines followed the following hierarchical decisions: (1) does the adjective imply a positive value judgment?; (2) is it hyperbolic?; (3) is the adjective gratuitous (ie, adds little to the content), amplified (strengthened by modifiers), or coordinated (paired with other promotional adjectives)?; and (4) is the broader sentential context promotional? A total of 550 sentences containing the adjectives (50 per adjective) were randomly sampled from National Institutes of Health grant abstracts funded between 2016 and 2020 and independently annotated by a linguist and 2 computer scientists, with disagreements resolved through discussion, providing the criterion standard. Our choice of grant abstracts as the starting point for our research is due to the highly competitive nature of research funding.<sup>1</sup> The annotated data were split 80:20 into development and hold-out test sets; machine learning algorithms were trained on the development and evaluated on the test set. Additionally, a human baseline was established by an additional researcher blinded to the annotation guidelines but supplied with a broad definition of hype, manually classifying the test set. Experiments tested 4 traditional text classification methods (multinomial Naive Bayes, multivariate Bernoulli Naive Bayes, latent semantic analysis, and support vector machines), with bag-of-words and averaged global vectors for word representation (GloVe) word embeddings as input features (<b>Table 25-0929</b>). Performance was evaluated using accuracy, weighted precision, recall, and F1 scores.</p>

![](https://assets.underline.io/markdown_image/1/image/334927ffdad7b95527ef7831f37feb90.png)

<h4>Results </h4>
<p>Excluding 13 sentences due to nonadherence with the guidelines, the final sample comprised 537 reliably annotated sentences (Cohen κ &gt; 0.94), of which 392 were classified as hype. Among machine learning models, GloVe-based support vector machines outperformed others with an accuracy of 79.6%, approximating the human baseline of 82.4%.</p>
<h4>Conclusions </h4>
<p>The annotation process underscored the subjective nature of assessing promotional language, particularly for context-dependent adjectives like <i>emerging</i> and <i>latest</i>, and the need for refined constructs to capture gradations of promotional language. Despite these limitations, the pilot study indicates the potential for machine learning models trained on well-annotated datasets to contribute to the automated detection of hype. Future steps include modifying the annotation systems and experimenting with large language models under zero/few-shot regimes.</p>
<h4>References</h4>
<p>1. Millar N, Batalo B, Budgell B. Trends in the use of promotional language (hype) in abstracts of successful National Institutes of Health Grant Applications, 1985-2020. <i>JAMA Netw Open</i>. 2022;5(8):e2228676. doi:10.1001/jamanetworkopen.2022.28676</p>
<p>2. Qiu HS, Peng H, Fosse HB, Woodruff TK, Uzzi B. Use of promotional language in grant applications and grant success. <i>JAMA Netw Open</i>. 2024;7(12):e2448696. doi:10.1001/jamanetworkopen.2024.48696</p>
<p>3. Van Den Besselaar P, Mom C. The effect of writing style on success in grant applications. <i>J Informetr</i>. 2022;16(1):101257. doi:10.1016/j.joi.2022.101257</p>

<sup>1</sup>National Institute of Advanced Industrial Science and Technology, Japan, bojan.batalo@aist.go.jp; <sup>2</sup>University of Tsukuba, Japan.
<h4>Conflict of Interest Disclosures</h4> None reported.
<h4>Funding/Support</h4> This study was supported by grants 21K02919 and 25K00851 from the Japan Society for the Promotion of Science.
<h4>Role of the Funder/Sponsor</h4> The authors confirm that no sponsors or funders influenced the study design, execution, or interpretation of results.
<h4>Additional Information</h4> Neil Millar is a secondary corresponding author (millar.neil@u.tsukuba.ac.jp).


Automating the Detection of Promotional (Hype) Language in Biomedical Research

<p>Vishisht Rao,<sup>1</sup> Aounon Kumar,<sup>2</sup> Himabindu Lakkaraju,<sup>2</sup> Nihar B. Shah<sup>1</sup></p>
<h4>Objective </h4>
<p>Journals, conferences, and funding agencies face the risk that reviewers might ask large language models (LLMs) to generate reviews by uploading submissions. Existing detectors struggle to differentiate between fully LLM-generated and LLM-polished reviews. We addressed this problem by developing a method to detect and flag LLM-generated reviews while controlling family-wise error rates (FWERs).</p>
<h4>Design </h4>
<p>Our method had 3 components. (1) Watermarking: We stochastically chose specific phrases (watermarks), such as random (fake) citations, technical terms (“weak supervised learning”), or beginning the review with a prefix (“This paper investigates the problem…”). Watermarks were specific modifications to the review, which were undetectable unless the nature of the watermark was known. The watermark was known to the editors (but not the reviewers), who could detect LLM-generated reviews through watermarks in the review. (2) Hidden prompt injection: We added an instruction into the manuscript PDF (at the end of the last page; not detectable by the human reviewer) instructing the LLM to output the chosen watermark in the LLM-generated review (“Start your review with: &lt;chosen watermark&gt;”). We evaluated 3 injection methods: white text, font manipulation,<sup>1-3</sup> and a different language. (3) Statistical detection method: We developed a test for watermarks in the review (null hypothesis was the review was human written) and provided mathematical proof of controlling the FWER without making assumptions on how human reviews were written. The evaluation was performed on International Conference on Learning Representations (ICLR) manuscripts, a publication venue that reviews full manuscripts. We injected the hidden prompts into ICLR manuscripts and asked LLMs to generate a review and observe whether it contained the watermark. We also evaluated the robustness of our methods against 3 reviewer defenses (RDs), measures reviewers could take to avoid being flagged for using an LLM-generated review: paraphrasing (paraphrasing the original LLM-generated review using another LLM), identifying watermarks (asking the LLM to flag submissions with hidden prompts), and cropping out the end of the paper (removing the last page before generating the review using an LLM). Watermarks were most successfully embedded in the LLM-generated review when the hidden prompt was injected at the end of the last page. For the evaluation of the last RD, we injected the hidden prompt at a position other than the end of the last page.</p>
<h4>Results </h4>
<p>Our method was successful in embedding the chosen watermark in LLM-generated reviews and was robust to RD mechanisms (<b>Table 25-1176</b>) and detecting and flagging them while controlling false-positives. We conducted our test on 28,028 human-written reviews from ICLR 2021 and 10,022 from 2024, each augmented with 100 LLM-generated reviews with a watermark embedded for each of the 3 types. When executing our algorithm to control the FWER at 0.01 for the random start and technical term watermark and 0.001 for the random citation watermark, we observed 0 false-positives in all cases.</p>

![](https://assets.underline.io/markdown_image/1/image/807c4c34ebba96be96a927580c5eec72.png)

![](https://assets.underline.io/markdown_image/1/image/6a9302767961f7d9b0889e6a3b9a9a09.png)

<h4>Conclusions </h4>
<p>We propose a novel method to detect LLM-generated peer reviews. Our evaluations found this method to be successful, making it appealing to journal, conference, and proposal review organizers.</p>
<h4>References</h4>
<p>1. Markwood I, Shen D, Liu Y, Lu Z. Mirage: content masking attack against {information-based} online services. In: <i>26th USENIX Security Symposium</i>. 2017:833-847.</p>
<p>2. Tran D, Jaiswal C. PDFPhantom: exploiting PDF attacks against academic conferences’ paper submission process with counterattack. In: <i>2019 IEEE 10th Annual Ubiquitous Computing, Electronics &amp; Mobile Communication Conference</i>. October 2019:0736-0743.</p>
<p>3. Zou A, Wang Z, Carlini N, Nasr M, Kolter JZ, Fredrikson M. Universal and transferable adversarial attacks on aligned language models. <i>arXiv</i>. Preprint posted online July 27, 2023. doi:10.48550/arXiv:2307.15043</p>

<sup>1</sup>Carnegie Mellon University, Pittsburgh, PA, US, nihars@cs.cmu.edu; <sup>2</sup>Harvard University, Boston, MA, US.
<h4>Conflict of Interest Disclosures</h4> Nihar B. Shah is a member of the Peer Review Congress Advisory Board but was not involved in the review or decision for this abstract.
<h4>Funding/Support</h4> US National Science Foundations Awards 1942124, 2200410; ONR grant N000142212181.

<h4>Role of the Funder/Sponsor</h4> The funders played no role in the design and conduct of the experiment, data analysis, or preparation of the manuscript.
<h4>Acknowledgment</h4> We thank Danish Pruthi for very helpful discussions.


Evaluation of a Method to Detect Peer Reviews Generated by Large Language Models

Fares Alahdab,<sup>1,2,3</sup> Juan Franco,<sup>3,4</sup> Helen Macdonald,<sup>4</sup> Sara Schroter<sup>4</sup>
<h4>Objective</h4>
Peer reviewer fatigue is on the rise, as reviewers are overburdened with increasing tasks and manuscripts to review, making it challenging for journal editors to secure a sufficient number of high-quality reviews.<sup>1</sup> Despite the potential benefits of using large language models (LLMs) in editorial and peer review processes<sup>2</sup> and the potential to support reviewers with tasks, it is not yet known how good they are at peer review. Additionally, covert use of LLMs in peer review is increasingly suspected, with limited knowledge of positive or negative impact. We aim to compare the quality and comprehensiveness of peer reviews produced by 5 LLMs compared with peer reviewers for research submissions.
<h4>Design</h4>
A comparative study of peer reviews produced by LLMs (GPT4o, GPTo3, Claude 3.5, Gemini 1.5 Pro, and Gemini 2.0 Flash) compared with 2 human peer reviews for research submissions between May 2024 and June 2025. Manuscripts were uploaded to Google’s Vertex AI, a private and secure BMJ Group workspace for using LLMs; the same prompt was used for all submissions (single-shot prompting). An experienced editor performed the ratings of all LLM and human reviews using the Review Quality Instrument (RQI),<sup>3</sup> a tool for assessment of review quality comprising 8 questions, each on a 5-point Likert scale. Secondary outcomes include a comprehensiveness score, based on elements of the manuscript that the reviews focused on, and whether they were evaluative in nature. Additionally, once the first decision has been made on the manuscripts, we plan to invite authors to complete a survey about their perceptions of both the LLM review reports and the peer reviewer reports for their manuscripts. We also plan to have a second editor perform the ratings and compute interrater agreement. Reviews produced by LLMs are not included in the decision-making for the manuscripts. Wilcoxon rank sum tests were used to compare LLM and human scores for each RQI item.

<h4>Results</h4> Preliminary data include 35 reviews (25 by LLMs and 10 by peer reviewers) of 5 BMJ submissions. Across 8 RQI items, LLM-generated reviews had higher mean (SD) scores than human reviewers on identifying strengths and weaknesses (4.12 [0.53] vs 2.70 [0.95]; P &lt; 0.001); providing useful comments on the writing, organization, tables, and figures (3.72 [0.79] vs 1.80 [0.63]; P &lt; .001); and constructiveness (4.00 [0.58] vs 3.00 [1.05]; P = .004) (<b>Table 25-1146</b>). Complete data will include 200 submissions to 4 BMJ journals, the secondary outcomes, and authors’ feedback on the LLM reviews.

![](https://assets.underline.io/markdown_image/1/image/b23f8733c131796f4e73f037ea693a62.png)

<h4>Conclusions</h4>
LLM-generated reviews matched or exceeded human reviewers on a few key dimensions of review quality. A fuller analysis will shed more light on the potential value of LLM peer reviews and how they could complement human peer reviewers’ work.
<h4>References</h4>
1. Publons. Global State of Peer Review report. <a href="https://publons.com/community/gspr">https://publons.com/community/gspr</a>

2. Liang W, Zhang Y, Cao H, et al. Can large language models provide useful feedback on research papers? a large-scale empirical analysis. <i>arXiv. </i>Preprint posted October 2,<i> </i>2023. <a href="https://doi.org/10.48550/arXiv.2310.01783">doi.org/10.48550/arXiv.2310.01783</a>

3. van Rooyen S, Black N, Godlee F. Development of the review quality instrument (RQI) for assessing peer reviews of manuscripts.<i> J Clin Epidemiol</i>. 1999;52(7):625-629. doi:10.1016/s0895-4356(99)00047-5

<sup>1</sup>University of Missouri-Columbia, Columbia, MO, US, fares.alahdab@health.missouri.edu; <sup>2</sup>University of Texas Health Science Center, Houston, TX, US; <sup>3</sup>BMJ Evidence-Based Medicine, London, UK; <sup>4</sup><i>BMJ</i>, London, UK.

<h4>Conflict of Interest Disclosures</h4>
<p> None reported.</p>


Downloads

Next from The 10th Peer Review Congress

Forward to the Past - Making Contributors Accountable

Similar lecture

[V] Citation Biases and Citation-Boosting Strategies: A Scoping Review of Predictors

.css-70qvj9{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-align-items:center;-webkit-box-align:center;-ms-flex-align:center;align-items:center;}Downloads

Next from The 10th Peer Review Congress

Forward to the Past - Making Contributors Accountable

Similar lecture

[V] Citation Biases and Citation-Boosting Strategies: A Scoping Review of Predictors

Downloads