United States

&lt;p&gt;Christos P. Kotanidis,&lt;sup&gt;1,2,3&lt;/sup&gt; Sarah Gorey,&lt;sup&gt;1,4&lt;/sup&gt; Harleen Marwah,&lt;sup&gt;1&lt;/sup&gt; Abarna Pearl,&lt;sup&gt;1&lt;/sup&gt; Darren Taichman,&lt;sup&gt;1&lt;/sup&gt; Mary Beth Hamel&lt;sup&gt;1,5&lt;/sup&gt;&lt;/p&gt;
&lt;h4&gt;Objective&lt;/h4&gt;
&lt;p&gt; Editorial review forms the cornerstone of scientific publishing, ensuring that published research adheres to the highest quality standards.&lt;sup&gt;1&lt;/sup&gt; We evaluated how editorial processes shape manuscripts by examining changes in abstracts between submission and publication.&lt;/p&gt;
&lt;h4&gt;Design&lt;/h4&gt;
&lt;p&gt; We assessed research articles submitted to the &lt;i&gt;New England Journal of Medicine&lt;/i&gt; (&lt;i&gt;NEJM&lt;/i&gt;) in 2022 and published in &lt;i&gt;NEJM&lt;/i&gt; or rejected and subsequently published in 437 other journals. Using author names, we searched PubMed for articles within 2 years of submission. Four physicians reviewed search results to match published manuscripts with submitted manuscripts. Abstract similarity was quantified using term frequency–inverse document frequency,&lt;sup&gt;2,3&lt;/sup&gt; assigning a cosine similarity score ranging from 0 (dissimilar) to 1 (identical). To assess if similarity changes reflected substantive differences, masked physician reviewers conducted qualitative comparisons of submitted and published versions of abstracts from clinical trials. Abstracts were assessed for substantive changes in the following 4 domains: trial design, primary outcome, adverse events, and conclusion (TPAC). A score of +1 was assigned for a substantive domain change favoring the published version and −1 for domain change favoring the submitted version, with the domain scores summed, resulting in a total TPAC score range of −4 (submitted better) to +4 (published better).&lt;/p&gt;
&lt;h4&gt;Results&lt;/h4&gt;
&lt;p&gt; We matched 1360 PubMed records from 2756 research articles initially submitted to &lt;i&gt;NEJM&lt;/i&gt;. Of these, 201 were published in &lt;i&gt;NEJM&lt;/i&gt; and 1159 were rejected by &lt;i&gt;NEJM&lt;/i&gt; and published in the 437 other journals. The median (IQR) similarity score between submitted and published abstracts was 0.71 (0.58-0.83). Abstracts of articles published in the 5 general medicine journals with a journal impact factor (JIF) above 50 per the 2023 Clarivate report had significantly lower similarity scores compared with those published in other journals (&lt;span class=&quot;CharOverride-4&quot;&gt;&lt;b&gt;Figure 25-0940&lt;/b&gt;&lt;/span&gt;, A). For qualitative comparisons, we identified abstracts from randomized controlled trials. On average, published abstracts improved in 0.9 domains, with conclusion being the most frequently modified domain (44.2%). Similarity score correlated negatively with TPAC (Spearman ρ coefficient, −0.39; &lt;span class=&quot;CharOverride-4&quot;&gt;&lt;b&gt;Figure 25-0940&lt;/b&gt;&lt;/span&gt;, B), indicating that lower similarity reflected more substantive revisions. Among abstracts published in the 5 high-JIF general medicine journals, 72.1% improved in at least 1 domain, compared with 48.3% of abstracts published in the other 433 journals. This difference was consistent across all TPAC domains, with conclusion changes being the most different (48% vs 27%).&lt;/p&gt;


![](https://assets.underline.io/markdown_image/1/image/bcf1d262f8a68ab13bfbe8e357b30334.png)

&lt;h4&gt;Conclusions&lt;/h4&gt;
&lt;p&gt; Our comparisons of submitted and published abstracts suggest that peer review and editorial processes result in substantial revision of research reports. This report focused on submissions to 1 general medical journal and within a 2-year publication window. These findings reinforce the importance of editorial oversight in scientific publishing, highlighting its role in refining study design communication, reporting of outcomes, and overall clarity in research dissemination.&lt;/p&gt;
&lt;h4&gt;References&lt;/h4&gt;
&lt;p&gt;1. Roll SC. The value and process of high-quality peer review in scientific professional journals. &lt;i&gt;J Diagn Med Sonography&lt;/i&gt;. 2019;35(5):359-362. doi:10.1177/8756479319853800&lt;/p&gt;
&lt;p&gt;2. Aizawa A. An information-theoretic perspective of tf–idf measures. &lt;i&gt;Inf Process &amp;amp; Manage&lt;/i&gt;. 2003;39(1):45-65. doi:10.1016/S0306-4573(02)00021-3&lt;/p&gt;
&lt;p&gt;3. Ramos J. Using tf-idf to determine word relevance in document queries. In: &lt;i&gt;Proceedings of the First Instructional Conference on Machine Learning&lt;/i&gt;. 2003.&lt;/p&gt;
&lt;sup&gt;1&lt;/sup&gt;New England Journal of Medicine, Boston, MA, US, ckotanidis@ nejm.org; &lt;sup&gt;2&lt;/sup&gt;Heart and Vascular Center, Division of Cardiovascular Medicine, Brigham and Women’s Hospital, Harvard Medical School, Boston, MA, US; &lt;sup&gt;3&lt;/sup&gt;Division of Cardiovascular Medicine, Radcliffe Department of Medicine, University of Oxford, Oxford, UK; &lt;sup&gt;4&lt;/sup&gt;Stroke Clinical Trials Network Ireland (SCTNI), University College Dublin Clinical Research Centre, Mater Hospital, Dublin, Ireland; &lt;sup&gt;5&lt;/sup&gt;Division of General Medicine and Primary Care, Department of Medicine, Harvard Medical School, Beth Israel Deaconess Medical Center, Boston, MA, US. &lt;h4&gt;Conflict of Interest Disclosures&lt;/h4&gt; Christos P. Kotanidis, Sarah Gorey, Harleen Marwah, and Abarna Pearl are Editorial Fellows, Darren Taichman is a Deputy Editor, and Mary Beth Hamel is the Executive Editor at the New England Journal of Medicine.



The 10th Peer Review Congress

Changes to Research Article Abstracts Between Submission and Publication

and scientometrics

informatics

quality of reporting

editorial and peer review process

bibliometrics

The Tenth Peer Review Congress featured 3 days for presentations of new research into peer review and all aspects of scientific publication, bias, quality of reporting, and information access and dissemination.

This year’s meeting was hybrid, and all plenary sessions were livestreamed with opportunities to view presentations and engage with participants in person and virtually during and after the meeting. All recordings are now available on Underline for viewing.

<div style="position:relative;padding-top:max(60%,324px);width:100%;height:0;"><iframe style="position:absolute;border:none;width:100%;height:100%;left:0;top:0;" src="https://online.fliphtml5.com/ebtyf/mget/"  seamless="seamless" scrolling="no" frameborder="0" allowtransparency="true" allowfullscreen="true" ></iframe></div>


#### Why? 
Our aim is to encourage research into the quality and credibility of peer review and scientific publication, to establish the evidence base on which scientists can improve the conduct, reporting, and dissemination of scientific research.

#### Who? 
Participants include editors and publishers of scientific peer-reviewed journals, researchers, funders, bibliometric and informatics experts, information innovators, librarians, journalists, policymakers, ethicists, scientific information disseminators, and anyone interested in the progress of the scientific information enterprise and the quality of scientific evidence.

The Congress embraces a wide range of disciplines, including biomedicine, health and life sciences, applied sciences, basic sciences, physical and chemical sciences, mathematics, computer sciences, engineering, economics, and social sciences as well as new and emerging disciplines.

#### What? 
The Congress provides a forum for the presentation and discussion of new research into the quality and credibility of editorial and funding peer review, scientific and scholarly publication, research methods and reporting practices, identification and management of biases, transparency, reproducibility, access, publication and dissemination metrics, identification and management of threats to the integrity of science, and advances in information exchange.

**Follow us**

[![](https://assets.underline.io/markdown_image/1/image/662a22766b9788ed3766cb23cd156e32.png)](https://x.com/peerrevcongress)  [![](https://assets.underline.io/markdown_image/1/image/bb105afb1773a01fa169aa97ca1af49a.png)](https://www.youtube.com/channel/UCsG_0nKi3_Zfyv0yRPdxjWQ)  [![](https://assets.underline.io/markdown_image/1/image/034e1e70a194a9a4178fdf470ef796db.png)](https://bsky.app/profile/peerreviewcongress.bsky.social)



You need to log in with the email address you registered with. 



Please log in to explore this event.

Registration is required to attend this event, please [here](https://peerreviewcongress.org/registration-information/) to secure your ticket. Please check your spam/promotional inbox too.



Please register!

The Tenth Peer Review Congress features 3 days for presentations of new research into peer review and all aspects of scientific publication, bias, quality of reporting, and information access and dissemination.

<p>Christos P. Kotanidis,<sup>1,2,3</sup> Sarah Gorey,<sup>1,4</sup> Harleen Marwah,<sup>1</sup> Abarna Pearl,<sup>1</sup> Darren Taichman,<sup>1</sup> Mary Beth Hamel<sup>1,5</sup></p>
<h4>Objective</h4>
<p> Editorial review forms the cornerstone of scientific publishing, ensuring that published research adheres to the highest quality standards.<sup>1</sup> We evaluated how editorial processes shape manuscripts by examining changes in abstracts between submission and publication.</p>
<h4>Design</h4>
<p> We assessed research articles submitted to the <i>New England Journal of Medicine</i> (<i>NEJM</i>) in 2022 and published in <i>NEJM</i> or rejected and subsequently published in 437 other journals. Using author names, we searched PubMed for articles within 2 years of submission. Four physicians reviewed search results to match published manuscripts with submitted manuscripts. Abstract similarity was quantified using term frequency–inverse document frequency,<sup>2,3</sup> assigning a cosine similarity score ranging from 0 (dissimilar) to 1 (identical). To assess if similarity changes reflected substantive differences, masked physician reviewers conducted qualitative comparisons of submitted and published versions of abstracts from clinical trials. Abstracts were assessed for substantive changes in the following 4 domains: trial design, primary outcome, adverse events, and conclusion (TPAC). A score of +1 was assigned for a substantive domain change favoring the published version and −1 for domain change favoring the submitted version, with the domain scores summed, resulting in a total TPAC score range of −4 (submitted better) to +4 (published better).</p>
<h4>Results</h4>
<p> We matched 1360 PubMed records from 2756 research articles initially submitted to <i>NEJM</i>. Of these, 201 were published in <i>NEJM</i> and 1159 were rejected by <i>NEJM</i> and published in the 437 other journals. The median (IQR) similarity score between submitted and published abstracts was 0.71 (0.58-0.83). Abstracts of articles published in the 5 general medicine journals with a journal impact factor (JIF) above 50 per the 2023 Clarivate report had significantly lower similarity scores compared with those published in other journals (<span class="CharOverride-4"><b>Figure 25-0940</b></span>, A). For qualitative comparisons, we identified abstracts from randomized controlled trials. On average, published abstracts improved in 0.9 domains, with conclusion being the most frequently modified domain (44.2%). Similarity score correlated negatively with TPAC (Spearman ρ coefficient, −0.39; <span class="CharOverride-4"><b>Figure 25-0940</b></span>, B), indicating that lower similarity reflected more substantive revisions. Among abstracts published in the 5 high-JIF general medicine journals, 72.1% improved in at least 1 domain, compared with 48.3% of abstracts published in the other 433 journals. This difference was consistent across all TPAC domains, with conclusion changes being the most different (48% vs 27%).</p>


![](https://assets.underline.io/markdown_image/1/image/bcf1d262f8a68ab13bfbe8e357b30334.png)

<h4>Conclusions</h4>
<p> Our comparisons of submitted and published abstracts suggest that peer review and editorial processes result in substantial revision of research reports. This report focused on submissions to 1 general medical journal and within a 2-year publication window. These findings reinforce the importance of editorial oversight in scientific publishing, highlighting its role in refining study design communication, reporting of outcomes, and overall clarity in research dissemination.</p>
<h4>References</h4>
<p>1. Roll SC. The value and process of high-quality peer review in scientific professional journals. <i>J Diagn Med Sonography</i>. 2019;35(5):359-362. doi:10.1177/8756479319853800</p>
<p>2. Aizawa A. An information-theoretic perspective of tf–idf measures. <i>Inf Process &amp; Manage</i>. 2003;39(1):45-65. doi:10.1016/S0306-4573(02)00021-3</p>
<p>3. Ramos J. Using tf-idf to determine word relevance in document queries. In: <i>Proceedings of the First Instructional Conference on Machine Learning</i>. 2003.</p>
<sup>1</sup>New England Journal of Medicine, Boston, MA, US, ckotanidis@ nejm.org; <sup>2</sup>Heart and Vascular Center, Division of Cardiovascular Medicine, Brigham and Women’s Hospital, Harvard Medical School, Boston, MA, US; <sup>3</sup>Division of Cardiovascular Medicine, Radcliffe Department of Medicine, University of Oxford, Oxford, UK; <sup>4</sup>Stroke Clinical Trials Network Ireland (SCTNI), University College Dublin Clinical Research Centre, Mater Hospital, Dublin, Ireland; <sup>5</sup>Division of General Medicine and Primary Care, Department of Medicine, Harvard Medical School, Beth Israel Deaconess Medical Center, Boston, MA, US. <h4>Conflict of Interest Disclosures</h4> Christos P. Kotanidis, Sarah Gorey, Harleen Marwah, and Abarna Pearl are Editorial Fellows, Darren Taichman is a Deputy Editor, and Mary Beth Hamel is the Executive Editor at the New England Journal of Medicine.



technical paper

<p>Fangwen Zhou,<sup>1</sup> Muhammad Afzal,<sup>2</sup> Rick Parrish,<sup>1</sup> Ashirbani Saha,<sup>3</sup> Wael Abdelkader,<sup>1</sup> R. Brian Haynes,<sup>1</sup> Alfonso Iorio,<sup>1,4</sup> Cynthia Lokker<sup>1</sup></p>
<h4>Objective </h4>
<p>Systematic reviews are considered one of the strongest levels of evidence, providing essential information for clinical guidelines and bedside practice. However, the broad spectrum of review articles, including narrative reviews and other types, complicates the identification of high-quality systematic reviews.<sup>1</sup> This study aims to fine-tune and evaluate pretrained encoder transformers to identify high-quality systematic reviews from review articles using a large, reputable dataset.</p>
<h4>Design </h4>
<p>Review articles from McMaster’s Premium Literature Service (PLUS) were retrieved.<sup>2</sup> Articles were considered rigorous if they were systematic reviews that (1) stated the clinical topic; (2) described methods, including databases searched and inclusion criteria; (3) searched ≥1 major database; (4) reported the number of retrieved, reviewed, and included articles; and (5) did not exclude randomized clinical trials for treatment, primary prevention, quality improvement, or economics reviews; or included “inception cohort” for prognosis reviews.<sup>3</sup> Ground truth was established by research methodology experts using article full texts. Articles from 2003 to 2023 were randomly split 80:10:10 into training, validation, and testing sets. Articles in 2024 were used for external testing. A grid search of 7 pretrained models, 3 learning rates, 5 batch sizes, and 3 random seeds, with or without class weight adjustments, was conducted. Models were trained for ≤10 epochs, with an early stopping patience of 3. Titles and abstracts were used as inputs. Those longer than 512 tokens were truncated, and those shorter were padded. The model that achieved the lowest log loss on the validation set was further evaluated. A threshold of ≥0.50 was used for classification. Bootstrapping of 1000 iterations was used to estimate 95% confidence intervals.</p>
<h4>Results </h4>
<p>Among 40,091 reviews, 26,332 (65.7%) were considered rigorous and 31,593 were used for training. Of the 630 fine-tuned models, BioELECTRA with no class weight adjustments, learning rate 5E-5, batch size 64, and seed 2 had the lowest validation log loss of 0.1840. <b>Table 25-0887</b> details the characteristics of the datasets and the model’s performance, achieving &gt;94% area under the receiver operating characteristic curve and &gt;95% sensitivity. In review articles published in 2024, there was a notable drop in specificity and mild degradation in other metrics.</p>

![](https://assets.underline.io/markdown_image/1/image/6a45410551b49fc307836396dc72f212.png)

<h4>Conclusions </h4>
<p>Pretrained encoder transformer models, such as BioELECTRA, demonstrate robust performance in identifying rigorous systematic reviews based on PLUS’s criteria. These models can streamline the identification of high-quality evidence, reducing manual effort and enhancing the efficiency of evidence curation. Performance degradation in articles in 2024 may be due to changes in article structure and content over time. Additionally, generalizability to other datasets is unknown. Future efforts should establish standardized benchmarking datasets for systematic reviews, develop models for tools, such as AMSTAR 2, and account for temporal data drift to better support knowledge translation.</p>
<h4>References</h4>
<p>1. Shaheen N, Shaheen A, Ramadan A, et al. Appraising systematic reviews: a comprehensive guide to ensuring validity and reliability. <i>Front Res Metr Anal</i>. Published online December 21, 2023. doi:10.3389/frma.2023.1268045</p>
<p>2. Holland J, Haynes RB; McMaster PLUS Team Health Information Research Unit. McMaster Premium Literature Service (PLUS): an evidence-based medicine information service delivered on the Web. <i>AMIA Annu Symp Proc</i>. 2005;2005:340-344. <a href="https://www.ncbi.nlm.nih.gov/pubmed/16779058"><span class="Hyperlink CharOverride-6">https://www.ncbi.nlm.nih.gov/pubmed/16779058</span></a></p>
<p>3. Methodological criteria. McMaster University Health Information Research Unit. Accessed August 19, 2024. <a href="https://hiruweb.mcmaster.ca/hkr/what-we-do/methodologic-criteria"><span class="Hyperlink CharOverride-6">https://hiruweb.mcmaster.ca/hkr/what-we-do/methodologic-criteria</span></a></p>
<sup>1</sup>Health Information Research Unit, Department of Health Research Methods, Evidence, and Impact, Faculty of Health Sciences, McMaster University, Hamilton, Ontario, Canada, lokkerc@mcmaster.ca; <sup>2</sup>Department of Computing, Faculty of Computing, Engineering and the Built Environment, Birmingham City University, Birmingham, UK; <sup>3</sup>Department of Oncology, Faculty of Health Sciences, McMaster University, Hamilton, Ontario, Canada; <sup>4</sup>Department of Medicine, Faculty of Health Sciences, McMaster University, Hamilton, Ontario, Canada.</p>
<h4>Conflict of Interest Disclosures </h4>
<p>McMaster University, a nonprofit public academic institution, has contracts through the Health Information Research Unit under the supervision of Alfonso Iorio and R. Brian Haynes. These contracts involve professional and commercial publishers to provide newly published studies, which are critically appraised for research methodology and assessed for clinical relevance as part of the McMaster Premium Literature Service (McMaster PLUS). Cynthia Lokker and Rick Parrish receive partial compensation, and R. Brian Haynes is remunerated for supervisory responsibilities and royalties. Ashirbani Saha, Fangwen Zhou, Muhammad Afzal, and Wael Abdelkader are not affiliated with McMaster PLUS.</p>
<h4>Funding/Support</h4>
<p> Fangwen Zhou was funded through the Mitacs Business Strategy Internship grant (IT42947) with matching funds from EBSCO Canada.</p>
<h4>Role of the Funder/Sponsor</h4>
<p> The funders were not involved in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the abstract; and decision to submit the abstract for presentation.</p>
<h4>Acknowledgment </h4>
<p>We thank the Digital Research Alliance of Canada for its computational resources.</p>


Domain-specific Pretrained Encoder Transformers for the Identification of Methodologically Rigorous Systematic Reviews: A Retrospective Modeling Study

<p>Beth Waymouth,<sup>1</sup> Heather Slater,<sup>1</sup> Angharad Goode,<sup>1</sup> Katie Allin,<sup>1</sup> Maria Kowalczuk<sup>1</sup></p>
<h4>Objective</h4>
<p> As publishers increasingly adopt artificial intelligence (AI)–driven tools, understanding editor engagement with AI-assisted editorial checks is crucial for maintaining publication standards. This study examines Handling Editors’ (HEs) interaction with an AI-powered check that flags manuscripts needing further verification during peer review and allows HEs and the editorial office (EOF) to collaborate on resolutions. After observing a decline in HE engagement, a survey was conducted to uncover the underlying reasons.</p>
<h4>Design</h4>
<p> We analyzed internal data tracking interaction with the AI-supported ethics check introduced to HEs in 2022. Existing editors were emailed about this new tool and given access and instructions, including a detailed Ethics Guidance document. Newly recruited HEs were introduced to the check during onboarding webinars. Data included the final status of the ethics check, collected quarterly between September 2022 and March 2024. Engagement was defined as the resolution of ethics checks by the HE rather than the EOF. To contextualize the engagement patterns, a survey was distributed to 4627 HEs across our journal portfolio who were active in 2024 and consented to be recontacted. Data were collected for 2 weeks in December 2024, with a reminder issued after the first week.</p>
<h4>Results</h4>
<p> Initial analysis showed that editor engagement dropped from 40% (2899 of 7394 total checks) to 20% (2170 of 10,692) between September 2022 and November 2023. Engagement rose to 25% (1227 of 4766 checks) between January and March 2024; however, this increase was not statistically significant. Of the surveyed editors, 725 (16%) responded to at least the first question. Of 662 respondents, 67% amended the ethics check on at least some manuscripts, with 24% interacting with the check for every manuscript. Among surveyed editors, 33% indicated that they were unaware of the check. In the “Other” category (n = 66), 38 HEs (58%) said no ethics issues required action or manuscripts didn’t need ethical approval; 4 cited unfamiliarity with the check, and 4 responded that they did not remember. Out of 478 responses from HEs not amending the check on some or all manuscripts, 67% cited an expectation that this task fell to the EOF. This was especially true when handling higher manuscript volumes (<b>Table 25-0897</b>).</p>

![](https://assets.underline.io/markdown_image/1/image/02ba50380f883f04074a683fa93e2342.png)

<h4>Conclusions</h4>
<p> These results underscore the risk of editor disengagement when AI-assisted tools lack sufficient onboarding support and adequate training. For AI integration to be effective, publishers must employ consistent monitoring of engagement and explore ways to ensure AI tools are visible, actionable, and reliably applied across editorial checkpoints.</p>
<sup>1</sup>Frontiers Media S.A., Switzerland, maria.kowalczuk@frontiersin.org.</p>
<h4>Conflict of Interest Disclosures</h4>
<p> Heather Slater, Angharad Goode, Katie Allin, and Maria Kowalczuk are employees of Frontiers Media S.A. Beth Waymouth was an employee of Frontiers Media S.A. at the time when the study was conducted and when the abstract was written and originally submitted. Since May 2025, Beth Waymouth has been an independent researcher.</p>
<h4>Acknowledgment</h4>
<p> We would like to thank all our Handling Editors who participated in the survey. We also wish to thank our colleagues Elena Vicario and Simone Ragavooloo for their valuable suggestions and insightful comments on this project and the abstract.</p>


Strategic Insights Into Editor Engagement With AI-Assisted Tools based on Survey and Data Analysis of AI-Assisted Ethics Checks

<p>Jeremy Y. Ng,<sup>1,2,3</sup> Malvika Krishnamurthy,<sup>2,3</sup> Gursimran Deol,<sup>2,3</sup> Wid Al-Zahraa Al-Khafaji,<sup>2,3</sup> Vetrivel Balaji,<sup>4</sup> Magdalene Abebe,<sup>2,3</sup> Jyot Adhvaryu,<sup>2,3</sup> Tejas Karrthik,<sup>2,3</sup> Pranavee Mohanakanthan,<sup>2,3</sup> Adharva Vellaparambil,<sup>2,3</sup> Lex M. Bouter,<sup>5,6</sup> R. Brian Haynes,<sup>7</sup> Alfonso Iorio,<sup>7,8</sup> Cynthia Lokker,<sup>7</sup> Hervé Maisonneuve,<sup>9,10</sup> Ana Marušić,<sup>11</sup> David Moher<sup>1,12</sup></p>
<h4>Objective</h4>
<p> This study aimed to examine the attitudes and perceptions of editors in chief (EICs) of biomedical journals regarding the integration of artificial intelligence chatbots (AICs) into the scholarly publishing process. Prior research has explored AI use in publishing broadly, but limited data exist on EIC perspectives. Although AICs offer opportunities to streamline editorial tasks, such as plagiarism detection and language editing, they also introduce ethical, technical, and operational challenges. Understanding EIC perspectives is critical to shaping guidelines, policies, and training that align with the evolving role of AICs in scholarly publishing.</p>
<h4>Design</h4>
<p> We conducted a cross-sectional survey of EICs of all biomedical journals, inclusive of medical, nursing, health sciences, dentistry, nursing, public health, and pharmacology and toxicology disciplines, published by Springer Nature (including BMC), Taylor &amp; Francis, Elsevier, Wiley, and Sage. Eligible journals were identified through a combination of automated web scraping and manual verification. EICs were invited to participate in an anonymous SurveyMonkey survey conducted over 5 weeks, which included 3 follow-up reminders, from July through August 2024. The survey covered familiarity with AICs, current use, perceived benefits and challenges, and anticipated future roles. Quantitative data were analyzed using descriptive statistics, while qualitative responses were coded and thematically analyzed to identify key themes. Our protocol was registered<sup>1</sup> and followed the CHERRIES reporting guideline.<sup>2</sup></p>
<h4>Results</h4>
<p> Of 3381 EICs contacted, 510 responded (15.1% response rate), with 505 eligible participants and a completion rate of 87.0%. Most were familiar with AICs (66.7% [325 of 487]) but had not used them in editorial workflows (83.7% [401 of 479]). Perceived benefits included enhanced language and grammar support (70.8% [308 of 435]) and plagiarism screening (67.3% [294 of 437]). However, respondents expressed concerns about initial setup and training (83.9% [360 of 429]), ethical risks (80.6% [345 of 428]), and technical reliability (75.2% [322 of 428]). While only 49.6% (240 of 484) of participants reported that their journal had formal AIC policies, 89.5% (419 of 468) of respondents supported training initiatives to promote ethical and effective use. Despite limited current adoption, 78.9% (370 of 469) believed AICs will play an important role in the future of scholarly publishing, and 77.2% (363 of 470) anticipated their significance in advancing scientific research. Themes identified through thematic analysis of open-ended questions included “no AI in authorship or peer review,” referring to EICs’ reporting of current journal and publisher policy on their use, and “ethical, integrity, and privacy concerns,” referring to EIC perceptions of challenges with the use of AICs in the scholarly publishing process. Our study manuscript has been preprinted.<sup>3</sup></p>
<h4>Conclusions</h4>
<p> Biomedical journal EICs recognized the potential of AICs to enhance editorial processes but highlighted critical barriers, including ethical dilemmas, resource limitations, and insufficient policies and training. Structured interventions, including targeted training programs and robust ethical guidelines, are essential for addressing these challenges and ensuring responsible and effective integration of AICs into publishing workflows.</p>
<h4>References</h4>
<p>1. Ng JY, Krishnamurthy M, Balaji V, et al. Attitudes and perceptions of biomedical journal editors-in-chief towards the use of artificial intelligence chatbots in the scholarly publishing process: a cross-sectional survey across multiple publishers. OSF Registries. August 10, 2024. Accessed July 2, 2025. <a href="https://osf.io/xt6f2"><span class="Hyperlink CharOverride-6">https://osf.io/xt6f2</span></a></p>
<p>2. Eysenbach G. Improving the quality of web surveys: the Checklist for Reporting Results of Internet E-Surveys (CHERRIES). <i>J Med Internet Res.</i> 2004;6(3):e34. doi:10.2196/jmir.6.3.e34</p>
<p>3. Ng JY, Krishnamurthy M, Deol G, et al. Attitudes and perceptions of biomedical journal editors-in-chief towards the use of artificial intelligence chatbots in the scholarly publishing process: a cross-sectional survey. <i>medRxiv</i>. Preprint posted online May 27, 2025. doi:10.1101/2025.05.26.25328101</p>
<sup>1</sup>Centre for Journalology, Ottawa Methods Centre, Ottawa Hospital Research Institute, Ottawa, Ontario, Canada, jeremyyng.phd@gmail.com; <sup>2</sup>Institute of General Practice and Interprofessional Care, University Hospital Tübingen, Tübingen, Germany; <sup>3</sup>Robert Bosch Center for Integrative Medicine and Health, Bosch Health Campus, Stuttgart, Germany; <sup>4</sup>Department of Computing and Software, Faculty of Engineering, McMaster University, Hamilton, Ontario, Canada;<sup> 5</sup>Department of Epidemiology and Data Sciences, Amsterdam Universities Medical Center, Amsterdam, the Netherlands, <sup>6</sup>Department of Philosophy, Faculty of Humanities, Vrije Universiteit, Amsterdam, the Netherlands; <sup>7</sup>Department of Health Research Methods, Evidence, and Impact, Faculty of Health Sciences, McMaster University, Hamilton, Ontario, Canada; <sup>8</sup>Department of Medicine, McMaster University, Hamilton, Ontario, Canada; <sup>9</sup>Consultant, Lyon, France; <sup>10</sup>Scientific Committee, Institute of Research and Action on Fraud and Plagiarism in Academia (IRAFPA), Geneva, Switzerland; <sup>11</sup>Department of Research in Biomedicine and Health and Center for Evidence-based Medicine, University of Split School of Medicine, Split, Croatia; <sup>12</sup>School of Epidemiology, Public Health and Preventive Medicine, Faculty of Medicine, University of Ottawa, Ottawa, Ontario, Canada.</p>
<h4>Conflict of Interest Disclosures</h4>
<p> Lex M. Bouter, Ana Marušić, and David Moher are members of the Peer Review Congress Advisory Board but were not involved in the review or decision for this abstract. No other disclosures were reported.</p>


Attitudes and Perceptions of Biomedical Journal Editors in Chief Towards the Use of Artificial Intelligence Chatbots in the Scholarly Publishing Process

<p>Jeremy Y. Ng,<sup>1,2,3</sup> Daivat Bhavsar,<sup>1,2</sup> Neha Dhanvanthry,<sup>1,2</sup> Lex Bouter,<sup>4,5</sup> Teresa M. Chan,<sup>6</sup> Holger Cramer,<sup>1,2</sup> Annette Flanagin,<sup>7</sup> Alfonso Iorio,<sup>8,9</sup> Cynthia Lokker,<sup>8</sup> Hervé Maisonneuve,<sup>10,11</sup> Ana Marušić,<sup>12</sup> David Moher<sup>3,13</sup></p>
<h4>Objective </h4>
<p>Peer review is a cornerstone of scientific publishing, ensuring the rigor and credibility of research. However, it is increasingly strained by underexplored challenges, such as variability in quality, reviewer fatigue, and biases. Artificial intelligence chatbots (AICs) are emerging tools with the potential to address these challenges by automating tasks, such as identifying methodological flaws and improving language clarity. This study aimed to explore peer reviewers’ attitudes and perceptions toward the use of AICs in medical journal peer review, focusing on their benefits, challenges, and ethical implications.</p>
<h4>Design </h4>
<p>A large-scale, international, cross-sectional survey targeting peer reviewers of medical journals was conducted. Eligible participants had completed at least 1 peer review report for a MEDLINE-indexed journal within the last 24 months at the point of invitation. The names and email addresses of a complete sample of 72,847 corresponding authors who published in MEDLINE-indexed journals from September 1, 2024, to October 15, 2024, were collected. The survey was administered and data were collected via the SurveyMonkey survey platform. The survey included questions about participant demographics, familiarity with AICs, experiences with their use, and perceptions of AIC’s roles, benefits, challenges, and ethical implications in peer review. The survey included multiple-choice, yes/no, Likert scale, and open-ended questions to ensure comprehensive data collection. The survey was first piloted with a group of 7 peer reviewers to optimize question clarity and relevance. The survey was launched in April 2025 and remained open for 7 weeks in total. This study’s protocol was registered<sup>1</sup> and the CHERRIES<sup>2</sup> and STROBE<sup>3</sup> reporting guidelines were used to inform the reporting of this survey study.</p>
<h4>Results </h4>
<p>The survey received 1194 respondents from 33,388 opened email invitations in the first 4 weeks, 1018 of whom completed the survey (response rate: 3.0%). The majority (578/1018 [56.8%]) were senior-career researchers (ie, &gt;10 years of career experience). Most reported never having used an AIC for peer review purposes (707/999 [70.8%]), and notably, many (460/1007 [45.7%]) did not anticipate using AICs in the future. However, most respondents indicated interest in receiving training on AIC use for peer review tasks (587/980 [59.9%]). The most-recognized benefit of AIC use in peer review processes (ie, ”agree” or “strongly agree”) was “reduc[ing] the workload” (601/958 [62.7%]). Conversely, the most-recognized challenge was “risk producing errors or inaccuracies” (738/934 [79.0%]).</p>
<h4>Conclusions </h4>
<p>This study provides critical insights into peer reviewers’ attitudes and perceptions toward the use of AICs in the peer review process. Findings may inform the development of evidence-based guidelines and policies to ensure the ethical, transparent, and effective use of AICs, contributing to improvements in efficiency, quality, and equity in scholarly publishing.</p>
<h4>References</h4>
<p>1. Ng JY, Bhavsar D, Dhanvanthry N, et al. Attitudes and perceptions toward the use of artificial intelligence chatbots in medical journal peer review: a large-scale, international cross-sectional survey. OSF. May 21, 2025. <a href="https://osf.io/fhc2m"><span class="Hyperlink CharOverride-6">https://osf.io/fhc2m</span></a></p>
<p>2. Eysenbach G. Improving the quality of Web surveys: the Checklist for Reporting Results of Internet E-Surveys (CHERRIES). <i>J Med Internet Res</i>. 2004;6(3):e34. doi:10.2196/jmir.6.3.e34</p>
<p>3. von Elm E, Altman DG, Egger M, Pocock SJ, Gøtzsche PC, Vandenbroucke JP; STROBE Initiative. The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies. <i>The Lancet</i>. 2007;370(9596):1453-1457. doi:10.1016/S0140-6736(07)61602-X</p>
<sup>1</sup>Institute of General Practice and Interprofessional Care, University Hospital Tübingen, Tübingen, Germany, jeremyyng.phd@gmail.com; <sup>2</sup>Robert Bosch Center for Integrative Medicine and Health, Bosch Health Campus, Stuttgart, Germany; <sup>3</sup>Centre for Journalology, Ottawa Methods Centre, Ottawa Hospital Research Institute, Ottawa, Ontario, Canada; <sup>4</sup>Department of Epidemiology and Data Science, Amsterdam University Medical Centers, Amsterdam, the Netherlands; <sup>5</sup>Department of Philosophy, Vrije Universiteit Amsterdam, Amsterdam, the Netherlands; <sup>6</sup>School of Medicine, Toronto Metropolitan University, Toronto, Ontario, Canada; <sup>7</sup><i>JAMA</i> and the JAMA Network, Chicago, IL, US; <sup>8</sup>Department of Health Research Methods, Evidence, and Impact, Faculty of Health Sciences, McMaster University, Hamilton, Ontario, Canada; <sup>9</sup>Department of Medicine, McMaster University, Hamilton, Ontario, Canada; <sup>10</sup>Consultant, Lyon, France; <sup>11</sup>Scientific Committee, Institute of Research and Action on Fraud and Plagiarism in Academia (IRAFPA), Geneva, Switzerland; <sup>12</sup>Department of Research in Biomedicine and Health and Center for Evidence-based Medicine, University of Split School of Medicine, Split, Croatia; <sup>13</sup>School of Epidemiology, Public Health and Preventive Medicine, Faculty of Medicine, University of Ottawa, Ottawa, Canada.</p>
<h4>Conflict of Interest Disclosures</h4>
<p> Lex Bouter, Annette Flanagin, Ana Marušić, and David Moher are members of the Peer Review Congress Advisory Board but were not involved in the review or decision for this abstract.</p>


Attitudes and Perceptions Toward the Use of Artificial Intelligence Chatbots in Medical Journal Peer Review: A Large-Scale, International Cross-Sectional Survey

<p>Alexander Goldberg,<sup>1</sup> Ihsan Ullah,<sup>2</sup> Thanh Gia Hieu Khuong,<sup>3</sup> Benedictus Kent Rachmat,<sup>3</sup> Zhen (Zach) Xu,<sup>4</sup> Isabelle Guyon,<sup>2,3,5</sup> Nihar B. Shah<sup>1</sup></p>
<h4>Objective </h4>
<p>This study assessed the utility of a large language model (LLM) in evaluating compliance with submission standards at the 2024 Neural Information Processing Systems (NeurIPS) conference—a top-tier publication venue in artificial intelligence (AI) with a 15% to 25% acceptance rate that reviews full papers, not just abstracts. NeurIPS requires authors to complete a 15-question checklist promoting reproducibility, transparency, and ethical standards, similar to CONSORT<sup>1</sup> and STROBE.<sup>2</sup> We introduced an optional Checklist Assistant using GPT-4 (OpenAI) to provide presubmission feedback to authors on checklist accuracy.</p>
<h4>Design </h4>
<p>We conducted a cross-sectional study in which the Checklist Assistant was available to authors 8 days before the submission deadline. Authors could optionally use the Checklist Assistant, which used a general purpose third-party LLM (gpt-4-turbo-preview) to evaluate the accuracy of their responses to a 15-question checklist. For each question, the LLM received the author’s response, justification, and the full paper (including appendices), and assessed the accuracy of the response using simple prompt engineering. We conducted surveys about authors’ perceptions of the tool at registration and after using the tool. We received 539 preusage survey responses (out of 17,491 total papers submitted to the conference), 234 submissions to the Checklist Assistant, and 65 distinct postusage survey responses. To evaluate robustness of the Checklist Assistant, we manipulated responses to the Assistant to test whether it would be possible for authors to manipulate the Assistant’s scores of checklist accuracy.</p>
<h4>Results</h4>
<p> In 65 postusage surveys (<b>Figure 25-0958</b>), 46 authors found the Assistant useful and 46 indicated they would revise their papers or checklist responses based on its feedback. Authors’ expectations for the Checklist Assistant were more positive (59 of 65 positive responses on usefulness) than their perceptions after usage (46 of 65), potentially reflecting an overly optimistic outlook on the usefulness of the tool. Analysis of resubmissions indicated that authors made substantive revisions to their submissions in response to specific feedback from the LLM. Of 40 instances when authors submitted to the Assistant multiple times, they changed their answers 39 of 40 times and often increased the length of their checklist justifications significantly. Inaccuracy (20 of 52 free-form survey responses) and excessive strictness (14 of 52) were the most frequent issues flagged by authors. For more than half of submissions, the LLM recommended changes to at least 12 of 15 checklist items. Authors most commonly reported plans to improve justifications by adding detail or citations (n = 14) and clarify experimental details or data descriptions (n = 6). On 14 of 15 questions, the Assistant could be manipulated to improve accuracy scores by changing checklist content without improving the paper.</p>

![](https://assets.underline.io/markdown_image/1/image/be46bf0146e81fb335482d0967f37894.png)

<h4>Conclusions</h4>
<p> In this study, an LLM Checklist Assistant was shown to aid a small group of authors to ensure scientific rigor. However, the tool should not be used as a fully automated review tool that replaces human review.</p>
<h4><a id="_idTextAnchor004"></a>References</h4>
<p>1. Moher D, Schulz KF, Altman DG. The CONSORT statement: revised recommendations for improving the quality of reports of parallel-group randomised trials. <i>Lancet</i>. 2001:1191-1194.</p>
<p>2. Vandenbroucke JP, et al. Strengthening the Reporting of Observational Studies in Epidemiology (STROBE): explanation and elaboration. <i>Int J Surg</i>. 2014:1500-1524.</p>
<sup>1</sup>School of Computer Science, Carnegie Mellon University, Pittsburgh, PA, US, akgoldbe@andrew.cmu.edu; <sup>2</sup>ChaLearn, Berkeley, CA, US; <sup>3</sup>University of Paris-Saclay, Paris, France; <sup>4</sup>Department of Computer Science, The University of Chicago, Chicago, IL, US; <sup>5</sup>Google DeepMind, San Francisco, CA, US.</p>
<h4>Conflict of Interest Disclosures </h4>
<p>Nihar B. Shah is a member of the Peer Review Congress Advisory Board but was not involved in the review or decision for this abstract.</p>
<h4>Funding/Support </h4>
<p>This research project has benefitted from the Microsoft Accelerating Foundation Models Research (AFMR) grant program, an INRIA Google Research grant, the ANR Chair of Artificial Intelligence HUMANIA ANR-19-CHIA-0022, NSF 1942124, and 2200410.</p>
<h4>Acknowledgment </h4>
<p>In preparing this experiment, we received advice and help from many people. We are particularly grateful to the NeurIPS 2024 organizers, including General Chair Amir Globerson; Program Chairs Danielle Belgrave, Cheng Zhang, Angela Fan, Jakub Tomczak, and Ulrich Paquet; and workflow team member Babak Rahmani, for participating in brainstorming discussions and contributing to the design. We have also received input and encouragement from Andrew McCallum of OpenReview, Anurag Acharya from Google Scholar, and Tristan Neuman from the NeurIPS board. Several volunteers contributed ideas and helped with various aspects of the preparation, including Jeremiah Liu, Lisheng Sun, Paulo Henrique Couto, Michael Brenner, Neha Nayak Kennard, and Adrien Pavao. This research project has benefited from the Microsoft AFMR grant program. We are grateful to Marc Schoenauer for supporting this effort with an INRIA Google Research grant. We acknowledge the support of ChaLearn, the ANR Chair of Artificial Intelligence HUMANIA ANR-19-CHIA-0022, NSF 1942124, and 2200410. Importantly, we are thankful to all the participants of the Checklist Assistant for volunteering to try it out and providing their valuable feedback.</p>


Use of an LLM as an Author Checklist Assistant for Scientific Papers: NeurIPS'24 Experiment

<p>Vicente Amado Olivo,<sup>1</sup> Wolfgang Kerzendorf,<sup>1</sup> Nutan Chen,<sup>2</sup> Joshua V. Shields,<sup>1</sup> Bangjing Lu,<sup>1</sup> Andreas Flörs<sup>3</sup></p>
<h4>Objective</h4>
<p> The process of identifying peer reviewers is becoming more difficult due to a surge in submissions and declining acceptance rates for review invitations.<sup>1</sup> Responses to a 2024 IOP Publishing online survey suggest the peer review system is unevenly distributed, with 10% of reviewers conducting 50% of all reviews, while early-career researchers and scholars from underrepresented regions remain underused despite being willing to review.<sup>2</sup> Reviewer identification tools have been introduced to assist editors, but the tools are proprietary and closed source, limiting their accessibility and transparency to the broader scientific community. To expand the global pool of peer reviewers, open-source, AI-powered methods are needed to uniquely identify researchers within the expanding scientific literature and match them with appropriate review opportunities. Current author name disambiguation methods often rely on extensive data features, such as institutional affiliations, email addresses, or publication venues, which are not consistently available across digital libraries. We describe the development of a name disambiguation tool with fewer data features.</p>
<h4>Design</h4>
<p> Given the lack of existing author disambiguation tools for the Smithsonian Astrophysical Observatory/National Aeronautics and Space Administration Astrophysics Data System, we developed the Neural Author Name Disambiguator (NAND), a similarity-based neural network, and trained it on pairs of publications authored by individuals sharing the same name labeled with Open Researcher and Contributor ID (ORCID) identifiers. The training dataset included 2,698,778 pairs balanced between classes (0 if they shared the same ORCID and 1 if they had different ORCIDs), 553,496 unique publications, and 125,486 ORCID profiles of authors from the 2020 ORCID open data snapshot (and annual public data release). NAND was trained to disambiguate publications with minimal features (ie, author name, title, and abstract). We validated NAND performance using standard classification metrics: accuracy (percentage of correctly classified publication pairs), precision (true positives / [true positives + false positives]), recall (true positives / [true positives + false negatives]), and F1 scores (true positives / [true positives + 0.5(false positives + false negatives)]).</p>
<h4>Results</h4>
<p> NAND achieved a mean (SD) 94.64% (0.04%) accuracy on test set pairs of authors within the same name block (eg, J. Smith or Y. Wang). Mean (SD) precision, recall, and F1 scores were 96.67% (0.05%), 95.21% (0.11%), and 95.94% (0.03%), respectively.</p>
<h4>Conclusions</h4>
<p> Combining the disambiguation model with semantic expertise matching techniques could establish a practical framework for identifying qualified and willing reviewers across global institutions.<sup>3</sup> The open framework may help to reduce the burden on overused reviewers by expanding the global pool of available reviewers. While results are promising, this analysis was limited to physics publications; further validation is needed to assess generalizability to other scientific domains.</p>
<h4>References</h4>
<p>1. Publons. 2018 Global State of Peer Review. Accessed January 29, 2025. <a href="https://publons.com/static/Publons-Global-State-Of-Peer-Review-2018.pdf"><span class="Hyperlink CharOverride-6">https://publons.com/static/Publons-Global-State-Of-Peer-Review-2018.pdf</span></a></p>
<p>2. Brigham L, Brent-Jones E, Coombs A, et al. State of peer review 2024. Accessed January 29, 2025. <a href="https://ioppublishing.org/state-of-peer-review-2024-results/"><span class="Hyperlink CharOverride-6">https://ioppublishing.org/state-of-peer-review-2024-results/</span></a></p>
<p>3. Kerzendorf WE, Patat F, Bordelon D, van de Ven G, Pritchard TA. Distributed peer review enhanced with natural language processing and machine learning. <i>Nat Astronomy</i>. 2020;4:711-717. doi:10.1038/s41550-020-1038-y</p>
<sup>1</sup>Michigan State University, East Lansing, MI, US, amadovic@msu.edu; <sup>2</sup>Volkswagen Group, Wolfsburg, Germany; <sup>3</sup>GSI Helmholtzzentrum für Schwerionenforschung, Darmstadt, Germany.</p>
<h4>Conflict of Interest Disclosures</h4>
<p> None reported.</p>
<h4>Funding/Support</h4>
<p> This work is supported in part by the National Science Foundation Research Traineeship Program (DGE-2152014) to Vicente Amado Olivo. Additionally, we gratefully acknowledge the European Space Agency for funding support through a traineeship for Vicente Amado Olivo.</p>
<h4>Additional Information</h4>
<p> We acknowledge the support and guidance from Markus Kissler-Patig and Jan Reerink at the European Space Agency. The authors used the free versions of ChatGPT (OpenAI) and Claude (Anthropic) for brainstorming and editing for logical flow. We take responsibility for the integrity of the content generated.</p>


Accuracy and Precision of a Neural Network Author Name Disambiguator

<p>Peiling Wang,<sup>1</sup> Dietmar Wolfram,<sup>2</sup> Scott Shumate<sup>3</sup></p>
<h4>Objective </h4>
<p>Open peer review processes promote transparency and accountability. As more publishers adopt open peer review,<sup>1</sup> accessible peer reviews become valuable corpora to develop AI tools for enhancing peer review processes.<sup>2</sup> The main purpose of peer review as stated in the Guidelines for Reviewers<sup>3</sup> is, “First, reviews provide constructive advice and recommendations to the authors on how their paper can be improved.” How can we determine the constructiveness of peer review reports? This exploratory study analyzed 11,321 peer review reports of the first version of 4100 manuscripts submitted to, and published in, <i>PeerJ</i> to identify constructive comments. Specifically, this dataset includes first reviews from November 11, 2012, to November 24, 2019. This study examines 3 research questions: (1) To what extent do reviewers provide constructive comments? (2) To what extent do reviewers use hedging to soften criticisms or recommendations? (3) Are there differences in constructive comments between signed and anonymous reviews?</p>
<h4>Design </h4>
<p>The gpt-4o-2024-08-06 model from OpenAI API was used to analyze reviews at the sentence level to identify criticisms or supportive comments about different aspects of the research, including study purpose, hypotheses, methods, results, discussion, and conclusions. A detailed system prompt (<a href="https://zenodo.org/records/15464322">https://zenodo.org/records/15464322</a>) was used with a temperature setting of 0. The model was instructed to categorize each sentence into a structured JSON output specifying section, aspects, evaluation, tone, and mode. Use of hedging tone includes phrases such as “I do not think the investigation was ‘rigorous’” or “has the potential to be much better and more relevant.” Building on the GPT analysis, we derived the following variables to measure review feature aspects, including (1) Pc, the percentage of critical sentences; (2) Ps, the percentage of supportive sentences; and (3) Ph, the percentage of hedging sentences. The constructive index (CI), considering both criticisms and supportive comments, is the percentage of sentences containing research comments.</p>
<h4>Results </h4>
<p>Of the 11,321 peer review reports, 7071 (62.5%) were anonymous and 4250 (37.5%) were signed (<b>Table 25-1055</b>). Few reports were supportive (overall median of 9%). Higher levels (27% of anonymous reviews and 25% of signed reviews) contained criticisms of the research. A similar level of hedging was observed. Significant differences were observed between the 2 groups; signed reviews showed significantly fewer criticisms and higher levels of hedging.</p>

![](https://assets.underline.io/markdown_image/1/image/c3c1db9d57bb2102d48d254f10bb4a4e.png)

<h4>Conclusions </h4>
<p>We measured the extent of research-related comments in peer review reports that authors opted to publish. Anonymous reviewers tend to be more critical. The use of hedging in reviews could obscure clear suggestions on how to improve the manuscript. This research represents an initial effort towards a model for an AI-assisted peer review system. Further analysis should fine-tune the AI algorithms for analysis of reviewer comments. In addition, authors’ perspectives on what constitutes constructive feedback warrant further study. This research had limitations. Review histories were not available for rejected manuscripts or for articles where authors opted out of publishing reviews.</p>
<h4>References</h4>
<p>1. Wolfram D, Wang P, Hembree A, Park H. Open peer review: promoting transparency in open science. <i>Scientometrics. </i>2020;125:1033-1051. doi:10.1007/s11192-020-03488-4</p>
<p>2. Wolfram D, Wang P, Abuzahra F. An exploration of referees’ comments published in open peer review journals: the characteristics of review language and the association between review scrutiny and citations. <i>Res Eval. </i>2021;30(3):314-322<i>. </i>doi:10.1093/reseval/rvab005</p>
<p>3. Technical Community on Real-Time Systems (TCRTS) of the IEEE Computer Society. Guidelines for reviewers. Accessed on May 29, 2025. <a href="https://cmte.ieee.org/tcrts/guidelines-for-reviewers"><span class="Hyperlink CharOverride-6">https://cmte.ieee.org/tcrts/guidelines-for-reviewers</span></a></p>
<sup>1</sup>University of Tennessee, Knoxville, US, peilingw@utk.edu; <sup>2</sup>University of Wisconsin, Milwaukee, US; <sup>3</sup>Austin Peay State University, Clarksville, TN, US.</p>
<h4>Conflict of Interest Disclosures</h4>
<p> None reported.</p>


An AI-assisted Analysis of Published PeerJ Open Peer Reviews

<p>Chirag J. Patel,<sup>1</sup> Arjun K. Manrai,<sup>1</sup> Randall J. Ellis,<sup>1</sup> John P. A. Ioannidis<sup>2</sup></p>
<h4>Background</h4>
<p> Large language models (LLMs) are increasingly used to synthesize biomedical evidence, yet it is unknown whether their inferred exposure-phenotype relationships mirror observational findings.<sup>1,2</sup> </p>
<h4>Objective</h4>
<p> To quantify concordance of exposure-phenotype empirical associations using National Health and Nutrition Examination Survey (NHANES) data with predictions from an LLM (ChatGPT 4o [OpenAI]).</p>
<h4>Methods</h4>
<p> We computed 115,056 exposure-phenotype partial correlations among 560 chemical, dietary, and clinical exposures and 268 phenotypes in 80,000 US adults (1999-2018 NHANES).<sup>3</sup> We then sampled 1500 pairs: 500 significant after Bonferroni correction, 500 significant only after false-discovery rate (FDR) correction, and 500 not significant—to fit within the LLM prompt window. An OpenAI-o1-aug24 model (temperature = 0) acting as a prompted “epidemiologist” was asked to predict the sign, magnitude, and <i>P</i> value of each pair and rated the evidence generated as high, trending, or borderline. Verifiability of LLM-cited references was assessed by estimating κ values and Spearman correlation.</p>
<h4>Results</h4>
<p> The LLM labeled 23 pairs as high evidence, 244 as trending evidence, and 1233 as borderline evidence. Sign concordance with NHANES was 73% for high-confidence pairs (κ = 0.49), 65% for Bonferroni pairs (κ = 0.25), 43% for FDR-only pairs (κ = 0.09), and 12% for borderline pairs (κ approximately 0) (<b>Figure 25-1123</b>). Random sign assignment yielded a κ of 0.0001. Concordance for correlation magnitudes was modest (Spearman ρ = 0.21). Permutation tests confirmed no greater-than-chance performance on nonsense data. Mechanistic explanations dominated high-confidence pairs; 41% of cited references (615 of 1500) were unverifiable.</p>

![](https://assets.underline.io/markdown_image/1/image/ca7fd3c645a90192139dbf1d062ea8bc.png)

<h4>Conclusions</h4>
<p> LLMs captured directionality for well-established exposure-phenotype associations but faltered on borderline signals, highlighting both the promise and current limits of LLM-based evidence synthesis in exposomic epidemiology.</p>
<h4>References</h4>
<p>1. Ioannidis JPA, Loy EY, Poulton R, Chia KS. Researching genetic versus nongenetic determinants of disease: a comparison and proposed unification. <i>Sci Transl Med.</i> 2009;1(7):7ps8. doi:10.1126/scitranslmed.3000247</p>
<p>2. Patel CJ, Ioannidis JPA. Studying the elusive environment in large scale. <i>JAMA</i>. 2014;311(21):2173-2174. doi:10.1001/jama.2014.4129</p>
<p>3. Patel CJ, Rehkopf DH, Leppert JT, et al. Systematic evaluation of environmental and behavioural factors associated with all-cause mortality in the United States National Health and Nutrition Examination Survey. <i>Int J Epidemiol</i>. 2013;42(6):1795-1810. doi:10.1093/ije/dyt208</p>
<sup>1</sup>Department of Biomedical Informatics, Harvard Medical School, Boston, MA, US, chirag@hms.harvard.edu; <sup>2</sup>Department of Medicine, Epidemiology and Population Health, Stanford University School of Medicine, Stanford, CA, US.</p>
<h4>Conflict of Interest Disclosures</h4>
<p> John P. A. Ioannidis is a member of the Peer Review Congress Advisory Board but was not involved in the review or decision for this abstract.</p>
<h4>Funding/Support</h4>
<p> This study was funded by grants R01ES032470 and U24ES036819 from the National Institutes of Environmental Health Sciences and grant R01DK137993 from the National Institute of Diabetes and Digestive and Kidney Diseases.</p>
<h4>Role of Funder/Sponsor</h4>
<p> The sponsors had no role in this study.</p>


Comparing Observational Exposure-Phenotype Correlations with Large Language Model Predictions

<p>Nicholas LaBerge,<sup>1</sup> Sam Zhang,<sup>1,2,3</sup> Quinten McElhiney,<sup>1</sup> Daniel B. Larremore,<sup>1,2</sup> Aaron Clauset<sup>1,2</sup></p>
<h4>Objective</h4>
<p> To make publicly available a deidentified dataset of manuscript submissions and associated editorial metadata at <i>Science</i> and <i>Science Advances</i> (2 elite multidisciplinary journals) and quantify the manuscript and author characteristics associated with outcomes over the 2 stages of evaluation at these highly selective journals: (1) editorial review, when journal editors screen and select a smaller set of submissions for detailed consideration, and (2) peer review, when editors recruit outside experts to evaluate manuscripts. Peer review at elite scientific journals is a high-stakes process whose outcomes have a broad influence in science and society. However, the need to maintain peer review’s confidentiality has limited the range of data available for scientific study. This lack of peer review data makes it difficult to assess how well elite journals achieve the scientific community’s ideals or to improve and test theories to improve their evaluation processes.</p>
<h4>Design</h4>
<p> We introduced and described a manuscript-level dataset of 110,303 deidentified evaluations of manuscripts submitted to <i>Science</i> and <i>Science Advances</i> over a 5-year period (2015-2019), each with a standard set of author and manuscript characteristics, and we conducted a series of logistic regression analyses to quantify the correlates of success in the initial editorial review stage (desk rejections) and the subsequent peer review stage at both journals.</p>
<h4>Results</h4></span> Each manuscript record includes author, editor, and manuscript characteristics, including topic, team size, institutional prestige, gender, geographic region, evaluation scores, and editor decisions; personally identifiable and reidentifiable information, including the text of the reviews, is excluded. Our statistical analyses revealed strong associations with institutional prestige, team size, manuscript topic, and country, which are primarily attributable (via a mediation analysis) to the influence of the editor, even as the tenor of advice from outside experts correlates strongly with the final editorial decision (<b>Figure 25-0972</b>). Corresponding authors who are men appear to have a small but significant advantage with editors and reviewers at <i>Science</i>, while authors based in China have a significant disadvantage.</p>

![](https://assets.underline.io/markdown_image/1/image/e1bd441e68d25c2bc4c3481b9d8f95e7.png)

<p class="abstract_abstract-text ParaOverride-5"><span class="CharOverride-4"><h4>Conclusions</h4></span> This deidentified dataset will support further investigation by peer review researchers into the 2-stage evaluation process of these 2 elite journals, and our results will help quantify the complex interactions among editors, reviewers, manuscript characteristics, and author characteristics at elite journals.</p>
<sup>1</sup>University of Colorado Boulder, Boulder, CO, US, aaron.clauset@ colorado.edu; <sup>2</sup>Santa Fe Institute, Santa Fe, NM, US; <sup>3</sup>University of Vermont, Burlington, VT, US. <h4>Conflict of Interest Disclosures</h4> Aaron Clauset is a Deputy Editor at Science Advances; all other authors declare no conflicts of interest. <h4>Funding/Support</h4> This work was supported by Air Force Office of Scientific Research Award FA9550-19-1-0329 (to Nicholas LaBerge, Sam Zhang, Daniel B. Larremore, and Aaron Clauset), National Science Foundation (NSF) SBE Award 2219609 (to Nicholas LaBerge, Sam Zhang, Daniel B. Larremore, and Aaron Clauset), NSF Graduate Research Fellowship Award DGE 2040434 (to Sam Zhang), and NSF Alan T. Waterman Award SMA-2226343 (to Daniel B. Larremore). <h4>Role of Funder/Sponsor</h4> The funders had no role in this research.



Manuscript Characteristics Associated With Editorial Review and Peer Review Outcomes at Science and Science Advances

<p>Nicola Adamson,<sup>1</sup> Andy Collings<sup>1</sup></p>
<h4>Objective </h4>
<p>In October 2022, <i>eLife</i> announced a Publish, Review, Curate model<sup>1</sup> in which all peer-reviewed submissions are published as reviewed preprints, accompanied by an <i>eLife</i> Assessment<sup>2</sup> and public reviews. Authors decide which revisions to undertake and when to declare a version of record (VOR), without an accept/reject decision after peer review. We investigated the extent to which authors revise and improve their work.</p>
<h4>Design </h4>
<p>An <i>eLife</i> Assessment is prepared for each submission sent for review and subsequent revisions, which summarizes the strength of evidence and significance of findings using terms from a common vocabulary. We extracted terms used in <i>eLife</i> Assessments for all 2918 reviewed preprints and their subsequent versions to the end of 2024. To ensure publications had completed the entire process and we could evaluate the full extent of changes, we selected the 1504 where a VOR had been declared. From these, 213 were excluded because either the VOR or original reviewed preprint had included multiple terms to describe the evidence. Using a retrospective analysis, we studied the remaining 1291 reviewed preprints and their respective VOR, calculating the distributions of and changes to terms between first and final versions and the number of rounds of revision.</p>
<h4>Results </h4>
<p>Panel A in <b>Figure 25-0989</b> shows the terms used to describe the strength of evidence in VORs and the first version of the reviewed preprints. The evidence term improved between versions in 39.4% of cases (n = 509) and remained the same in 57.7% of VORs (n = 745). Of VORs in which the evidence was described as incomplete in the first reviewed preprint, 76.5% (n = 199) improved to solid or better, and in papers originally described as solid, 49.3% improved to convincing or better (n = 217). In papers in which the evidence was originally described as inadequate (n = 23), this improved in 78.3% of instances (n = 18) and approximately one-half ended as solid or better (47.8%, n = 11). The majority of VORs (82.6%, n = 1066) were declared after 1 round of revision and 13.5% (n = 174) after 2 or more rounds. Of the 4.0% of VORs declared without revisions (n = 51), only 13.7% had an evidence term of inadequate (n = 0) or incomplete (n = 7). The significance of the findings (<b>Figure 25-0989</b>, B) was most frequently described using valuable (VOR: 31.1%, n = 401) or important (VOR: 49.0%, n = 633); overall, significance terms remained the same in 78.2% of cases (n = 1010).</p>

![](https://assets.underline.io/markdown_image/1/image/b3f8c42937978d78fd6b86c9caa9c9d2.png)

<h4>Conclusions </h4>
<p>Authors generally complete at least 1 round of revision, even though they have the option to proceed without, and multiple rounds of revision are uncommon. Where the evidence was originally described as incomplete, the majority of authors revised to improve this before the VOR. Terms used to describe the significance of the findings change less often than evidence terms, perhaps due to significance being more closely linked to the original research question under investigation.</p>
<h4>References</h4>
<p>1. Eisen MB, Akhmanova A, Behrens TE, et al. Scientific publishing: peer review without gatekeeping. <i>eLife</i>. October 20, 2022;11:e83889. doi:10.7554/eLife.83889</p>
<p>2. <i>eLife</i>’s New Model: What is an <i>eLife</i> assessment? <i>eLife.</i> October 20, 2022. Accessed January 30, 2025. <a href="https://elifesciences.org/inside-elife/db24dd46/elife-s-new-model-what-is-an-elife-assessment">https://elifesciences.org/inside-elife/db24dd46/elife-s-new-model-what-is-an-elife-assessment</a></p>
<sup>1</sup>eLife Sciences Publications Ltd, 95 Regent Street, Cambridge, CB2 1AW, UK, n.adamson@elifesciences.org. <h4>Conflict of Interest Disclosures</h4> We have no competing interests to declare. Both authors are employed by eLife Sciences Publications Ltd; we received no external funding for this work. <h4>Acknowledgment</h4> We thank Fiona Hutton, Fred Atherden, Emily Packer, and George Currie for their comments and suggestions on the submission, and Fred Atherden for extracting data for analysis.



Downloads

Next from The 10th Peer Review Congress

Domain-specific Pretrained Encoder Transformers for the Identification of Methodologically Rigorous Systematic Reviews: A Retrospective Modeling Study

Similar lecture

Representation of Authors From Low- and Middle-Income Countries (LMIC) in Trials With Participants From LMICs

.css-70qvj9{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-align-items:center;-webkit-box-align:center;-ms-flex-align:center;align-items:center;}Downloads

Next from The 10th Peer Review Congress

Domain-specific Pretrained Encoder Transformers for the Identification of Methodologically Rigorous Systematic Reviews: A Retrospective Modeling Study

Similar lecture

Representation of Authors From Low- and Middle-Income Countries (LMIC) in Trials With Participants From LMICs

Downloads