United States

&lt;p&gt;Alexander Goldberg,&lt;sup&gt;1&lt;/sup&gt; Ivan Stelmakh,&lt;sup&gt;2&lt;/sup&gt; Kyunghyun Cho,&lt;sup&gt;3&lt;/sup&gt; Alice Oh,&lt;sup&gt;4&lt;/sup&gt; Alekh Agarwal,&lt;sup&gt;5&lt;/sup&gt; Danielle Belgrave,&lt;sup&gt;6&lt;/sup&gt; Nihar B. Shah&lt;sup&gt;1&lt;/sup&gt;&lt;/p&gt;
&lt;h4&gt;Objective &lt;/h4&gt;
&lt;p&gt;What biases and errors arise in evaluating the quality of peer reviews? We studied this question, driven by 2 primary motivations: (1) designing incentive systems for high-quality reviewing based on review quality assessments and (2) evaluating experiments within peer review processes.&lt;/p&gt;
&lt;h4&gt;Design &lt;/h4&gt;
&lt;p&gt;We conducted a large-scale study at the 2022 Conference on Neural Information Processing Systems (NeurIPS), a top-tier venue in machine learning, inviting reviewers, conference chairs, and authors to evaluate reviews. This study was approved by Carnegie Mellon University Institutional Review Board. Evaluators rated reviews (scores and text) on overall quality (7-point Likert scale) and individual criteria (constructiveness, coverage, understanding, and substantiation; 5-point Likert scale). To assess uselessly elongated review bias in randomized controlled trials, we created elongated versions of 10 paper reviews by adding noninformative content (duplicated instructions and copied paper abstracts), increasing length from 200-300 to 600-850 words. We randomly assigned 458 reviewers into 2 equally sized groups: the control group evaluated a short review, and the treatment group evaluated an artificially lengthened review. We used a Mann-Whitney &lt;i&gt;U&lt;/i&gt; test with a test statistic that measures the proportion of paper review pairs in which the treatment (long review) was rated higher than the control (short review). To assess author outcome bias, we collected observational data from 3429 authors and 4311 paper reviewers and/or conference organizers evaluating 9870 reviews. We compared author ratings on “accept” vs “reject” reviews of their own papers, controlling for review quality by matching 418 review pairs with similar nonauthor ratings. A Mann-Whitney &lt;i&gt;U&lt;/i&gt; test was conducted on these pairs. To assess interevaluator (dis)agreement, miscalibration, and subjectivity, we estimated the magnitude of various sources of error using the observational data described above.&lt;/p&gt;
&lt;h4&gt;Results &lt;/h4&gt;
&lt;p&gt;Evaluators exhibited a significant elongated review bias (&lt;i&gt;P&lt;/i&gt; &amp;lt; .001). Elongated reviews received a mean score of 4.29 vs. 3.73 for original reviews. Each individual criterion also showed length bias (&lt;i&gt;P&lt;/i&gt; &amp;lt; .05). Authors were positively biased toward reviews recommending acceptance (&lt;span class=&quot;CharOverride-4&quot;&gt;&lt;b&gt;Figure 24-0809&lt;/b&gt;&lt;/span&gt;). Authors gave “reject” reviews a mean score of 3.24 compared with 4.65 on “accept” reviews. We also found a statistically significant bias in criteria scores (&lt;i&gt;P&lt;/i&gt; &amp;lt; .001 for all criteria). The magnitude of error due to inconsistency, miscalibration, and subjectivity were similar for reviews of reviews and for reviews of papers at NeurIPS.&lt;/p&gt;

![](https://assets.underline.io/markdown_image/1/image/688e99baaf2057220b6c8658dfb72e64.png)

&lt;h4&gt;Conclusions &lt;/h4&gt;
&lt;p&gt;Our results suggest that the various problems that exist in reviews of papers, most notably bias toward factors like length and review recommendation, also arise in reviewing of reviews. Since our experiment in 2022, our results are increasingly relevant to the evaluation of large language models in peer review settings,&lt;sup&gt;1,2&lt;/sup&gt; where the length bias we established through our randomized controlled trial can bias results on the perceived efficacy of large language models.&lt;sup&gt;3&lt;/sup&gt;&lt;/p&gt;
&lt;h4&gt;References&lt;/h4&gt;
&lt;p&gt;1. Thakkar N, Yuksekgonul M, Silberg J, et al. Can LLM feedback enhance review quality? a randomized study of 20K reviews at ICLR&amp;#160;2025. &lt;i&gt;arXiv&lt;/i&gt;. Preprint posted online April 13, 2025. doi:10.48550/arXiv.2504.09737&lt;/p&gt;
&lt;p&gt;2. D’Arcy M, Hope T, Birnbaum L, Downey D. MARG: multi-agent review generation for scientific papers. &lt;i&gt;arXiv&lt;/i&gt;. Preprint posted online January 8, 2024. doi:10.48550/arXiv.2401.04259&lt;/p&gt;
&lt;p&gt;3. Steyvers M, Tejeda H, Kumar A, et al. What large language models know and what people think they know. &lt;i&gt;Nature Machine Intelligence&lt;/i&gt;. 2025;7:221-231. doi:10.1038/s42256-024-00976-7&lt;/p&gt;
&lt;sup&gt;1&lt;/sup&gt;School of Computer Science, Carnegie Mellon University, Pittsburgh, PA, US, akgoldbe@andrew.cmu.edu; &lt;sup&gt;2&lt;/sup&gt;New Economic School, Moscow, Russia; &lt;sup&gt;3&lt;/sup&gt;Center for Data Science, New York University, New York, NY, US; &lt;sup&gt;4&lt;/sup&gt;School of Computing, KAIST, Daejeon, South Korea; &lt;sup&gt;5&lt;/sup&gt;Google Research, New York, NY; &lt;sup&gt;6&lt;/sup&gt;GSK, London, UK. &lt;h4&gt;Conflict of Interest Disclosures&lt;/h4&gt; Kyunghyun Cho is affiliated with Genentech. Nihar B. Shah is a member of the Peer Review Congress Advisory Board but was not involved in the review or decision for this abstract. &lt;h4&gt;Acknowledgment&lt;/h4&gt; We are greatly indebted to the participants of this experiment for providing evaluations of reviews, thereby helping understand the promises and challenges of evaluating review quality, and consequently also shedding light on the design of incentives and experiments in peer review.






The 10th Peer Review Congress

Peer Reviews of Peer Reviews: A Randomized Controlled Trial and Other Assessments

editorial and peer review process

peer review

quality assurance

The Tenth Peer Review Congress featured 3 days for presentations of new research into peer review and all aspects of scientific publication, bias, quality of reporting, and information access and dissemination.

This year’s meeting was hybrid, and all plenary sessions were livestreamed with opportunities to view presentations and engage with participants in person and virtually during and after the meeting. All recordings are now available on Underline for viewing.

<div style="position:relative;padding-top:max(60%,324px);width:100%;height:0;"><iframe style="position:absolute;border:none;width:100%;height:100%;left:0;top:0;" src="https://online.fliphtml5.com/ebtyf/mget/"  seamless="seamless" scrolling="no" frameborder="0" allowtransparency="true" allowfullscreen="true" ></iframe></div>


#### Why? 
Our aim is to encourage research into the quality and credibility of peer review and scientific publication, to establish the evidence base on which scientists can improve the conduct, reporting, and dissemination of scientific research.

#### Who? 
Participants include editors and publishers of scientific peer-reviewed journals, researchers, funders, bibliometric and informatics experts, information innovators, librarians, journalists, policymakers, ethicists, scientific information disseminators, and anyone interested in the progress of the scientific information enterprise and the quality of scientific evidence.

The Congress embraces a wide range of disciplines, including biomedicine, health and life sciences, applied sciences, basic sciences, physical and chemical sciences, mathematics, computer sciences, engineering, economics, and social sciences as well as new and emerging disciplines.

#### What? 
The Congress provides a forum for the presentation and discussion of new research into the quality and credibility of editorial and funding peer review, scientific and scholarly publication, research methods and reporting practices, identification and management of biases, transparency, reproducibility, access, publication and dissemination metrics, identification and management of threats to the integrity of science, and advances in information exchange.

**Follow us**

[![](https://assets.underline.io/markdown_image/1/image/662a22766b9788ed3766cb23cd156e32.png)](https://x.com/peerrevcongress)  [![](https://assets.underline.io/markdown_image/1/image/bb105afb1773a01fa169aa97ca1af49a.png)](https://www.youtube.com/channel/UCsG_0nKi3_Zfyv0yRPdxjWQ)  [![](https://assets.underline.io/markdown_image/1/image/034e1e70a194a9a4178fdf470ef796db.png)](https://bsky.app/profile/peerreviewcongress.bsky.social)



You need to log in with the email address you registered with. 



Please log in to explore this event.

Registration is required to attend this event, please [here](https://peerreviewcongress.org/registration-information/) to secure your ticket. Please check your spam/promotional inbox too.



Please register!

The Tenth Peer Review Congress features 3 days for presentations of new research into peer review and all aspects of scientific publication, bias, quality of reporting, and information access and dissemination.

<p>Alexander Goldberg,<sup>1</sup> Ivan Stelmakh,<sup>2</sup> Kyunghyun Cho,<sup>3</sup> Alice Oh,<sup>4</sup> Alekh Agarwal,<sup>5</sup> Danielle Belgrave,<sup>6</sup> Nihar B. Shah<sup>1</sup></p>
<h4>Objective </h4>
<p>What biases and errors arise in evaluating the quality of peer reviews? We studied this question, driven by 2 primary motivations: (1) designing incentive systems for high-quality reviewing based on review quality assessments and (2) evaluating experiments within peer review processes.</p>
<h4>Design </h4>
<p>We conducted a large-scale study at the 2022 Conference on Neural Information Processing Systems (NeurIPS), a top-tier venue in machine learning, inviting reviewers, conference chairs, and authors to evaluate reviews. This study was approved by Carnegie Mellon University Institutional Review Board. Evaluators rated reviews (scores and text) on overall quality (7-point Likert scale) and individual criteria (constructiveness, coverage, understanding, and substantiation; 5-point Likert scale). To assess uselessly elongated review bias in randomized controlled trials, we created elongated versions of 10 paper reviews by adding noninformative content (duplicated instructions and copied paper abstracts), increasing length from 200-300 to 600-850 words. We randomly assigned 458 reviewers into 2 equally sized groups: the control group evaluated a short review, and the treatment group evaluated an artificially lengthened review. We used a Mann-Whitney <i>U</i> test with a test statistic that measures the proportion of paper review pairs in which the treatment (long review) was rated higher than the control (short review). To assess author outcome bias, we collected observational data from 3429 authors and 4311 paper reviewers and/or conference organizers evaluating 9870 reviews. We compared author ratings on “accept” vs “reject” reviews of their own papers, controlling for review quality by matching 418 review pairs with similar nonauthor ratings. A Mann-Whitney <i>U</i> test was conducted on these pairs. To assess interevaluator (dis)agreement, miscalibration, and subjectivity, we estimated the magnitude of various sources of error using the observational data described above.</p>
<h4>Results </h4>
<p>Evaluators exhibited a significant elongated review bias (<i>P</i> &lt; .001). Elongated reviews received a mean score of 4.29 vs. 3.73 for original reviews. Each individual criterion also showed length bias (<i>P</i> &lt; .05). Authors were positively biased toward reviews recommending acceptance (<span class="CharOverride-4"><b>Figure 24-0809</b></span>). Authors gave “reject” reviews a mean score of 3.24 compared with 4.65 on “accept” reviews. We also found a statistically significant bias in criteria scores (<i>P</i> &lt; .001 for all criteria). The magnitude of error due to inconsistency, miscalibration, and subjectivity were similar for reviews of reviews and for reviews of papers at NeurIPS.</p>

![](https://assets.underline.io/markdown_image/1/image/688e99baaf2057220b6c8658dfb72e64.png)

<h4>Conclusions </h4>
<p>Our results suggest that the various problems that exist in reviews of papers, most notably bias toward factors like length and review recommendation, also arise in reviewing of reviews. Since our experiment in 2022, our results are increasingly relevant to the evaluation of large language models in peer review settings,<sup>1,2</sup> where the length bias we established through our randomized controlled trial can bias results on the perceived efficacy of large language models.<sup>3</sup></p>
<h4>References</h4>
<p>1. Thakkar N, Yuksekgonul M, Silberg J, et al. Can LLM feedback enhance review quality? a randomized study of 20K reviews at ICLR&#160;2025. <i>arXiv</i>. Preprint posted online April 13, 2025. doi:10.48550/arXiv.2504.09737</p>
<p>2. D’Arcy M, Hope T, Birnbaum L, Downey D. MARG: multi-agent review generation for scientific papers. <i>arXiv</i>. Preprint posted online January 8, 2024. doi:10.48550/arXiv.2401.04259</p>
<p>3. Steyvers M, Tejeda H, Kumar A, et al. What large language models know and what people think they know. <i>Nature Machine Intelligence</i>. 2025;7:221-231. doi:10.1038/s42256-024-00976-7</p>
<sup>1</sup>School of Computer Science, Carnegie Mellon University, Pittsburgh, PA, US, akgoldbe@andrew.cmu.edu; <sup>2</sup>New Economic School, Moscow, Russia; <sup>3</sup>Center for Data Science, New York University, New York, NY, US; <sup>4</sup>School of Computing, KAIST, Daejeon, South Korea; <sup>5</sup>Google Research, New York, NY; <sup>6</sup>GSK, London, UK. <h4>Conflict of Interest Disclosures</h4> Kyunghyun Cho is affiliated with Genentech. Nihar B. Shah is a member of the Peer Review Congress Advisory Board but was not involved in the review or decision for this abstract. <h4>Acknowledgment</h4> We are greatly indebted to the participants of this experiment for providing evaluations of reviews, thereby helping understand the promises and challenges of evaluating review quality, and consequently also shedding light on the design of incentives and experiments in peer review.






technical paper

<p>Fangwen Zhou,<sup>1</sup> Muhammad Afzal,<sup>2</sup> Rick Parrish,<sup>1</sup> Ashirbani Saha,<sup>3</sup> Wael Abdelkader,<sup>1</sup> R. Brian Haynes,<sup>1</sup> Alfonso Iorio,<sup>1,4</sup> Cynthia Lokker<sup>1</sup></p>
<h4>Objective </h4>
<p>Systematic reviews are considered one of the strongest levels of evidence, providing essential information for clinical guidelines and bedside practice. However, the broad spectrum of review articles, including narrative reviews and other types, complicates the identification of high-quality systematic reviews.<sup>1</sup> This study aims to fine-tune and evaluate pretrained encoder transformers to identify high-quality systematic reviews from review articles using a large, reputable dataset.</p>
<h4>Design </h4>
<p>Review articles from McMaster’s Premium Literature Service (PLUS) were retrieved.<sup>2</sup> Articles were considered rigorous if they were systematic reviews that (1) stated the clinical topic; (2) described methods, including databases searched and inclusion criteria; (3) searched ≥1 major database; (4) reported the number of retrieved, reviewed, and included articles; and (5) did not exclude randomized clinical trials for treatment, primary prevention, quality improvement, or economics reviews; or included “inception cohort” for prognosis reviews.<sup>3</sup> Ground truth was established by research methodology experts using article full texts. Articles from 2003 to 2023 were randomly split 80:10:10 into training, validation, and testing sets. Articles in 2024 were used for external testing. A grid search of 7 pretrained models, 3 learning rates, 5 batch sizes, and 3 random seeds, with or without class weight adjustments, was conducted. Models were trained for ≤10 epochs, with an early stopping patience of 3. Titles and abstracts were used as inputs. Those longer than 512 tokens were truncated, and those shorter were padded. The model that achieved the lowest log loss on the validation set was further evaluated. A threshold of ≥0.50 was used for classification. Bootstrapping of 1000 iterations was used to estimate 95% confidence intervals.</p>
<h4>Results </h4>
<p>Among 40,091 reviews, 26,332 (65.7%) were considered rigorous and 31,593 were used for training. Of the 630 fine-tuned models, BioELECTRA with no class weight adjustments, learning rate 5E-5, batch size 64, and seed 2 had the lowest validation log loss of 0.1840. <b>Table 25-0887</b> details the characteristics of the datasets and the model’s performance, achieving &gt;94% area under the receiver operating characteristic curve and &gt;95% sensitivity. In review articles published in 2024, there was a notable drop in specificity and mild degradation in other metrics.</p>

![](https://assets.underline.io/markdown_image/1/image/6a45410551b49fc307836396dc72f212.png)

<h4>Conclusions </h4>
<p>Pretrained encoder transformer models, such as BioELECTRA, demonstrate robust performance in identifying rigorous systematic reviews based on PLUS’s criteria. These models can streamline the identification of high-quality evidence, reducing manual effort and enhancing the efficiency of evidence curation. Performance degradation in articles in 2024 may be due to changes in article structure and content over time. Additionally, generalizability to other datasets is unknown. Future efforts should establish standardized benchmarking datasets for systematic reviews, develop models for tools, such as AMSTAR 2, and account for temporal data drift to better support knowledge translation.</p>
<h4>References</h4>
<p>1. Shaheen N, Shaheen A, Ramadan A, et al. Appraising systematic reviews: a comprehensive guide to ensuring validity and reliability. <i>Front Res Metr Anal</i>. Published online December 21, 2023. doi:10.3389/frma.2023.1268045</p>
<p>2. Holland J, Haynes RB; McMaster PLUS Team Health Information Research Unit. McMaster Premium Literature Service (PLUS): an evidence-based medicine information service delivered on the Web. <i>AMIA Annu Symp Proc</i>. 2005;2005:340-344. <a href="https://www.ncbi.nlm.nih.gov/pubmed/16779058"><span class="Hyperlink CharOverride-6">https://www.ncbi.nlm.nih.gov/pubmed/16779058</span></a></p>
<p>3. Methodological criteria. McMaster University Health Information Research Unit. Accessed August 19, 2024. <a href="https://hiruweb.mcmaster.ca/hkr/what-we-do/methodologic-criteria"><span class="Hyperlink CharOverride-6">https://hiruweb.mcmaster.ca/hkr/what-we-do/methodologic-criteria</span></a></p>
<sup>1</sup>Health Information Research Unit, Department of Health Research Methods, Evidence, and Impact, Faculty of Health Sciences, McMaster University, Hamilton, Ontario, Canada, lokkerc@mcmaster.ca; <sup>2</sup>Department of Computing, Faculty of Computing, Engineering and the Built Environment, Birmingham City University, Birmingham, UK; <sup>3</sup>Department of Oncology, Faculty of Health Sciences, McMaster University, Hamilton, Ontario, Canada; <sup>4</sup>Department of Medicine, Faculty of Health Sciences, McMaster University, Hamilton, Ontario, Canada.</p>
<h4>Conflict of Interest Disclosures </h4>
<p>McMaster University, a nonprofit public academic institution, has contracts through the Health Information Research Unit under the supervision of Alfonso Iorio and R. Brian Haynes. These contracts involve professional and commercial publishers to provide newly published studies, which are critically appraised for research methodology and assessed for clinical relevance as part of the McMaster Premium Literature Service (McMaster PLUS). Cynthia Lokker and Rick Parrish receive partial compensation, and R. Brian Haynes is remunerated for supervisory responsibilities and royalties. Ashirbani Saha, Fangwen Zhou, Muhammad Afzal, and Wael Abdelkader are not affiliated with McMaster PLUS.</p>
<h4>Funding/Support</h4>
<p> Fangwen Zhou was funded through the Mitacs Business Strategy Internship grant (IT42947) with matching funds from EBSCO Canada.</p>
<h4>Role of the Funder/Sponsor</h4>
<p> The funders were not involved in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the abstract; and decision to submit the abstract for presentation.</p>
<h4>Acknowledgment </h4>
<p>We thank the Digital Research Alliance of Canada for its computational resources.</p>


Domain-specific Pretrained Encoder Transformers for the Identification of Methodologically Rigorous Systematic Reviews: A Retrospective Modeling Study

<p>Beth Waymouth,<sup>1</sup> Heather Slater,<sup>1</sup> Angharad Goode,<sup>1</sup> Katie Allin,<sup>1</sup> Maria Kowalczuk<sup>1</sup></p>
<h4>Objective</h4>
<p> As publishers increasingly adopt artificial intelligence (AI)–driven tools, understanding editor engagement with AI-assisted editorial checks is crucial for maintaining publication standards. This study examines Handling Editors’ (HEs) interaction with an AI-powered check that flags manuscripts needing further verification during peer review and allows HEs and the editorial office (EOF) to collaborate on resolutions. After observing a decline in HE engagement, a survey was conducted to uncover the underlying reasons.</p>
<h4>Design</h4>
<p> We analyzed internal data tracking interaction with the AI-supported ethics check introduced to HEs in 2022. Existing editors were emailed about this new tool and given access and instructions, including a detailed Ethics Guidance document. Newly recruited HEs were introduced to the check during onboarding webinars. Data included the final status of the ethics check, collected quarterly between September 2022 and March 2024. Engagement was defined as the resolution of ethics checks by the HE rather than the EOF. To contextualize the engagement patterns, a survey was distributed to 4627 HEs across our journal portfolio who were active in 2024 and consented to be recontacted. Data were collected for 2 weeks in December 2024, with a reminder issued after the first week.</p>
<h4>Results</h4>
<p> Initial analysis showed that editor engagement dropped from 40% (2899 of 7394 total checks) to 20% (2170 of 10,692) between September 2022 and November 2023. Engagement rose to 25% (1227 of 4766 checks) between January and March 2024; however, this increase was not statistically significant. Of the surveyed editors, 725 (16%) responded to at least the first question. Of 662 respondents, 67% amended the ethics check on at least some manuscripts, with 24% interacting with the check for every manuscript. Among surveyed editors, 33% indicated that they were unaware of the check. In the “Other” category (n = 66), 38 HEs (58%) said no ethics issues required action or manuscripts didn’t need ethical approval; 4 cited unfamiliarity with the check, and 4 responded that they did not remember. Out of 478 responses from HEs not amending the check on some or all manuscripts, 67% cited an expectation that this task fell to the EOF. This was especially true when handling higher manuscript volumes (<b>Table 25-0897</b>).</p>

![](https://assets.underline.io/markdown_image/1/image/02ba50380f883f04074a683fa93e2342.png)

<h4>Conclusions</h4>
<p> These results underscore the risk of editor disengagement when AI-assisted tools lack sufficient onboarding support and adequate training. For AI integration to be effective, publishers must employ consistent monitoring of engagement and explore ways to ensure AI tools are visible, actionable, and reliably applied across editorial checkpoints.</p>
<sup>1</sup>Frontiers Media S.A., Switzerland, maria.kowalczuk@frontiersin.org.</p>
<h4>Conflict of Interest Disclosures</h4>
<p> Heather Slater, Angharad Goode, Katie Allin, and Maria Kowalczuk are employees of Frontiers Media S.A. Beth Waymouth was an employee of Frontiers Media S.A. at the time when the study was conducted and when the abstract was written and originally submitted. Since May 2025, Beth Waymouth has been an independent researcher.</p>
<h4>Acknowledgment</h4>
<p> We would like to thank all our Handling Editors who participated in the survey. We also wish to thank our colleagues Elena Vicario and Simone Ragavooloo for their valuable suggestions and insightful comments on this project and the abstract.</p>


Strategic Insights Into Editor Engagement With AI-Assisted Tools based on Survey and Data Analysis of AI-Assisted Ethics Checks

<p>Jeremy Y. Ng,<sup>1,2,3</sup> Malvika Krishnamurthy,<sup>2,3</sup> Gursimran Deol,<sup>2,3</sup> Wid Al-Zahraa Al-Khafaji,<sup>2,3</sup> Vetrivel Balaji,<sup>4</sup> Magdalene Abebe,<sup>2,3</sup> Jyot Adhvaryu,<sup>2,3</sup> Tejas Karrthik,<sup>2,3</sup> Pranavee Mohanakanthan,<sup>2,3</sup> Adharva Vellaparambil,<sup>2,3</sup> Lex M. Bouter,<sup>5,6</sup> R. Brian Haynes,<sup>7</sup> Alfonso Iorio,<sup>7,8</sup> Cynthia Lokker,<sup>7</sup> Hervé Maisonneuve,<sup>9,10</sup> Ana Marušić,<sup>11</sup> David Moher<sup>1,12</sup></p>
<h4>Objective</h4>
<p> This study aimed to examine the attitudes and perceptions of editors in chief (EICs) of biomedical journals regarding the integration of artificial intelligence chatbots (AICs) into the scholarly publishing process. Prior research has explored AI use in publishing broadly, but limited data exist on EIC perspectives. Although AICs offer opportunities to streamline editorial tasks, such as plagiarism detection and language editing, they also introduce ethical, technical, and operational challenges. Understanding EIC perspectives is critical to shaping guidelines, policies, and training that align with the evolving role of AICs in scholarly publishing.</p>
<h4>Design</h4>
<p> We conducted a cross-sectional survey of EICs of all biomedical journals, inclusive of medical, nursing, health sciences, dentistry, nursing, public health, and pharmacology and toxicology disciplines, published by Springer Nature (including BMC), Taylor &amp; Francis, Elsevier, Wiley, and Sage. Eligible journals were identified through a combination of automated web scraping and manual verification. EICs were invited to participate in an anonymous SurveyMonkey survey conducted over 5 weeks, which included 3 follow-up reminders, from July through August 2024. The survey covered familiarity with AICs, current use, perceived benefits and challenges, and anticipated future roles. Quantitative data were analyzed using descriptive statistics, while qualitative responses were coded and thematically analyzed to identify key themes. Our protocol was registered<sup>1</sup> and followed the CHERRIES reporting guideline.<sup>2</sup></p>
<h4>Results</h4>
<p> Of 3381 EICs contacted, 510 responded (15.1% response rate), with 505 eligible participants and a completion rate of 87.0%. Most were familiar with AICs (66.7% [325 of 487]) but had not used them in editorial workflows (83.7% [401 of 479]). Perceived benefits included enhanced language and grammar support (70.8% [308 of 435]) and plagiarism screening (67.3% [294 of 437]). However, respondents expressed concerns about initial setup and training (83.9% [360 of 429]), ethical risks (80.6% [345 of 428]), and technical reliability (75.2% [322 of 428]). While only 49.6% (240 of 484) of participants reported that their journal had formal AIC policies, 89.5% (419 of 468) of respondents supported training initiatives to promote ethical and effective use. Despite limited current adoption, 78.9% (370 of 469) believed AICs will play an important role in the future of scholarly publishing, and 77.2% (363 of 470) anticipated their significance in advancing scientific research. Themes identified through thematic analysis of open-ended questions included “no AI in authorship or peer review,” referring to EICs’ reporting of current journal and publisher policy on their use, and “ethical, integrity, and privacy concerns,” referring to EIC perceptions of challenges with the use of AICs in the scholarly publishing process. Our study manuscript has been preprinted.<sup>3</sup></p>
<h4>Conclusions</h4>
<p> Biomedical journal EICs recognized the potential of AICs to enhance editorial processes but highlighted critical barriers, including ethical dilemmas, resource limitations, and insufficient policies and training. Structured interventions, including targeted training programs and robust ethical guidelines, are essential for addressing these challenges and ensuring responsible and effective integration of AICs into publishing workflows.</p>
<h4>References</h4>
<p>1. Ng JY, Krishnamurthy M, Balaji V, et al. Attitudes and perceptions of biomedical journal editors-in-chief towards the use of artificial intelligence chatbots in the scholarly publishing process: a cross-sectional survey across multiple publishers. OSF Registries. August 10, 2024. Accessed July 2, 2025. <a href="https://osf.io/xt6f2"><span class="Hyperlink CharOverride-6">https://osf.io/xt6f2</span></a></p>
<p>2. Eysenbach G. Improving the quality of web surveys: the Checklist for Reporting Results of Internet E-Surveys (CHERRIES). <i>J Med Internet Res.</i> 2004;6(3):e34. doi:10.2196/jmir.6.3.e34</p>
<p>3. Ng JY, Krishnamurthy M, Deol G, et al. Attitudes and perceptions of biomedical journal editors-in-chief towards the use of artificial intelligence chatbots in the scholarly publishing process: a cross-sectional survey. <i>medRxiv</i>. Preprint posted online May 27, 2025. doi:10.1101/2025.05.26.25328101</p>
<sup>1</sup>Centre for Journalology, Ottawa Methods Centre, Ottawa Hospital Research Institute, Ottawa, Ontario, Canada, jeremyyng.phd@gmail.com; <sup>2</sup>Institute of General Practice and Interprofessional Care, University Hospital Tübingen, Tübingen, Germany; <sup>3</sup>Robert Bosch Center for Integrative Medicine and Health, Bosch Health Campus, Stuttgart, Germany; <sup>4</sup>Department of Computing and Software, Faculty of Engineering, McMaster University, Hamilton, Ontario, Canada;<sup> 5</sup>Department of Epidemiology and Data Sciences, Amsterdam Universities Medical Center, Amsterdam, the Netherlands, <sup>6</sup>Department of Philosophy, Faculty of Humanities, Vrije Universiteit, Amsterdam, the Netherlands; <sup>7</sup>Department of Health Research Methods, Evidence, and Impact, Faculty of Health Sciences, McMaster University, Hamilton, Ontario, Canada; <sup>8</sup>Department of Medicine, McMaster University, Hamilton, Ontario, Canada; <sup>9</sup>Consultant, Lyon, France; <sup>10</sup>Scientific Committee, Institute of Research and Action on Fraud and Plagiarism in Academia (IRAFPA), Geneva, Switzerland; <sup>11</sup>Department of Research in Biomedicine and Health and Center for Evidence-based Medicine, University of Split School of Medicine, Split, Croatia; <sup>12</sup>School of Epidemiology, Public Health and Preventive Medicine, Faculty of Medicine, University of Ottawa, Ottawa, Ontario, Canada.</p>
<h4>Conflict of Interest Disclosures</h4>
<p> Lex M. Bouter, Ana Marušić, and David Moher are members of the Peer Review Congress Advisory Board but were not involved in the review or decision for this abstract. No other disclosures were reported.</p>


Attitudes and Perceptions of Biomedical Journal Editors in Chief Towards the Use of Artificial Intelligence Chatbots in the Scholarly Publishing Process

<p>Jeremy Y. Ng,<sup>1,2,3</sup> Daivat Bhavsar,<sup>1,2</sup> Neha Dhanvanthry,<sup>1,2</sup> Lex Bouter,<sup>4,5</sup> Teresa M. Chan,<sup>6</sup> Holger Cramer,<sup>1,2</sup> Annette Flanagin,<sup>7</sup> Alfonso Iorio,<sup>8,9</sup> Cynthia Lokker,<sup>8</sup> Hervé Maisonneuve,<sup>10,11</sup> Ana Marušić,<sup>12</sup> David Moher<sup>3,13</sup></p>
<h4>Objective </h4>
<p>Peer review is a cornerstone of scientific publishing, ensuring the rigor and credibility of research. However, it is increasingly strained by underexplored challenges, such as variability in quality, reviewer fatigue, and biases. Artificial intelligence chatbots (AICs) are emerging tools with the potential to address these challenges by automating tasks, such as identifying methodological flaws and improving language clarity. This study aimed to explore peer reviewers’ attitudes and perceptions toward the use of AICs in medical journal peer review, focusing on their benefits, challenges, and ethical implications.</p>
<h4>Design </h4>
<p>A large-scale, international, cross-sectional survey targeting peer reviewers of medical journals was conducted. Eligible participants had completed at least 1 peer review report for a MEDLINE-indexed journal within the last 24 months at the point of invitation. The names and email addresses of a complete sample of 72,847 corresponding authors who published in MEDLINE-indexed journals from September 1, 2024, to October 15, 2024, were collected. The survey was administered and data were collected via the SurveyMonkey survey platform. The survey included questions about participant demographics, familiarity with AICs, experiences with their use, and perceptions of AIC’s roles, benefits, challenges, and ethical implications in peer review. The survey included multiple-choice, yes/no, Likert scale, and open-ended questions to ensure comprehensive data collection. The survey was first piloted with a group of 7 peer reviewers to optimize question clarity and relevance. The survey was launched in April 2025 and remained open for 7 weeks in total. This study’s protocol was registered<sup>1</sup> and the CHERRIES<sup>2</sup> and STROBE<sup>3</sup> reporting guidelines were used to inform the reporting of this survey study.</p>
<h4>Results </h4>
<p>The survey received 1194 respondents from 33,388 opened email invitations in the first 4 weeks, 1018 of whom completed the survey (response rate: 3.0%). The majority (578/1018 [56.8%]) were senior-career researchers (ie, &gt;10 years of career experience). Most reported never having used an AIC for peer review purposes (707/999 [70.8%]), and notably, many (460/1007 [45.7%]) did not anticipate using AICs in the future. However, most respondents indicated interest in receiving training on AIC use for peer review tasks (587/980 [59.9%]). The most-recognized benefit of AIC use in peer review processes (ie, ”agree” or “strongly agree”) was “reduc[ing] the workload” (601/958 [62.7%]). Conversely, the most-recognized challenge was “risk producing errors or inaccuracies” (738/934 [79.0%]).</p>
<h4>Conclusions </h4>
<p>This study provides critical insights into peer reviewers’ attitudes and perceptions toward the use of AICs in the peer review process. Findings may inform the development of evidence-based guidelines and policies to ensure the ethical, transparent, and effective use of AICs, contributing to improvements in efficiency, quality, and equity in scholarly publishing.</p>
<h4>References</h4>
<p>1. Ng JY, Bhavsar D, Dhanvanthry N, et al. Attitudes and perceptions toward the use of artificial intelligence chatbots in medical journal peer review: a large-scale, international cross-sectional survey. OSF. May 21, 2025. <a href="https://osf.io/fhc2m"><span class="Hyperlink CharOverride-6">https://osf.io/fhc2m</span></a></p>
<p>2. Eysenbach G. Improving the quality of Web surveys: the Checklist for Reporting Results of Internet E-Surveys (CHERRIES). <i>J Med Internet Res</i>. 2004;6(3):e34. doi:10.2196/jmir.6.3.e34</p>
<p>3. von Elm E, Altman DG, Egger M, Pocock SJ, Gøtzsche PC, Vandenbroucke JP; STROBE Initiative. The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies. <i>The Lancet</i>. 2007;370(9596):1453-1457. doi:10.1016/S0140-6736(07)61602-X</p>
<sup>1</sup>Institute of General Practice and Interprofessional Care, University Hospital Tübingen, Tübingen, Germany, jeremyyng.phd@gmail.com; <sup>2</sup>Robert Bosch Center for Integrative Medicine and Health, Bosch Health Campus, Stuttgart, Germany; <sup>3</sup>Centre for Journalology, Ottawa Methods Centre, Ottawa Hospital Research Institute, Ottawa, Ontario, Canada; <sup>4</sup>Department of Epidemiology and Data Science, Amsterdam University Medical Centers, Amsterdam, the Netherlands; <sup>5</sup>Department of Philosophy, Vrije Universiteit Amsterdam, Amsterdam, the Netherlands; <sup>6</sup>School of Medicine, Toronto Metropolitan University, Toronto, Ontario, Canada; <sup>7</sup><i>JAMA</i> and the JAMA Network, Chicago, IL, US; <sup>8</sup>Department of Health Research Methods, Evidence, and Impact, Faculty of Health Sciences, McMaster University, Hamilton, Ontario, Canada; <sup>9</sup>Department of Medicine, McMaster University, Hamilton, Ontario, Canada; <sup>10</sup>Consultant, Lyon, France; <sup>11</sup>Scientific Committee, Institute of Research and Action on Fraud and Plagiarism in Academia (IRAFPA), Geneva, Switzerland; <sup>12</sup>Department of Research in Biomedicine and Health and Center for Evidence-based Medicine, University of Split School of Medicine, Split, Croatia; <sup>13</sup>School of Epidemiology, Public Health and Preventive Medicine, Faculty of Medicine, University of Ottawa, Ottawa, Canada.</p>
<h4>Conflict of Interest Disclosures</h4>
<p> Lex Bouter, Annette Flanagin, Ana Marušić, and David Moher are members of the Peer Review Congress Advisory Board but were not involved in the review or decision for this abstract.</p>


Attitudes and Perceptions Toward the Use of Artificial Intelligence Chatbots in Medical Journal Peer Review: A Large-Scale, International Cross-Sectional Survey

<p>Yang Zhang,<sup>1,2</sup> Mengyuan Duan,<sup>1,2</sup> Guoguang Zhao,<sup>3</sup> Penghu Wei,<sup>2,3</sup> Xiuyuan Hao<sup>4</sup></p>
<h4>Objective </h4>
<p>Given the increasing role of artificial intelligence (AI) in academic writing, it is important to understand how these tools are integrated into research practices. This study aimed to explore the use of AI in academic writing by Chinese medical scholars and to identify potential factors influencing their adoption and perceptions of AI. </p>
<h4>Design </h4>
<p>A total of 378 peer reviewers of <i>Chinese Medical Journal</i> were invited to participate in a survey using an electronic questionnaire via email between January 2, 2025, and January 12, 2025. The questionnaire included questions about AI tool use in research paper writing, reasons for nonuse, stages of use, experience of declaration, and perception of effects of AI use. The survey also explored what role AI should have and where researchers believe AI should be acknowledged in publications. The Fisher exact test was used to identify significant differences between subgroups based on age and use.</p>
<h4>Results </h4>
<p>Of the 378 reviewers invited, 159 (42.1%) provided valid responses. Among the respondents, 93 (58.5%) reported using AI tools in their research paper writing, and 66 (41.5%) did not (<b>Table 25-0878</b>). There was a significant difference in declared use of AI across age groups, with younger researchers more likely to use AI tools (76.1% of those ≤30 years, 59.1% of those 31-45 years, 27.8% of those 46-60 years, and 14.3% of those &gt;60 years; <i>P</i> &lt; .001). The primary reasons for not using AI tools included copyright or ethical issues (53.0%), concerns about inaccuracy (47.0%), and unfamiliarity with AI tools (40.9%). Notably, there were still 27.3% non-AI users considering using AI when technology matures. The most common stages at which AI tools were used was after writing for language translation and polishing (90.3%), followed by idea generation before writing (16.1%) and structural organization during writing (15.1%). However, only 29.0% of participants declared AI use in past submissions. A total of 60.2% found the tools to be very helpful, while none considered them to have negative impact. The preferred roles and acknowledgement of AI tools in papers included a separate AI writing declaration section (61.6%), not acknowledging AI at all (17.0%), reporting in the methods section (12.6%), reporting in the conflict of interest section (5.0%), and including AI as a co-author (3.8%), which did not significantly differ between AI users and nonusers; nor did these views differ significantly by age.</p>

![](https://assets.underline.io/markdown_image/1/image/5a3f77baa1c566657fd81faf9e7dac2a.png)

<h4>Conclusions </h4>
<p>In this study, nearly 60% of Chinese medical scholars reported use of AI tools in writing and preparing papers, particularly for language translation and polishing. However, many reported concerns about the accuracy, ethics, and maturity of AI technology. Current AI guidelines in academic publishing lack a unified standard, particularly in usage scope and enforcement. Some journals lack mandatory disclosure policies for AI use, standardized templates for reporting, or requirements to submit archival copies of original data or content before AI use, which might account for incomplete reporting or failure to report use of AI. Further research is needed to assess the long-term impact of AI on academic writing.</p>
<p><sup>1</sup>National Center for Neurological Disorders, Xuanwu Hospital Capital Medical University, Beijing, China; <sup>2</sup>Editorial Office, Brain Network Disorders, Xuanwu Hospital Capital Medical University, Beijing, China; <sup>3</sup>Department of Neurosurgery, Xuanwu Hospital Capital Medical University, Beijing, China; <sup>4</sup>Editorial Office, Chinese Medical Journal, Chinese Medical Association Publishing House, Beijing, China, haoxiuyuan@cma.org.cn. 
<h4>Conflict of Interest Disclosures</h4> None reported. 
<h4>AdditionaI Information</h4> ChatGPT (version GPT-4o mini; OpenAI) was used on January 20, 2025, for language translation, grammar checking, and manuscript polishing to ensure the clarity and coherence of this abstract.


Artificial Intelligence Use and Acknowledgment in Medical Research Writing Among Chinese Scholars

<p>Charvi Rastogi,<sup>1</sup> Xiangchen Song,<sup>2</sup> Zhijing Jin,<sup>3,4</sup> Ivan Stelmakh,<sup>5</sup> Hal Daumé III,<sup>6</sup> Kun Zhang,<sup>2</sup> Nihar B. Shah<sup>2</sup></p>
<h4>Objective </h4>
<p>Many peer-review processes in computer science involve reviewers submitting independent reviews followed by a discussion between reviewers of each article on a typed forum (online discussion board). A key policy question is whether reviewers should remain anonymous to each other. This study investigated 7 research questions (RQs): RQ 1. Do reviewers discuss more when anonymous to each other or not? RQ 2. Are decisions closer to senior or junior reviewers’ opinions across conditions? RQ 3. Are reviewers more polite when not anonymous? RQ 4. Do self-reported reviewer experiences differ? RQ 5. Do reviewers prefer one condition? RQ 6. What factors do reviewers consider important in this policy decision? RQ 7. Have reviewers experienced dishonest behavior when their identity is revealed to other reviewers?</p>
<h4>Design </h4>
<p>A randomized controlled trial was conducted in the Conference on Uncertainty in Artificial Intelligence (UAI) in 2022, where full articles (not abstracts) were reviewed. Reviewers and articles were randomly assigned to either a condition where reviewer identities were hidden from each other or one where they were visible. Reviewers were then matched to articles within each condition using a semiautomated procedure.<sup>1</sup> An anonymous survey of reviewers was also administered. The following measurements were made: RQ 1. Average posts per reviewer-article pairs were compared; test statistic: difference across conditions. RQ 2. Test statistic: difference in the fraction of articles where the reviewer closest to the final decision was senior. RQ 3. Politeness was scored 1 to 5 using a locally deployed large language model<sup>2</sup> with few-shot prompting; scores were averaged across iterations and paraphrased prompts; test statistic: normalized Mann-Whitney U test. RQ 4. Reviewers rated 5 aspects of their experience on a 5-point Likert scale; differences across conditions were tested using a normalized Mann-Whitney U test. RQ 5. Reviewers rated overall preference on a 5-point scale mapped from −2 to 2; test statistic: Cohen <i>d</i>. RQ 6. Reviewers rated the importance of 6 factors in deciding on anonymity policy, each from 1 (least important) to 6. RQ 7. Reviewers reported experience of any dishonest behavior due to reviewer identities being visible to other reviewers, with checkboxes “Yes, in UAI 2022,” “Yes, in another venue,” “Not sure,” and “No.” The test statistics also served as a measure of the effect sizes. <i>P</i> values were computed via permutation testing.</p>
<h4>Results </h4>
<p>Overall, 322 papers were reviewed under the anonymous condition (116 accepted) and 310 papers under nonanonymous (114 accepted), with exactly 289 reviewers in both conditions. There were 611 discussion posts made by reviewers in the anonymous condition and 514 in the nonanonymous condition. The results for the 7 research questions are provided in <b>Table 24-0812</b>.</p>

![](https://assets.underline.io/markdown_image/1/image/bd5b7dcf2758e3af6ff9169164e2cab0.png)

<h4>Conclusions </h4>
<p>Small but significant differences favoring anonymous discussions were found. Subsequent computer science conferences have drawn on these findings for their policy choices, with a greater inclination toward anonymity in reviewer discussions.</p>
<h4>References</h4>
<p>1. Shah N. An overview of challenges, experiments, and computational solutions in peer review (extended version). July 7, 2025. Accessed July 16, 2025. <a href="https://www.cs.cmu.edu/~nihars/preprints/SurveyPeerReview.pdf"><span class="CharOverride-10">https://www.cs.cmu.edu/~nihars/preprints/SurveyPeerReview.pdf</span></a></p>
<p>2. Chiang W, Li Z, Lin Z, et al. Vicuna: an open-source chatbot impressing GPT-4 with 90%* ChatGPT quality. LMSYSORG. March 2023. <a href="https://lmsys.org/blog/2023-03-30-vicuna/"><span class="CharOverride-10">https://lmsys.org/blog/2023-03-30-vicuna/</span></a></p>
<sup>1</sup>Google DeepMind, New York, NY, US; <sup>2</sup>Carnegie Mellon University, Pittsburgh, PA, US, nihars@cs.cmu.edu; <sup>3</sup>ETH Zurich, Zurich, Switzerland; <sup>4</sup>Max Planck Institute, Tübingen, Germany; <sup>5</sup>New Economic School, Moscow, Russia; <sup>6</sup>University of Maryland, Baltimore, MD, US. <h4>Conflict of Interest Disclosures</h4> As per author affiliations above. In addition, Zhijing Jin is going to join the University of Toronto, and Kun Zhang has a partial appointment at Mohamed bin Zayed University of Artificial Intelligence. <h4>Funding/Support</h4> ONR N000142212181, NSF 1942124, 2200410, 2229881, NIH R01HL159805. <h4>Role of Funder/Sponsor</h4> The funders played no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the abstract; and decision to submit the abstract for presentation. <h4>Acknowledgment</h4> This work was conducted when Charvi Rastogi and Ivan St

Anonymizing Reviewers to Each Other in Peer Review Discussions: A Randomized Controlled Trial

<p>John Carpenter,<sup>1</sup> Andrea Corvillón<sup>1</sup></p>
<h4>Objective </h4>
<p>In 2021,<sup>1</sup> the Atacama Large Millimeter/submillimeter Array (ALMA) transitioned from single-anonymous panel reviews to dual-anonymous distributed peer review to manage the growing volume of proposal submissions and mitigate potential biases.<sup>2</sup> We conducted a retrospective cohort study to examine associations between this procedural change and proposal rankings across principal investigator (PI) demographic characteristics, using 7 years of data under the previous format (2012-2018) and 4 years under the new process (2021-2024).</p>
<h4>Design </h4>
<p>We analyzed proposal rankings from 12 ALMA cycles. From 2011 to 2018 (cycles 0-6), proposals were reviewed in topical panels under single-anonymous peer review. In 2019 (cycle 7), investigator lists were randomized while panels were retained. No review process was held in 2020 due to the COVID-19 pandemic. In 2021 (cycle 8), ALMA implemented dual-anonymous, distributed peer review for most proposals. We examined rankings by 3 PI demographic characteristics: (1) experience (number of cycles in which the PI submitted proposals), (2) regional affiliation (Chile, East Asia, Europe, North America, or other), and (3) sex. We grouped proposals by review era: single-anonymous panel review (cycles 1-6; 2012-2018; 9091 proposals) and dual-anonymous distributed peer review (cycles 8-11; 2021-2024; 6490 proposals). Cycle 0 and cycle 7 were excluded from the analysis: cycle 0 because all PIs were, by definition, first-time users of ALMA, and cycle 7 because it was a transitional year for implementing dual anonymity.</p>
<h4>Results </h4>
<p>Proposal rankings were normalized from 0 (best) to 1 (worst) for comparability across cycles. Proposal counts by demographic subgroup and review era are reported in <b>Table 25-1053</b>. We compared median normalized rankings between review eras using 10,000 bootstrap samples to generate 95% CIs and 2-sided <i>P</i> values. We observed the following associations: (1) for experience, PIs submitting in all cycles had better rankings during single-anonymous panel review than under dual-anonymous distributed review (<i>P</i> = .006), and first-time PIs ranked lowest in both systems with no significant change in rankings (<i>P</i> = .19); (2) for regional affiliation, East Asian PIs showed improved rankings after the transition (<i>P</i> &lt; .001), rankings for European PIs declined (<i>P</i> = .009) but remained above average, and rankings for PIs from North America, Chile, and other regions did not show significant changes (<i>P</i> &gt; .60); and (3) for sex, no statistically significant differences in rankings were observed between male- and female-led proposals in either review system (<i>P</i> = .12).</p>

![](https://assets.underline.io/markdown_image/1/image/ef41fffb9d8404953c23a5d7853259dc.png)

<h4>Conclusions </h4>
<p>Dual-anonymous, distributed peer review was associated with reduced disparities by PI experience and region, consistent with reduced prestige and geographic bias. While increased score variability may have contributed, the nonuniform changes (ie, some groups improved, others remained stable) are inconsistent with a purely noise-driven explanation. These findings suggest that systematic shifts in reviewer behavior, not merely increased randomness, underlie the observed trends. Disentangling the effects of dual anonymity, distributed review, and other concurrent changes (eg, increased use of artificial intelligence) remains an important direction for future research.</p>
<h4>References</h4>
<p>1. Donovan Meyer J, Corvillón A, Carpenter JM, et al. Analysis of the ALMA Cycle 8 distributed peer review process. <i>Bull Am Astron Soc</i>. 2022;54(1):43. doi:10.3847/25c2cfeb.4ece85d4</p>
<p>2. Carpenter JM, Corvillón A, Donovan Meyer J, et al. Update on the systematics in the ALMA proposal review process after Cycle 8. <i>Publ Astron Soc Pac</i>. 2022;134:045001. doi:10.1088/1538-3873/ac5b89</p>
<sup>1</sup>Joint ALMA Observatory, Santiago, Chile, john.carpenter@alma.cl. <h4>Conflict of Interest Disclosures</h4> John Carpenter and Andrea Corvillón are employed by the Joint ALMA Observatory, which is jointly managed by Associated Universities Inc/National Radio Astronomy Observatory, the European Organisation for Astronomical Research in the Southern Hemisphere, and the National Astronomical Observatory of Japan on behalf of the ALMA partnership. <h4>Funding/Support</h4> This work was supported by the Joint ALMA Observatory. <h4>Role of the Funder/Sponsor</h4> The funder had a role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; and decision to submit the abstract for presentation. <h4>Acknowledgment</h4> ALMA is a partnership of the European Organization for Astronomical Research in the Southern Hemisphere (representing its member states), National Science Foundation (US), and National Institutes of Natural Sciences (Japan), together with the National Research Council of Canada (Canada), the National Science and Technology Council and Academia Sinica Institute of Astronomy and Astrophysics (Taiwan), and the Korea Astronomy and Space Science Institute (Republic of Korea), in cooperation with the Republic of Chile.



Dual Anonymous and Distributed Peer Review for Proposal Review Rankings at the ALMA Observatory

<p>Christos P. Kotanidis,<sup>1,2,3</sup> Sarah Gorey,<sup>1,4</sup> Harleen Marwah,<sup>1</sup> Abarna Pearl,<sup>1</sup> Darren Taichman,<sup>1</sup> Mary Beth Hamel<sup>1,5</sup></p>
<h4>Objective</h4>
<p> Editorial review forms the cornerstone of scientific publishing, ensuring that published research adheres to the highest quality standards.<sup>1</sup> We evaluated how editorial processes shape manuscripts by examining changes in abstracts between submission and publication.</p>
<h4>Design</h4>
<p> We assessed research articles submitted to the <i>New England Journal of Medicine</i> (<i>NEJM</i>) in 2022 and published in <i>NEJM</i> or rejected and subsequently published in 437 other journals. Using author names, we searched PubMed for articles within 2 years of submission. Four physicians reviewed search results to match published manuscripts with submitted manuscripts. Abstract similarity was quantified using term frequency–inverse document frequency,<sup>2,3</sup> assigning a cosine similarity score ranging from 0 (dissimilar) to 1 (identical). To assess if similarity changes reflected substantive differences, masked physician reviewers conducted qualitative comparisons of submitted and published versions of abstracts from clinical trials. Abstracts were assessed for substantive changes in the following 4 domains: trial design, primary outcome, adverse events, and conclusion (TPAC). A score of +1 was assigned for a substantive domain change favoring the published version and −1 for domain change favoring the submitted version, with the domain scores summed, resulting in a total TPAC score range of −4 (submitted better) to +4 (published better).</p>
<h4>Results</h4>
<p> We matched 1360 PubMed records from 2756 research articles initially submitted to <i>NEJM</i>. Of these, 201 were published in <i>NEJM</i> and 1159 were rejected by <i>NEJM</i> and published in the 437 other journals. The median (IQR) similarity score between submitted and published abstracts was 0.71 (0.58-0.83). Abstracts of articles published in the 5 general medicine journals with a journal impact factor (JIF) above 50 per the 2023 Clarivate report had significantly lower similarity scores compared with those published in other journals (<span class="CharOverride-4"><b>Figure 25-0940</b></span>, A). For qualitative comparisons, we identified abstracts from randomized controlled trials. On average, published abstracts improved in 0.9 domains, with conclusion being the most frequently modified domain (44.2%). Similarity score correlated negatively with TPAC (Spearman ρ coefficient, −0.39; <span class="CharOverride-4"><b>Figure 25-0940</b></span>, B), indicating that lower similarity reflected more substantive revisions. Among abstracts published in the 5 high-JIF general medicine journals, 72.1% improved in at least 1 domain, compared with 48.3% of abstracts published in the other 433 journals. This difference was consistent across all TPAC domains, with conclusion changes being the most different (48% vs 27%).</p>


![](https://assets.underline.io/markdown_image/1/image/bcf1d262f8a68ab13bfbe8e357b30334.png)

<h4>Conclusions</h4>
<p> Our comparisons of submitted and published abstracts suggest that peer review and editorial processes result in substantial revision of research reports. This report focused on submissions to 1 general medical journal and within a 2-year publication window. These findings reinforce the importance of editorial oversight in scientific publishing, highlighting its role in refining study design communication, reporting of outcomes, and overall clarity in research dissemination.</p>
<h4>References</h4>
<p>1. Roll SC. The value and process of high-quality peer review in scientific professional journals. <i>J Diagn Med Sonography</i>. 2019;35(5):359-362. doi:10.1177/8756479319853800</p>
<p>2. Aizawa A. An information-theoretic perspective of tf–idf measures. <i>Inf Process &amp; Manage</i>. 2003;39(1):45-65. doi:10.1016/S0306-4573(02)00021-3</p>
<p>3. Ramos J. Using tf-idf to determine word relevance in document queries. In: <i>Proceedings of the First Instructional Conference on Machine Learning</i>. 2003.</p>
<sup>1</sup>New England Journal of Medicine, Boston, MA, US, ckotanidis@ nejm.org; <sup>2</sup>Heart and Vascular Center, Division of Cardiovascular Medicine, Brigham and Women’s Hospital, Harvard Medical School, Boston, MA, US; <sup>3</sup>Division of Cardiovascular Medicine, Radcliffe Department of Medicine, University of Oxford, Oxford, UK; <sup>4</sup>Stroke Clinical Trials Network Ireland (SCTNI), University College Dublin Clinical Research Centre, Mater Hospital, Dublin, Ireland; <sup>5</sup>Division of General Medicine and Primary Care, Department of Medicine, Harvard Medical School, Beth Israel Deaconess Medical Center, Boston, MA, US. <h4>Conflict of Interest Disclosures</h4> Christos P. Kotanidis, Sarah Gorey, Harleen Marwah, and Abarna Pearl are Editorial Fellows, Darren Taichman is a Deputy Editor, and Mary Beth Hamel is the Executive Editor at the New England Journal of Medicine.



Changes to Research Article Abstracts Between Submission and Publication

<p>Nicholas LaBerge,<sup>1</sup> Sam Zhang,<sup>1,2,3</sup> Quinten McElhiney,<sup>1</sup> Daniel B. Larremore,<sup>1,2</sup> Aaron Clauset<sup>1,2</sup></p>
<h4>Objective</h4>
<p> To make publicly available a deidentified dataset of manuscript submissions and associated editorial metadata at <i>Science</i> and <i>Science Advances</i> (2 elite multidisciplinary journals) and quantify the manuscript and author characteristics associated with outcomes over the 2 stages of evaluation at these highly selective journals: (1) editorial review, when journal editors screen and select a smaller set of submissions for detailed consideration, and (2) peer review, when editors recruit outside experts to evaluate manuscripts. Peer review at elite scientific journals is a high-stakes process whose outcomes have a broad influence in science and society. However, the need to maintain peer review’s confidentiality has limited the range of data available for scientific study. This lack of peer review data makes it difficult to assess how well elite journals achieve the scientific community’s ideals or to improve and test theories to improve their evaluation processes.</p>
<h4>Design</h4>
<p> We introduced and described a manuscript-level dataset of 110,303 deidentified evaluations of manuscripts submitted to <i>Science</i> and <i>Science Advances</i> over a 5-year period (2015-2019), each with a standard set of author and manuscript characteristics, and we conducted a series of logistic regression analyses to quantify the correlates of success in the initial editorial review stage (desk rejections) and the subsequent peer review stage at both journals.</p>
<h4>Results</h4></span> Each manuscript record includes author, editor, and manuscript characteristics, including topic, team size, institutional prestige, gender, geographic region, evaluation scores, and editor decisions; personally identifiable and reidentifiable information, including the text of the reviews, is excluded. Our statistical analyses revealed strong associations with institutional prestige, team size, manuscript topic, and country, which are primarily attributable (via a mediation analysis) to the influence of the editor, even as the tenor of advice from outside experts correlates strongly with the final editorial decision (<b>Figure 25-0972</b>). Corresponding authors who are men appear to have a small but significant advantage with editors and reviewers at <i>Science</i>, while authors based in China have a significant disadvantage.</p>

![](https://assets.underline.io/markdown_image/1/image/e1bd441e68d25c2bc4c3481b9d8f95e7.png)

<p class="abstract_abstract-text ParaOverride-5"><span class="CharOverride-4"><h4>Conclusions</h4></span> This deidentified dataset will support further investigation by peer review researchers into the 2-stage evaluation process of these 2 elite journals, and our results will help quantify the complex interactions among editors, reviewers, manuscript characteristics, and author characteristics at elite journals.</p>
<sup>1</sup>University of Colorado Boulder, Boulder, CO, US, aaron.clauset@ colorado.edu; <sup>2</sup>Santa Fe Institute, Santa Fe, NM, US; <sup>3</sup>University of Vermont, Burlington, VT, US. <h4>Conflict of Interest Disclosures</h4> Aaron Clauset is a Deputy Editor at Science Advances; all other authors declare no conflicts of interest. <h4>Funding/Support</h4> This work was supported by Air Force Office of Scientific Research Award FA9550-19-1-0329 (to Nicholas LaBerge, Sam Zhang, Daniel B. Larremore, and Aaron Clauset), National Science Foundation (NSF) SBE Award 2219609 (to Nicholas LaBerge, Sam Zhang, Daniel B. Larremore, and Aaron Clauset), NSF Graduate Research Fellowship Award DGE 2040434 (to Sam Zhang), and NSF Alan T. Waterman Award SMA-2226343 (to Daniel B. Larremore). <h4>Role of Funder/Sponsor</h4> The funders had no role in this research.



Manuscript Characteristics Associated With Editorial Review and Peer Review Outcomes at Science and Science Advances

<p>Nicola Adamson,<sup>1</sup> Andy Collings<sup>1</sup></p>
<h4>Objective </h4>
<p>In October 2022, <i>eLife</i> announced a Publish, Review, Curate model<sup>1</sup> in which all peer-reviewed submissions are published as reviewed preprints, accompanied by an <i>eLife</i> Assessment<sup>2</sup> and public reviews. Authors decide which revisions to undertake and when to declare a version of record (VOR), without an accept/reject decision after peer review. We investigated the extent to which authors revise and improve their work.</p>
<h4>Design </h4>
<p>An <i>eLife</i> Assessment is prepared for each submission sent for review and subsequent revisions, which summarizes the strength of evidence and significance of findings using terms from a common vocabulary. We extracted terms used in <i>eLife</i> Assessments for all 2918 reviewed preprints and their subsequent versions to the end of 2024. To ensure publications had completed the entire process and we could evaluate the full extent of changes, we selected the 1504 where a VOR had been declared. From these, 213 were excluded because either the VOR or original reviewed preprint had included multiple terms to describe the evidence. Using a retrospective analysis, we studied the remaining 1291 reviewed preprints and their respective VOR, calculating the distributions of and changes to terms between first and final versions and the number of rounds of revision.</p>
<h4>Results </h4>
<p>Panel A in <b>Figure 25-0989</b> shows the terms used to describe the strength of evidence in VORs and the first version of the reviewed preprints. The evidence term improved between versions in 39.4% of cases (n = 509) and remained the same in 57.7% of VORs (n = 745). Of VORs in which the evidence was described as incomplete in the first reviewed preprint, 76.5% (n = 199) improved to solid or better, and in papers originally described as solid, 49.3% improved to convincing or better (n = 217). In papers in which the evidence was originally described as inadequate (n = 23), this improved in 78.3% of instances (n = 18) and approximately one-half ended as solid or better (47.8%, n = 11). The majority of VORs (82.6%, n = 1066) were declared after 1 round of revision and 13.5% (n = 174) after 2 or more rounds. Of the 4.0% of VORs declared without revisions (n = 51), only 13.7% had an evidence term of inadequate (n = 0) or incomplete (n = 7). The significance of the findings (<b>Figure 25-0989</b>, B) was most frequently described using valuable (VOR: 31.1%, n = 401) or important (VOR: 49.0%, n = 633); overall, significance terms remained the same in 78.2% of cases (n = 1010).</p>

![](https://assets.underline.io/markdown_image/1/image/b3f8c42937978d78fd6b86c9caa9c9d2.png)

<h4>Conclusions </h4>
<p>Authors generally complete at least 1 round of revision, even though they have the option to proceed without, and multiple rounds of revision are uncommon. Where the evidence was originally described as incomplete, the majority of authors revised to improve this before the VOR. Terms used to describe the significance of the findings change less often than evidence terms, perhaps due to significance being more closely linked to the original research question under investigation.</p>
<h4>References</h4>
<p>1. Eisen MB, Akhmanova A, Behrens TE, et al. Scientific publishing: peer review without gatekeeping. <i>eLife</i>. October 20, 2022;11:e83889. doi:10.7554/eLife.83889</p>
<p>2. <i>eLife</i>’s New Model: What is an <i>eLife</i> assessment? <i>eLife.</i> October 20, 2022. Accessed January 30, 2025. <a href="https://elifesciences.org/inside-elife/db24dd46/elife-s-new-model-what-is-an-elife-assessment">https://elifesciences.org/inside-elife/db24dd46/elife-s-new-model-what-is-an-elife-assessment</a></p>
<sup>1</sup>eLife Sciences Publications Ltd, 95 Regent Street, Cambridge, CB2 1AW, UK, n.adamson@elifesciences.org. <h4>Conflict of Interest Disclosures</h4> We have no competing interests to declare. Both authors are employed by eLife Sciences Publications Ltd; we received no external funding for this work. <h4>Acknowledgment</h4> We thank Fiona Hutton, Fred Atherden, Emily Packer, and George Currie for their comments and suggestions on the submission, and Fred Atherden for extracting data for analysis.



Downloads

Next from The 10th Peer Review Congress

Domain-specific Pretrained Encoder Transformers for the Identification of Methodologically Rigorous Systematic Reviews: A Retrospective Modeling Study

Similar lecture

Quality and Comprehensiveness of Peer Reviews of Journal Submissions Produced by Large Language Models vs Humans

.css-70qvj9{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-align-items:center;-webkit-box-align:center;-ms-flex-align:center;align-items:center;}Downloads

Next from The 10th Peer Review Congress

Domain-specific Pretrained Encoder Transformers for the Identification of Methodologically Rigorous Systematic Reviews: A Retrospective Modeling Study

Similar lecture

Quality and Comprehensiveness of Peer Reviews of Journal Submissions Produced by Large Language Models vs Humans

Downloads