United States

&lt;p&gt;Alexander Goldberg,&lt;sup&gt;1&lt;/sup&gt; Ihsan Ullah,&lt;sup&gt;2&lt;/sup&gt; Thanh Gia Hieu Khuong,&lt;sup&gt;3&lt;/sup&gt; Benedictus Kent Rachmat,&lt;sup&gt;3&lt;/sup&gt; Zhen (Zach) Xu,&lt;sup&gt;4&lt;/sup&gt; Isabelle Guyon,&lt;sup&gt;2,3,5&lt;/sup&gt; Nihar B. Shah&lt;sup&gt;1&lt;/sup&gt;&lt;/p&gt;
&lt;h4&gt;Objective &lt;/h4&gt;
&lt;p&gt;This study assessed the utility of a large language model (LLM) in evaluating compliance with submission standards at the 2024 Neural Information Processing Systems (NeurIPS) conference—a top-tier publication venue in artificial intelligence (AI) with a 15% to 25% acceptance rate that reviews full papers, not just abstracts. NeurIPS requires authors to complete a 15-question checklist promoting reproducibility, transparency, and ethical standards, similar to CONSORT&lt;sup&gt;1&lt;/sup&gt; and STROBE.&lt;sup&gt;2&lt;/sup&gt; We introduced an optional Checklist Assistant using GPT-4 (OpenAI) to provide presubmission feedback to authors on checklist accuracy.&lt;/p&gt;
&lt;h4&gt;Design &lt;/h4&gt;
&lt;p&gt;We conducted a cross-sectional study in which the Checklist Assistant was available to authors 8 days before the submission deadline. Authors could optionally use the Checklist Assistant, which used a general purpose third-party LLM (gpt-4-turbo-preview) to evaluate the accuracy of their responses to a 15-question checklist. For each question, the LLM received the author’s response, justification, and the full paper (including appendices), and assessed the accuracy of the response using simple prompt engineering. We conducted surveys about authors’ perceptions of the tool at registration and after using the tool. We received 539 preusage survey responses (out of 17,491 total papers submitted to the conference), 234 submissions to the Checklist Assistant, and 65 distinct postusage survey responses. To evaluate robustness of the Checklist Assistant, we manipulated responses to the Assistant to test whether it would be possible for authors to manipulate the Assistant’s scores of checklist accuracy.&lt;/p&gt;
&lt;h4&gt;Results&lt;/h4&gt;
&lt;p&gt; In 65 postusage surveys (&lt;b&gt;Figure 25-0958&lt;/b&gt;), 46 authors found the Assistant useful and 46 indicated they would revise their papers or checklist responses based on its feedback. Authors’ expectations for the Checklist Assistant were more positive (59 of 65 positive responses on usefulness) than their perceptions after usage (46 of 65), potentially reflecting an overly optimistic outlook on the usefulness of the tool. Analysis of resubmissions indicated that authors made substantive revisions to their submissions in response to specific feedback from the LLM. Of 40 instances when authors submitted to the Assistant multiple times, they changed their answers 39 of 40 times and often increased the length of their checklist justifications significantly. Inaccuracy (20 of 52 free-form survey responses) and excessive strictness (14 of 52) were the most frequent issues flagged by authors. For more than half of submissions, the LLM recommended changes to at least 12 of 15 checklist items. Authors most commonly reported plans to improve justifications by adding detail or citations (n = 14) and clarify experimental details or data descriptions (n = 6). On 14 of 15 questions, the Assistant could be manipulated to improve accuracy scores by changing checklist content without improving the paper.&lt;/p&gt;

![](https://assets.underline.io/markdown_image/1/image/be46bf0146e81fb335482d0967f37894.png)

&lt;h4&gt;Conclusions&lt;/h4&gt;
&lt;p&gt; In this study, an LLM Checklist Assistant was shown to aid a small group of authors to ensure scientific rigor. However, the tool should not be used as a fully automated review tool that replaces human review.&lt;/p&gt;
&lt;h4&gt;&lt;a id=&quot;_idTextAnchor004&quot;&gt;&lt;/a&gt;References&lt;/h4&gt;
&lt;p&gt;1. Moher D, Schulz KF, Altman DG. The CONSORT statement: revised recommendations for improving the quality of reports of parallel-group randomised trials. &lt;i&gt;Lancet&lt;/i&gt;. 2001:1191-1194.&lt;/p&gt;
&lt;p&gt;2. Vandenbroucke JP, et al. Strengthening the Reporting of Observational Studies in Epidemiology (STROBE): explanation and elaboration. &lt;i&gt;Int J Surg&lt;/i&gt;. 2014:1500-1524.&lt;/p&gt;
&lt;sup&gt;1&lt;/sup&gt;School of Computer Science, Carnegie Mellon University, Pittsburgh, PA, US, akgoldbe@andrew.cmu.edu; &lt;sup&gt;2&lt;/sup&gt;ChaLearn, Berkeley, CA, US; &lt;sup&gt;3&lt;/sup&gt;University of Paris-Saclay, Paris, France; &lt;sup&gt;4&lt;/sup&gt;Department of Computer Science, The University of Chicago, Chicago, IL, US; &lt;sup&gt;5&lt;/sup&gt;Google DeepMind, San Francisco, CA, US.&lt;/p&gt;
&lt;h4&gt;Conflict of Interest Disclosures &lt;/h4&gt;
&lt;p&gt;Nihar B. Shah is a member of the Peer Review Congress Advisory Board but was not involved in the review or decision for this abstract.&lt;/p&gt;
&lt;h4&gt;Funding/Support &lt;/h4&gt;
&lt;p&gt;This research project has benefitted from the Microsoft Accelerating Foundation Models Research (AFMR) grant program, an INRIA Google Research grant, the ANR Chair of Artificial Intelligence HUMANIA ANR-19-CHIA-0022, NSF 1942124, and 2200410.&lt;/p&gt;
&lt;h4&gt;Acknowledgment &lt;/h4&gt;
&lt;p&gt;In preparing this experiment, we received advice and help from many people. We are particularly grateful to the NeurIPS 2024 organizers, including General Chair Amir Globerson; Program Chairs Danielle Belgrave, Cheng Zhang, Angela Fan, Jakub Tomczak, and Ulrich Paquet; and workflow team member Babak Rahmani, for participating in brainstorming discussions and contributing to the design. We have also received input and encouragement from Andrew McCallum of OpenReview, Anurag Acharya from Google Scholar, and Tristan Neuman from the NeurIPS board. Several volunteers contributed ideas and helped with various aspects of the preparation, including Jeremiah Liu, Lisheng Sun, Paulo Henrique Couto, Michael Brenner, Neha Nayak Kennard, and Adrien Pavao. This research project has benefited from the Microsoft AFMR grant program. We are grateful to Marc Schoenauer for supporting this effort with an INRIA Google Research grant. We acknowledge the support of ChaLearn, the ANR Chair of Artificial Intelligence HUMANIA ANR-19-CHIA-0022, NSF 1942124, and 2200410. Importantly, we are thankful to all the participants of the Checklist Assistant for volunteering to try it out and providing their valuable feedback.&lt;/p&gt;


The 10th Peer Review Congress

Use of an LLM as an Author Checklist Assistant for Scientific Papers: NeurIPS&#39;24 Experiment

artificial intelligence and large language models

quality of reporting

reporting guidelines

The Tenth Peer Review Congress featured 3 days for presentations of new research into peer review and all aspects of scientific publication, bias, quality of reporting, and information access and dissemination.

This year’s meeting was hybrid, and all plenary sessions were livestreamed with opportunities to view presentations and engage with participants in person and virtually during and after the meeting. All recordings are now available on Underline for viewing.

<div style="position:relative;padding-top:max(60%,324px);width:100%;height:0;"><iframe style="position:absolute;border:none;width:100%;height:100%;left:0;top:0;" src="https://online.fliphtml5.com/ebtyf/mget/"  seamless="seamless" scrolling="no" frameborder="0" allowtransparency="true" allowfullscreen="true" ></iframe></div>


#### Why? 
Our aim is to encourage research into the quality and credibility of peer review and scientific publication, to establish the evidence base on which scientists can improve the conduct, reporting, and dissemination of scientific research.

#### Who? 
Participants include editors and publishers of scientific peer-reviewed journals, researchers, funders, bibliometric and informatics experts, information innovators, librarians, journalists, policymakers, ethicists, scientific information disseminators, and anyone interested in the progress of the scientific information enterprise and the quality of scientific evidence.

The Congress embraces a wide range of disciplines, including biomedicine, health and life sciences, applied sciences, basic sciences, physical and chemical sciences, mathematics, computer sciences, engineering, economics, and social sciences as well as new and emerging disciplines.

#### What? 
The Congress provides a forum for the presentation and discussion of new research into the quality and credibility of editorial and funding peer review, scientific and scholarly publication, research methods and reporting practices, identification and management of biases, transparency, reproducibility, access, publication and dissemination metrics, identification and management of threats to the integrity of science, and advances in information exchange.

**Follow us**

[![](https://assets.underline.io/markdown_image/1/image/662a22766b9788ed3766cb23cd156e32.png)](https://x.com/peerrevcongress)  [![](https://assets.underline.io/markdown_image/1/image/bb105afb1773a01fa169aa97ca1af49a.png)](https://www.youtube.com/channel/UCsG_0nKi3_Zfyv0yRPdxjWQ)  [![](https://assets.underline.io/markdown_image/1/image/034e1e70a194a9a4178fdf470ef796db.png)](https://bsky.app/profile/peerreviewcongress.bsky.social)



You need to log in with the email address you registered with. 



Please log in to explore this event.

Registration is required to attend this event, please [here](https://peerreviewcongress.org/registration-information/) to secure your ticket. Please check your spam/promotional inbox too.



Please register!

The Tenth Peer Review Congress features 3 days for presentations of new research into peer review and all aspects of scientific publication, bias, quality of reporting, and information access and dissemination.

<p>Alexander Goldberg,<sup>1</sup> Ihsan Ullah,<sup>2</sup> Thanh Gia Hieu Khuong,<sup>3</sup> Benedictus Kent Rachmat,<sup>3</sup> Zhen (Zach) Xu,<sup>4</sup> Isabelle Guyon,<sup>2,3,5</sup> Nihar B. Shah<sup>1</sup></p>
<h4>Objective </h4>
<p>This study assessed the utility of a large language model (LLM) in evaluating compliance with submission standards at the 2024 Neural Information Processing Systems (NeurIPS) conference—a top-tier publication venue in artificial intelligence (AI) with a 15% to 25% acceptance rate that reviews full papers, not just abstracts. NeurIPS requires authors to complete a 15-question checklist promoting reproducibility, transparency, and ethical standards, similar to CONSORT<sup>1</sup> and STROBE.<sup>2</sup> We introduced an optional Checklist Assistant using GPT-4 (OpenAI) to provide presubmission feedback to authors on checklist accuracy.</p>
<h4>Design </h4>
<p>We conducted a cross-sectional study in which the Checklist Assistant was available to authors 8 days before the submission deadline. Authors could optionally use the Checklist Assistant, which used a general purpose third-party LLM (gpt-4-turbo-preview) to evaluate the accuracy of their responses to a 15-question checklist. For each question, the LLM received the author’s response, justification, and the full paper (including appendices), and assessed the accuracy of the response using simple prompt engineering. We conducted surveys about authors’ perceptions of the tool at registration and after using the tool. We received 539 preusage survey responses (out of 17,491 total papers submitted to the conference), 234 submissions to the Checklist Assistant, and 65 distinct postusage survey responses. To evaluate robustness of the Checklist Assistant, we manipulated responses to the Assistant to test whether it would be possible for authors to manipulate the Assistant’s scores of checklist accuracy.</p>
<h4>Results</h4>
<p> In 65 postusage surveys (<b>Figure 25-0958</b>), 46 authors found the Assistant useful and 46 indicated they would revise their papers or checklist responses based on its feedback. Authors’ expectations for the Checklist Assistant were more positive (59 of 65 positive responses on usefulness) than their perceptions after usage (46 of 65), potentially reflecting an overly optimistic outlook on the usefulness of the tool. Analysis of resubmissions indicated that authors made substantive revisions to their submissions in response to specific feedback from the LLM. Of 40 instances when authors submitted to the Assistant multiple times, they changed their answers 39 of 40 times and often increased the length of their checklist justifications significantly. Inaccuracy (20 of 52 free-form survey responses) and excessive strictness (14 of 52) were the most frequent issues flagged by authors. For more than half of submissions, the LLM recommended changes to at least 12 of 15 checklist items. Authors most commonly reported plans to improve justifications by adding detail or citations (n = 14) and clarify experimental details or data descriptions (n = 6). On 14 of 15 questions, the Assistant could be manipulated to improve accuracy scores by changing checklist content without improving the paper.</p>

![](https://assets.underline.io/markdown_image/1/image/be46bf0146e81fb335482d0967f37894.png)

<h4>Conclusions</h4>
<p> In this study, an LLM Checklist Assistant was shown to aid a small group of authors to ensure scientific rigor. However, the tool should not be used as a fully automated review tool that replaces human review.</p>
<h4><a id="_idTextAnchor004"></a>References</h4>
<p>1. Moher D, Schulz KF, Altman DG. The CONSORT statement: revised recommendations for improving the quality of reports of parallel-group randomised trials. <i>Lancet</i>. 2001:1191-1194.</p>
<p>2. Vandenbroucke JP, et al. Strengthening the Reporting of Observational Studies in Epidemiology (STROBE): explanation and elaboration. <i>Int J Surg</i>. 2014:1500-1524.</p>
<sup>1</sup>School of Computer Science, Carnegie Mellon University, Pittsburgh, PA, US, akgoldbe@andrew.cmu.edu; <sup>2</sup>ChaLearn, Berkeley, CA, US; <sup>3</sup>University of Paris-Saclay, Paris, France; <sup>4</sup>Department of Computer Science, The University of Chicago, Chicago, IL, US; <sup>5</sup>Google DeepMind, San Francisco, CA, US.</p>
<h4>Conflict of Interest Disclosures </h4>
<p>Nihar B. Shah is a member of the Peer Review Congress Advisory Board but was not involved in the review or decision for this abstract.</p>
<h4>Funding/Support </h4>
<p>This research project has benefitted from the Microsoft Accelerating Foundation Models Research (AFMR) grant program, an INRIA Google Research grant, the ANR Chair of Artificial Intelligence HUMANIA ANR-19-CHIA-0022, NSF 1942124, and 2200410.</p>
<h4>Acknowledgment </h4>
<p>In preparing this experiment, we received advice and help from many people. We are particularly grateful to the NeurIPS 2024 organizers, including General Chair Amir Globerson; Program Chairs Danielle Belgrave, Cheng Zhang, Angela Fan, Jakub Tomczak, and Ulrich Paquet; and workflow team member Babak Rahmani, for participating in brainstorming discussions and contributing to the design. We have also received input and encouragement from Andrew McCallum of OpenReview, Anurag Acharya from Google Scholar, and Tristan Neuman from the NeurIPS board. Several volunteers contributed ideas and helped with various aspects of the preparation, including Jeremiah Liu, Lisheng Sun, Paulo Henrique Couto, Michael Brenner, Neha Nayak Kennard, and Adrien Pavao. This research project has benefited from the Microsoft AFMR grant program. We are grateful to Marc Schoenauer for supporting this effort with an INRIA Google Research grant. We acknowledge the support of ChaLearn, the ANR Chair of Artificial Intelligence HUMANIA ANR-19-CHIA-0022, NSF 1942124, and 2200410. Importantly, we are thankful to all the participants of the Checklist Assistant for volunteering to try it out and providing their valuable feedback.</p>


Use of an LLM as an Author Checklist Assistant for Scientific Papers: NeurIPS'24 Experiment

poster

<p>Vicente Amado Olivo,<sup>1</sup> Wolfgang Kerzendorf,<sup>1</sup> Nutan Chen,<sup>2</sup> Joshua V. Shields,<sup>1</sup> Bangjing Lu,<sup>1</sup> Andreas Flörs<sup>3</sup></p>
<h4>Objective</h4>
<p> The process of identifying peer reviewers is becoming more difficult due to a surge in submissions and declining acceptance rates for review invitations.<sup>1</sup> Responses to a 2024 IOP Publishing online survey suggest the peer review system is unevenly distributed, with 10% of reviewers conducting 50% of all reviews, while early-career researchers and scholars from underrepresented regions remain underused despite being willing to review.<sup>2</sup> Reviewer identification tools have been introduced to assist editors, but the tools are proprietary and closed source, limiting their accessibility and transparency to the broader scientific community. To expand the global pool of peer reviewers, open-source, AI-powered methods are needed to uniquely identify researchers within the expanding scientific literature and match them with appropriate review opportunities. Current author name disambiguation methods often rely on extensive data features, such as institutional affiliations, email addresses, or publication venues, which are not consistently available across digital libraries. We describe the development of a name disambiguation tool with fewer data features.</p>
<h4>Design</h4>
<p> Given the lack of existing author disambiguation tools for the Smithsonian Astrophysical Observatory/National Aeronautics and Space Administration Astrophysics Data System, we developed the Neural Author Name Disambiguator (NAND), a similarity-based neural network, and trained it on pairs of publications authored by individuals sharing the same name labeled with Open Researcher and Contributor ID (ORCID) identifiers. The training dataset included 2,698,778 pairs balanced between classes (0 if they shared the same ORCID and 1 if they had different ORCIDs), 553,496 unique publications, and 125,486 ORCID profiles of authors from the 2020 ORCID open data snapshot (and annual public data release). NAND was trained to disambiguate publications with minimal features (ie, author name, title, and abstract). We validated NAND performance using standard classification metrics: accuracy (percentage of correctly classified publication pairs), precision (true positives / [true positives + false positives]), recall (true positives / [true positives + false negatives]), and F1 scores (true positives / [true positives + 0.5(false positives + false negatives)]).</p>
<h4>Results</h4>
<p> NAND achieved a mean (SD) 94.64% (0.04%) accuracy on test set pairs of authors within the same name block (eg, J. Smith or Y. Wang). Mean (SD) precision, recall, and F1 scores were 96.67% (0.05%), 95.21% (0.11%), and 95.94% (0.03%), respectively.</p>
<h4>Conclusions</h4>
<p> Combining the disambiguation model with semantic expertise matching techniques could establish a practical framework for identifying qualified and willing reviewers across global institutions.<sup>3</sup> The open framework may help to reduce the burden on overused reviewers by expanding the global pool of available reviewers. While results are promising, this analysis was limited to physics publications; further validation is needed to assess generalizability to other scientific domains.</p>
<h4>References</h4>
<p>1. Publons. 2018 Global State of Peer Review. Accessed January 29, 2025. <a href="https://publons.com/static/Publons-Global-State-Of-Peer-Review-2018.pdf"><span class="Hyperlink CharOverride-6">https://publons.com/static/Publons-Global-State-Of-Peer-Review-2018.pdf</span></a></p>
<p>2. Brigham L, Brent-Jones E, Coombs A, et al. State of peer review 2024. Accessed January 29, 2025. <a href="https://ioppublishing.org/state-of-peer-review-2024-results/"><span class="Hyperlink CharOverride-6">https://ioppublishing.org/state-of-peer-review-2024-results/</span></a></p>
<p>3. Kerzendorf WE, Patat F, Bordelon D, van de Ven G, Pritchard TA. Distributed peer review enhanced with natural language processing and machine learning. <i>Nat Astronomy</i>. 2020;4:711-717. doi:10.1038/s41550-020-1038-y</p>
<sup>1</sup>Michigan State University, East Lansing, MI, US, amadovic@msu.edu; <sup>2</sup>Volkswagen Group, Wolfsburg, Germany; <sup>3</sup>GSI Helmholtzzentrum für Schwerionenforschung, Darmstadt, Germany.</p>
<h4>Conflict of Interest Disclosures</h4>
<p> None reported.</p>
<h4>Funding/Support</h4>
<p> This work is supported in part by the National Science Foundation Research Traineeship Program (DGE-2152014) to Vicente Amado Olivo. Additionally, we gratefully acknowledge the European Space Agency for funding support through a traineeship for Vicente Amado Olivo.</p>
<h4>Additional Information</h4>
<p> We acknowledge the support and guidance from Markus Kissler-Patig and Jan Reerink at the European Space Agency. The authors used the free versions of ChatGPT (OpenAI) and Claude (Anthropic) for brainstorming and editing for logical flow. We take responsibility for the integrity of the content generated.</p>


Accuracy and Precision of a Neural Network Author Name Disambiguator

<p>Peiling Wang,<sup>1</sup> Dietmar Wolfram,<sup>2</sup> Scott Shumate<sup>3</sup></p>
<h4>Objective </h4>
<p>Open peer review processes promote transparency and accountability. As more publishers adopt open peer review,<sup>1</sup> accessible peer reviews become valuable corpora to develop AI tools for enhancing peer review processes.<sup>2</sup> The main purpose of peer review as stated in the Guidelines for Reviewers<sup>3</sup> is, “First, reviews provide constructive advice and recommendations to the authors on how their paper can be improved.” How can we determine the constructiveness of peer review reports? This exploratory study analyzed 11,321 peer review reports of the first version of 4100 manuscripts submitted to, and published in, <i>PeerJ</i> to identify constructive comments. Specifically, this dataset includes first reviews from November 11, 2012, to November 24, 2019. This study examines 3 research questions: (1) To what extent do reviewers provide constructive comments? (2) To what extent do reviewers use hedging to soften criticisms or recommendations? (3) Are there differences in constructive comments between signed and anonymous reviews?</p>
<h4>Design </h4>
<p>The gpt-4o-2024-08-06 model from OpenAI API was used to analyze reviews at the sentence level to identify criticisms or supportive comments about different aspects of the research, including study purpose, hypotheses, methods, results, discussion, and conclusions. A detailed system prompt (<a href="https://zenodo.org/records/15464322">https://zenodo.org/records/15464322</a>) was used with a temperature setting of 0. The model was instructed to categorize each sentence into a structured JSON output specifying section, aspects, evaluation, tone, and mode. Use of hedging tone includes phrases such as “I do not think the investigation was ‘rigorous’” or “has the potential to be much better and more relevant.” Building on the GPT analysis, we derived the following variables to measure review feature aspects, including (1) Pc, the percentage of critical sentences; (2) Ps, the percentage of supportive sentences; and (3) Ph, the percentage of hedging sentences. The constructive index (CI), considering both criticisms and supportive comments, is the percentage of sentences containing research comments.</p>
<h4>Results </h4>
<p>Of the 11,321 peer review reports, 7071 (62.5%) were anonymous and 4250 (37.5%) were signed (<b>Table 25-1055</b>). Few reports were supportive (overall median of 9%). Higher levels (27% of anonymous reviews and 25% of signed reviews) contained criticisms of the research. A similar level of hedging was observed. Significant differences were observed between the 2 groups; signed reviews showed significantly fewer criticisms and higher levels of hedging.</p>

![](https://assets.underline.io/markdown_image/1/image/c3c1db9d57bb2102d48d254f10bb4a4e.png)

<h4>Conclusions </h4>
<p>We measured the extent of research-related comments in peer review reports that authors opted to publish. Anonymous reviewers tend to be more critical. The use of hedging in reviews could obscure clear suggestions on how to improve the manuscript. This research represents an initial effort towards a model for an AI-assisted peer review system. Further analysis should fine-tune the AI algorithms for analysis of reviewer comments. In addition, authors’ perspectives on what constitutes constructive feedback warrant further study. This research had limitations. Review histories were not available for rejected manuscripts or for articles where authors opted out of publishing reviews.</p>
<h4>References</h4>
<p>1. Wolfram D, Wang P, Hembree A, Park H. Open peer review: promoting transparency in open science. <i>Scientometrics. </i>2020;125:1033-1051. doi:10.1007/s11192-020-03488-4</p>
<p>2. Wolfram D, Wang P, Abuzahra F. An exploration of referees’ comments published in open peer review journals: the characteristics of review language and the association between review scrutiny and citations. <i>Res Eval. </i>2021;30(3):314-322<i>. </i>doi:10.1093/reseval/rvab005</p>
<p>3. Technical Community on Real-Time Systems (TCRTS) of the IEEE Computer Society. Guidelines for reviewers. Accessed on May 29, 2025. <a href="https://cmte.ieee.org/tcrts/guidelines-for-reviewers"><span class="Hyperlink CharOverride-6">https://cmte.ieee.org/tcrts/guidelines-for-reviewers</span></a></p>
<sup>1</sup>University of Tennessee, Knoxville, US, peilingw@utk.edu; <sup>2</sup>University of Wisconsin, Milwaukee, US; <sup>3</sup>Austin Peay State University, Clarksville, TN, US.</p>
<h4>Conflict of Interest Disclosures</h4>
<p> None reported.</p>


An AI-assisted Analysis of Published PeerJ Open Peer Reviews

<p>Chirag J. Patel,<sup>1</sup> Arjun K. Manrai,<sup>1</sup> Randall J. Ellis,<sup>1</sup> John P. A. Ioannidis<sup>2</sup></p>
<h4>Background</h4>
<p> Large language models (LLMs) are increasingly used to synthesize biomedical evidence, yet it is unknown whether their inferred exposure-phenotype relationships mirror observational findings.<sup>1,2</sup> </p>
<h4>Objective</h4>
<p> To quantify concordance of exposure-phenotype empirical associations using National Health and Nutrition Examination Survey (NHANES) data with predictions from an LLM (ChatGPT 4o [OpenAI]).</p>
<h4>Methods</h4>
<p> We computed 115,056 exposure-phenotype partial correlations among 560 chemical, dietary, and clinical exposures and 268 phenotypes in 80,000 US adults (1999-2018 NHANES).<sup>3</sup> We then sampled 1500 pairs: 500 significant after Bonferroni correction, 500 significant only after false-discovery rate (FDR) correction, and 500 not significant—to fit within the LLM prompt window. An OpenAI-o1-aug24 model (temperature = 0) acting as a prompted “epidemiologist” was asked to predict the sign, magnitude, and <i>P</i> value of each pair and rated the evidence generated as high, trending, or borderline. Verifiability of LLM-cited references was assessed by estimating κ values and Spearman correlation.</p>
<h4>Results</h4>
<p> The LLM labeled 23 pairs as high evidence, 244 as trending evidence, and 1233 as borderline evidence. Sign concordance with NHANES was 73% for high-confidence pairs (κ = 0.49), 65% for Bonferroni pairs (κ = 0.25), 43% for FDR-only pairs (κ = 0.09), and 12% for borderline pairs (κ approximately 0) (<b>Figure 25-1123</b>). Random sign assignment yielded a κ of 0.0001. Concordance for correlation magnitudes was modest (Spearman ρ = 0.21). Permutation tests confirmed no greater-than-chance performance on nonsense data. Mechanistic explanations dominated high-confidence pairs; 41% of cited references (615 of 1500) were unverifiable.</p>

![](https://assets.underline.io/markdown_image/1/image/ca7fd3c645a90192139dbf1d062ea8bc.png)

<h4>Conclusions</h4>
<p> LLMs captured directionality for well-established exposure-phenotype associations but faltered on borderline signals, highlighting both the promise and current limits of LLM-based evidence synthesis in exposomic epidemiology.</p>
<h4>References</h4>
<p>1. Ioannidis JPA, Loy EY, Poulton R, Chia KS. Researching genetic versus nongenetic determinants of disease: a comparison and proposed unification. <i>Sci Transl Med.</i> 2009;1(7):7ps8. doi:10.1126/scitranslmed.3000247</p>
<p>2. Patel CJ, Ioannidis JPA. Studying the elusive environment in large scale. <i>JAMA</i>. 2014;311(21):2173-2174. doi:10.1001/jama.2014.4129</p>
<p>3. Patel CJ, Rehkopf DH, Leppert JT, et al. Systematic evaluation of environmental and behavioural factors associated with all-cause mortality in the United States National Health and Nutrition Examination Survey. <i>Int J Epidemiol</i>. 2013;42(6):1795-1810. doi:10.1093/ije/dyt208</p>
<sup>1</sup>Department of Biomedical Informatics, Harvard Medical School, Boston, MA, US, chirag@hms.harvard.edu; <sup>2</sup>Department of Medicine, Epidemiology and Population Health, Stanford University School of Medicine, Stanford, CA, US.</p>
<h4>Conflict of Interest Disclosures</h4>
<p> John P. A. Ioannidis is a member of the Peer Review Congress Advisory Board but was not involved in the review or decision for this abstract.</p>
<h4>Funding/Support</h4>
<p> This study was funded by grants R01ES032470 and U24ES036819 from the National Institutes of Environmental Health Sciences and grant R01DK137993 from the National Institute of Diabetes and Digestive and Kidney Diseases.</p>
<h4>Role of Funder/Sponsor</h4>
<p> The sponsors had no role in this study.</p>


Comparing Observational Exposure-Phenotype Correlations with Large Language Model Predictions

<p>Nicola Di Girolamo,<sup>1,2</sup> Reint Meursinge Reynders,<sup>3,4</sup> Ugo Di Girolamo<sup>5</sup></p>
<h4>Objective </h4>
<p>Consolidated Standard of Reporting of Trials (CONSORT) checklists help authors to adhere to reporting standards and allow editors and peer reviewers to ensure that critical information is reported in a randomized controlled trial (RCT) at a glance.<sup>1</sup> We pragmatically assessed the ability of 3 artificial intelligence (AI) large language model (LLM) chatbots provided with minimal instructions to properly complete a CONSORT checklist from manuscripts reporting RCTs before peer review in 2025.</p>
<h4>Design </h4>
<p>This cross-sectional study was registered on the Open Science Framework.<sup>2</sup> PDFs of manuscripts reporting RCTs before peer review that were consecutively published in <i>The BMJ</i> were extracted in reverse-chronological order for a total of 50 manuscripts. <i>The BMJ</i> granted permission for this study. In February 2025, each PDF was uploaded exactly as submitted by the authors to 3 LLMs (ChatGPT-4o, Gemini Advanced 2.0 Flash, and Claude 3.5 Sonnet), which were instructed to complete the CONSORT checklist in a table format,<sup>3</sup> including the reporting of the 37 CONSORT items and extraction of the text where the item was reported. The number of iterations and total time (seconds) required to obtain a visually acceptable table were recorded. Sensitivity, specificity, and accuracy for each LLM were calculated with human manual extraction as the reference standard. Manual extraction was performed by 1 operator (N.D.) with expertise in RCT methods. Generalized linear mixed models were built to evaluate the effect of PDF characteristics (number of pages, file size, and number of words) on the disagreements between AI and human assessment.</p>
<h4>Results </h4>
<p>Of the 1850 CONSORT items, ChatGPT provided decisions for 1776 (96.0%); 1159 of these were in agreement with human extraction (overall accuracy, 65.3% [95% CI, 63.0%-67.5%]; sensitivity, 61.3% [95% CI, 55.0%-67.3%]; specificity, 65.9% [95% CI, 63.5%-68.3%]). Gemini provided decisions for 1629 items (88.0%), of which 1521 were in agreement with human extraction (accuracy, 93.4% [95% CI, 92.1%-94.5%]; sensitivity, 97.6% [95% CI, 94.8%-99.1%]; specificity, 92.6% [95% CI, 91.1%-94.0%]). Claude provided decisions for 1480 items (80.0%), of which 1375 were in agreement with human extraction (accuracy, 92.9% [95% CI, 91.5%- 94.2%]; sensitivity, 56.1% [95% CI, 49.3%-62.8%]; specificity, 99.4% [95% CI, 98.8%-99.7%]). Accuracy of LLMs was particularly low for certain CONSORT items (<b>Figure 25-1152</b>). There were no significant associations between PDF characteristics and AI and human disagreements. Trial PDFs were a mean of 49 pages and 17,268 words. LLMs were able to compile visually acceptable CONSORT checklists on the first iteration in 76.7% (115 of 150) attempts: ChatGPT for 40 RCTs, Gemini for 36 RCTs, and Claude for 39 RCTs. Generation of a CONSORT checklist required a mean of 75 seconds, 38 seconds, and 47 seconds for ChatGPT, Gemini, and Claude, respectively (range, 16-249 seconds).</p>

![](https://assets.underline.io/markdown_image/1/image/39e42ee902f1b80f6a363027b6df1d1e.png)

<h4>Conclusions </h4>
<p>In early 2025, the LLMs Gemini and Claude, but not ChatGPT, were able to generate accurate CONSORT checklists from PDFs of manuscripts reporting RCTs before peer review in over 90% of cases, given minimal instructions and often in less than 1 minute.</p>
<h4>References<a id="_idTextAnchor008"></a><a id="_idTextAnchor009"></a></h4>
<p>1. Moher D, Jones A, Lepage L; CONSORT Group (Consolidated Standards for Reporting of Trials). Use of the CONSORT statement and quality of reports of randomized trials: a comparative before-and-after evaluation. <i>JAMA</i>. 2001;285(15):1992-1995. doi:10.1001/jama.285.15.1992</p>
<p>2. Di Girolamo N, Meursinge Reynders R, Di Girolamo U. Comparison of different AI large language models for extraction of CONSORT items from submitted randomized controlled trials before peer-review. Open Science Framework. Registered February 13, 2025. Accessed July 5, 2025. <a href="https://osf.io/5v6h2"><span class="Hyperlink CharOverride-6">https://osf.io/5v6h2</span></a></p>
<p>3. Schulz KF, Altman DG, Moher D; CONSORT Group. CONSORT 2010 Statement: updated guidelines for reporting parallel group randomised trials. <i>BMJ</i>. 2010;340:c332. doi:10.1136/bmj.c332</p>
<sup>1</sup>Department of Clinical Sciences, College of Veterinary Medicine, Cornell University, Ithaca, NY, US, nd374@cornell.edu; <sup>2</sup>Journal of Small Animal Practice, British Small Animal Veterinary Association, Gloucestershire, UK; <sup>3</sup>Department of Oral and Maxillofacial Surgery, Amsterdam University Medical Center, University of Amsterdam, Amsterdam, the Netherlands; <sup>4</sup>Private Practice of Orthodontics, Milan, Italy; <sup>5</sup>Compass, New York, NY, US.</p>
<h4>Conflict of Interest Disclosures </h4>
<p>Nicola Di Girolamo is editor in chief of 2 peer-reviewed journals, one published by Elsevier and one by Wiley. No other disclosures were reported.</p>


Pragmatic Assessment of Different AI Large Language Models for Extraction of CONSORT Items From Randomized Controlled Trials Before Peer Review

<p>Josh Major,<sup>1</sup> Kurt Mahnken,<sup>1</sup> Alec Young,<sup>1</sup> Cameron O’Brien,<sup>1</sup> Andrew V. Tran,<sup>1</sup> Patrick Crotty,<sup>1</sup> Alicia Ito Ford,<sup>1,2</sup> Matt Vassar<sup>1,2</sup></p>
<h4>Objective</h4>
<p> The integration of artificial intelligence (AI) in scientific research presents both opportunities and challenges, particularly concerning transparency, ethics, and reproducibility.<sup>1</sup> This study aimed to evaluate how leading orthopedic and sports medicine (OSM) journals address 2 distinct outcomes: (1) the extent to which AI is permitted in research and publication and (2) the degree to which OSM journals are endorsing AI-specific reporting guidelines (RGs). The primary outcome was to assess the presence of AI-related policies, including requirements for AI disclosure, acceptance or prohibition of AI-generated images and content, and AI’s role in manuscript writing and authorship. Secondary outcomes examined the endorsement of AI-specific RGs and the relationship between AI policies and Journal Impact Factor.</p>
<h4>Design</h4>
<p> We conducted a cross-sectional analysis of the top 100 OSM journals ranked by the 2023 SCImago Journal Rank (SJR) indicator. Eligibility criteria included actively publishing peer-reviewed clinical journals with publicly accessible Instructions for Authors in English language. On September 29, 2024, 2 investigators independently screened, extracted, and reconciled data in a masked, duplicate manner. Data were extracted from journals’ Instructions for Authors pages and affiliated publisher websites where indicated. If a journal had no mention of AI policies or AI-specific RGs, the editorial teams were contacted directly to inquire about these gaps. Biserial correlation analyses in R version 4.4.1 (The R Foundation) and RStudio (Posit) examined AI policy relationships with Journal Impact Factor and SJR.</p>
<h4>Results</h4>
<p> The initial search yielded 319 journals, from which the top 100 were analyzed. Of these, 94 mentioned AI in their Instructions for Authors, with all 94 requiring disclosure of AI use and prohibiting AI as an author (<b>Table 25-1175</b>). Additionally, AI-assisted manuscript writing was permitted in all cases, while 87% allowed AI-assisted content generation and 64% permitted AI-generated images. Despite this widespread recognition of AI policies, AI-specific RGs were endorsed in only 1% of journals, recommending the use of the Checklist for Artificial Intelligence in Medical Imaging (CLAIM) RG. No significant associations were observed between Journal Impact Factor and the presence of AI-related editorial policies. Additionally, no differences were found among the AI policies in the 4 quartiles of the OSM journals in this study.</p>
<a href="/wp-content/uploads/2025_conference/tables/prc25-1175t1.png"> <img loading="lazy" src="/wp-content/uploads/2025_conference/tables/prc25-1175t1.png"> </a>
<h4>Conclusions</h4>
<p> While some AI use in research is widely acknowledged by OSM journals, the lack of consistency in policies regarding AI-generated images and content suggest an area for improvement to standardize how AI can be applied in these journals. Furthermore, the lack of endorsement for AI-specific RGs suggests a critical gap in ensuring the transparency and methodological rigor of AI-integrated research. To address these issues, we recommend that OSM journals establish clear AI policies and endorse AI-specific RGs. The International Committee of Medical Journal Editors, Committee on Publication Ethics, and World Association of Medical Editors have taken initial steps; however, implementation may be difficult.<sup>2</sup> Implementing these measures will bridge existing gaps, promote transparency, and improve research quality in scientific publishing.</p>
<h4>References</h4>
<p>1. Prasana P, Mandal PK, Hussain D, et al. Intersection of orthopaedics and artificial intelligence: a review. <i>SSR Inst Int J Life Sci</i>. 2024;10(3):5544-5552. doi:10.21276/ssr-iijls.2024.10.3.21</p>
<p>2. Simera I, Moher D, Hoey J, Schulz KF, Altman DG. The EQUATOR Network and reporting guidelines: helping to achieve high standards in reporting health research studies. <i>Maturitas</i>. 2009;63(1):4-6. doi:10.1016/j.maturitas.2009.03.011</p>
<sup>1</sup>Office of Medical Student Research, Oklahoma State University Center for Health Sciences, Tulsa, OK, US, youngalec9.r@gmail.com; <sup>2</sup>Department of Psychiatry and Behavioral Sciences, Oklahoma State University Center for Health Sciences, Tulsa, OK, US.</p>
<h4>Conflict of Interest Disclosures</h4>
<p> Alicia Ito Ford reports receipt of funding from the Center for Integrative Research on Childhood Adversity, the Oklahoma Shared Clinical and Translational Resources, and internal grants from Oklahoma State University and Oklahoma State University Center for Health Sciences outside of the present work. Matt Vassar reports receipt of funding from the National Institute on Drug Abuse, the National Institute on Alcohol Abuse and Alcoholism, the US Office of Research Integrity, Oklahoma Center for Advancement of Science and Technology, and internal grants from Oklahoma State University Center for Health Sciences outside of the present work. No other disclosures were reported.</p>
<h4>Additional Information</h4>
<p> Andrew V. Tran is a co–corresponding author (andrewtranresearch@gmail.com).</p>


Artificial Intelligence Editorial Policies and Reporting Standards in Orthopedic and Sports Medicine Journals

<p>Heather Goodell,<sup>1</sup> Christine Beaty,<sup>1</sup> Jonathan Schultz,<sup>1</sup> Shilpi Mehra,<sup>2</sup> Chirag Jay Patel<sup>3</sup></p>
<h4>Objective</h4>
<p> The increase in abstract submissions to major scientific conferences requires efficient and reliable methods to ensure compliance with research integrity standards, which are essential for maintaining the credibility of science. Traditional review methods may not fully capture key issues, requiring a structured approach to evaluate submissions before final acceptance. This study presents a review process that uses artificial intelligence (AI) with human validation to assess the integrity of abstracts submitted to the American Heart Association (AHA) Scientific Sessions.</p>
<h4>Design</h4>
<p> An integrity evaluation process was applied by combining AI-driven analysis with human review of abstracts submitted for the 2024 AHA Scientific Sessions. An AI tool (Paperpal Preflight for Editorial Desk) was used to identify research integrity issues, including AI-generated text, author expertise misalignment, and reference accuracy. Subject matter experts then reviewed abstracts flagged for validation. AI checks categorized abstracts into <i>passed</i>, <i>warning</i>, or <i>critical</i> levels, with those not marked as <i>passed</i> requiring further human review.</p>
<h4>Results</h4>
<p> Among the 8477 submitted abstracts that were analyzed, 42 were flagged with a warning for integrity concerns (AI-generated text, author expertise misalignment, and reference accuracy); human reviewers cleared 3 of these. Further analysis revealed that 167 authors submitted 15 or more abstracts, 63 authors submitted 20 or more, and 13 authors submitted more than 30 abstracts. A total of 2624 abstracts had at least 1 author with 10 or more submissions, and 1438 abstracts had at least 1 author with 15 or more submissions. The number of submissions was considered when assessing research integrity concerns. Analysis of author submission counts helped flag questionable cases for closer review and the identification of potentially suspect activities.</p>
<h4>Conclusions</h4>
<p> The hybrid AI-human review process for abstract submissions was demonstrated to be an efficient and accurate evaluation method that should be scalable. By addressing concerns related to author submission volume and content errors, this approach can strengthen conference proceedings and support broader efforts to enhance research quality and trust in scientific dissemination. Future efforts will focus on refining and customizing AI models and optimizing reviewer workflows to further enhance this process.</p>
<sup>1</sup>American Heart Association, Dallas, TX, US; <sup>2</sup>Cactus Communications Pvt Ltd, Mumbai, India; <sup>3</sup>Cactus Communications Inc, Princeton, NJ, US; chirag.patel@cactusglobal.com.</p>
<h4>Conflict of Interest Disclosures</h4>
<p> Heather Goodell, Christine Beaty, and Jonathan Schultz are employed by the American Heart Association. Shilpi Mehra and Chirag Jay Patel are employed by Cactus Communications, which owns Paperpal Preflight for Editorial Desk. No other disclosures were reported.</p>


Enhancing Research Integrity in Abstract Submissions With a Hybrid AI-Human Post-Review Process

<p>Ting Gao,<sup>1</sup> Xiuyuan Hao<sup>2</sup></p>
<h4>Objective</h4>
<p> Authorship confers credit and has important academic, social, and financial implications.<sup>1</sup> Although certain journals, such as <i>JAMA</i>, enforce strict authorship policies, the recognition of equal contributions by 2 or more co–first and/or co–corresponding authors is increasingly common in biomedical publishing.<sup>2,3</sup> Both the <i>Chinese Medical Journal</i> (<i>CMJ</i>) and <i>JAMA</i> are peer-reviewed general medical journals covering all major medical disciplines. This study evaluated the prevalence of co–first and co–corresponding authors in <i>CMJ</i> and <i>JAMA</i>.</p>
<h4>Design</h4>
<p> From January 1 to January 21, 2025, we examined all articles published in <i>CMJ</i> and <i>JAMA</i> in 2024, excluding News, Editors’ Notes, Corrigendum/Correction, Humanities, and miscellaneous sections, to assess the frequency of co–first and co–corresponding authorship. Other characteristics examined included article type, subject, and geographic location of corresponding authors. A multivariate logistic regression model was used to identify characteristics associated with co–first or co–corresponding authors.</p>
<h4>Results</h4>
<p> Of the 473 articles published in <i>CMJ</i> in 2024, 233 (49.3%) included co–first authors and 180 (38.1%) included co–corresponding authors. <i>JAMA</i> published 1011 articles in 2024; among them, 42 (4.2%) had co–first authors and 18 (1.8%) had co–corresponding authors. Original articles most frequently featured co–first and co–corresponding authors in both journals (<b>Table 25-0882</b>). In <i>CMJ</i>, gastroenterology was the most frequent subject associated with both forms of shared authorship. In <i>JAMA</i>, public health had the highest number of co–first authors, while cardiology had the most co–first and co–corresponding authors. In <i>CMJ</i>, most corresponding authors of articles with co–first or co–corresponding authorship were based in China. Logistic regression analysis revealed that article type, subject, and corresponding author’s geographic location were not significantly associated with shared authorship status in <i>CMJ</i>. By contrast, in <i>JAMA</i>, article type was significantly associated with both co–first and co–corresponding authorship, and corresponding author’s geographic location was significantly associated with co–corresponding authorship. <i>CMJ</i> showed a higher prevalence of co–first (χ² = 431.3; <i>P</i> &lt;.001) and co–corresponding (χ² = 363.6; <i>P</i> &lt; .001) authorship than <i>JAMA</i>.</p>

![](https://assets.underline.io/markdown_image/1/image/432ce7332177690f74ec1e349e31502a.png)

<h4>Conclusions</h4>
<p> This study demonstrates a significant difference in the prevalence of co–first and co–corresponding authorship between <i>CMJ</i> and <i>JAMA</i>.<i> </i>China and the United States are among the leading countries in scientific output. Compared with US-based authors, Chinese authors were more likely to designate co–first and co–corresponding authors. This trend may reflect the academic, social, and economic significance of shared authorship roles within the Chinese research evaluation system. While journals such as <i>JAMA</i> impose restrictions on shared authorship, increasing multidisciplinary collaborations and the rise of team science have led others, such as <i>CMJ</i>, to adopt more flexible policies. Establishing a standardized framework for managing co–first and co–corresponding authorship is needed.</p>
<h4>References</h4>
<p>1. International Committee of Medical Journal Editors. Recommendations for the conduct, reporting, editing, and publication of scholarly work in medical journals. Updated May 2023. Accessed January 18, 2025. <a href="https://www.icmje.org/recommendations/"><span class="Hyperlink CharOverride-6">https://www.icmje.org/recommendations/</span></a></p>
<p>2. Aakhus E, Mitra N, Lautenbach E, Joffe S. Gender and byline placement of co-first authors in clinical and basic science journals with high impact factors. <i>JAMA</i>. 2018;319(6): 610-611. doi:10.1001/jama.2017.18672</p>
<p>3. Hosseini M. Equal co-authorship practices: Review and recommendations. <i>Sci Eng Ethics</i>. 2020;26(3):1133-1148. doi:10.1007/s11948-020-00183-8</p>
<sup>1</sup>Editorial Department, <i>Chinese Medical Journal</i>, Chinese Medical Association Publishing House, Beijing, China; <sup>2</sup>Editorial Department, <i>Chinese Medical Journal</i>, Chinese Medical Association Publishing House, Beijing, China, haoxiuyuan@163.com.</p>
<h4>Conflict of Interest Disclosures</h4>
<p> None reported.</p>


Co-First authors and Co-Corresponding Authors in the Chinese Medical Journal and JAMA

<p>Cecilia Munguti,<sup>1</sup> James Kiilu,<sup>1</sup> James Kigera,<sup>1</sup> Michael Mwachiro<sup>1</sup></p>
<h4>Objective</h4>
<p> The International Committee of Medical Journal Editors (ICMJE) authorship criteria ensure appropriate credit for significant research contributions, while the Contributor Role Taxonomy (CRediT) framework offers a detailed account of individual scholarly contributions. Despite these frameworks, honorary authorship remains widespread in health sciences,<sup>1</sup> and surveys from Africa have documented its prevalence and the difficulty of applying authorship standards.<sup>2</sup> This study evaluated the fulfillment of ICMJE authorship criteria and CRediT contributions in manuscripts submitted between 2017 and 2023.</p>
<h4>Design</h4>
<p> This retrospective study analyzed the fulfillment of ICMJE authorship criteria and author-reported CRediT contributor roles for manuscripts submitted to <i>Annals of African Surgery</i>, a journal that endorses the ICMJE authorship criteria, between 2017 and 2023. The evaluated ICMJE criteria included criteria 1 (intellectual input) and 2 (manuscript drafting or critical revision) because these are the only author-level items captured as discrete checkboxes in the submission system. Criteria 3 and 4 were not assessed because they are captured as a collective attestation in a signed mandatory licence agreement required at submission. Data were retrieved from the journal’s online manuscript management system. CRediT roles were scored using a weighted system: 3 points for lead, 2 for equal, and 1 for supporting contributions across 14 roles, with a maximum possible score of 42 converted to a percentage. Descriptive statistics summarized ICMJE compliance and CRediT role distribution.</p>
<h4>Results</h4>
<p> A total of 1619 authors and 448 manuscripts with complete authorship data were evaluated. Among the authors, 1027 (63.4%) met both ICMJE criteria 1 and 2, 460 (28.4%) met only 1 criterion, and 132 (8.2%) met none. Of the 1619 authors, 709 (43.8%) were supervisors; among these supervisor authors, 523 (73.8%) met both criteria, 124 (17.5%) met only 1 criterion, and 62 (8.7%) failed to meet any. A total of 9610 CRediT contributor roles were analyzed. On weighted CRediT scores, 275 (17%) authors had a contribution score above 50%, while 971 (60%) scored 25% or less. Among 709 supervisors, 241 (34%) had scores above 50% and 269 (38%) scored 25% or less. Funding acquisition was credited to 296 authors (18.3%). Among the supervisor authors, 248 (35.0%) reported funding acquisition, accounting for 83.8% of all authors who claimed that role.</p>
<h4>Conclusions</h4>
<p> Using ICMJE criteria, supervisor involvement appeared to be limited, suggesting a high prevalence of guest authorship. However, weighted CRediT scores indicated that supervisors often contributed significantly, more so than other coauthors, highlighting a disconnect between traditional authorship criteria and actual contributions. Although limited to 1 journal, this study demonstrates the value of integrating authorship criteria audits with CRediT data, while noting the potential for checkbox behavior by submitting authors who may not grasp the significance of their declarations.</p>
<h4>References</h4>
<p>1. Meursinge Reynders RA, ter Riet G, Di Girolamo N, Cavagnetto D, Malički M. Honorary authorship is highly prevalent in health sciences: systematic review and meta-analysis of surveys. <i>Sci Rep</i>. 2024;14:4385. doi:10.1038/s41598-024-31526-2</p>
<p>2. Breet E, Botha J, Horn L, Swartz L. Academic and scientific authorship practices: a survey among South African researchers. <i>J Empir Res Hum Res Ethics</i>. 2018;13(5):412-420. doi:10.1177/1556264618789253</p>
<sup>1</sup><i>Annals of African Surgery</i>, Nairobi, Kenya, ceciliamunguti@annalsofafricansurgery.com.</p>
<h4>Conflict of Interest Disclosures</h4>
<p> James Kigera is a member of the Peer Review Congress Advisory Board but was not involved in the review or decision for this abstract. No other disclosures were reported.</p>
<h4>Additional Information</h4>
<p> James Kiilu is a co–corresponding author (jameskiilu@annalsofafricansurgery.com).</p>


Authorship and Contributorship Criteria and Practices at the Annals of African Surgery

<p>Mohammad Hosseini,<sup>1</sup> Sofie Elaine Adams,<sup>2</sup> Yensi Flores,<sup>3</sup> Kathleen Hall Jamieson,<sup>2</sup> Joerg Heber,<sup>4</sup> Jennifer Heimberg,<sup>5</sup><span lang="ar-SA"> Véronique Kiermer,</span><sup>6</sup> Arthur Lupia,<sup>7</sup> Ana Marušić,<sup>8</sup> Beau Nielsen,<sup>5</sup> Magdalena Skipper,<sup>9</sup> Geeta Swamy,<sup>10</sup> Susan Wolf<sup>11</sup></p>
<h4>Objective </h4>
<p>Although authorship is a cornerstone of the reward system of science, it is often fraught with disagreements and questionable practices.<sup>1</sup> A working group (including authors of this abstract and additional experts) was convened in January 2024 by the National Academy of Sciences (NAS) Strategic Council on Research Excellence, Integrity, and Trust to examine the impact of previous recommendations on institutional and journal authorship policies.<sup>1,2</sup></p>
<h4>Design </h4>
<p>Our analysis consisted of 2 parts completed in 2024 and 2025. First, we examined authorship policies of the AAAS, AGU, APS, <i>BMJ</i>, <i>Cell</i>, ICMJE, IEEE, JACS, <i>Nature</i>, <i>NEJM</i>, PLOS, <i>PNAS</i>, and SAGE (represented in the group members who authored the NAS 2018 recommendation<sup>2</sup>) to examine the extent to which they had adapted policies. This review captured trends regarding authorship guidelines, use of ORCIDs, and the CRediT taxonomy. Second, we searched the websites of all R1 (very high research activity) institutions (search terms <i>authorship</i>, <i>authorship guidelines</i>, <i>research guidelines</i>, and <i>research integrity</i>) in the US (n = 146) to determine whether credit and accountability or responsibility were mentioned in authorship guidelines and, if so, whether they were cast as related concepts. We then checked whether guidelines included links to external instructions (eg, ICMJE) and encouraged meetings to discuss authorship. Both phases were followed by brainstorming sessions among working group members.</p>
<h4>Results </h4>
<p>Of the examined journal groups, 5 adopted ICMJE guidelines, 2 referenced the NAS 2018 recommendations, and others had different recommendations. Seven groups recommended using the CRediT taxonomy, but only 3 required ORCIDs for all authors. Of the 146 R1 institutions, 59 included a credit-accountability link in their guidelines, but in only 38 cases did the credit-accountability link appear at the top of the guideline. Guidelines of 60 institutions suggested authors meet proactively to discuss authorship.</p>
<h4>Conclusions </h4>
<p>Anchoring authorship in credit (give credit where it is due) and accountability (authors are accountable for the integrity of the work) principles<sup>2</sup> readily explains why questionable practices (eg, gifts, ghost writing, and coercive authorship) are problematic and helps determine when authorship credit is warranted. Providing a clear description of contributions reinforces the application of these principles and fosters transparency as an important complementary principle.<sup>3</sup> While journals should choose authorship guidelines that suit their context, encouraging the use of ORCID and taxonomies such as CRediT are beneficial. Aligning institutional guidelines with the credit-accountability framework offers an opportunity to improve authorship practices. We plan to engage with institutional leaders to explore ways to support these improvements (eg, by offering educational materials or courses, providing mechanisms to resolve conflicts, and offering a safe environment for discussing authorship).</p>
<h4>References</h4>
<p>1. National Academies of Sciences, Engineering, and Medicine. <i>Fostering Integrity in Research</i>. 2017; National Academies Press.<a href="https://doi.org/10.17226/21896"><span class="Hyperlink CharOverride-6"> doi:10.17226/21896</span></a></p>
<p>2. McNutt MK, Bradford M, Drazen JM, et al. Transparency in authors’ contributions and responsibilities to promote integrity in scientific publication. <i>Proc Natl Acad Sci U S A</i>. 2018;115(11):2557-2560.<a href="https://doi.org/10.1073/pnas.1715374115"><span class="Hyperlink CharOverride-6"> doi:10.1073/pnas.1715374115</span></a></p>
<p>3. Allen L, Scott J, Brand A, Hlava M, Altman M. Publishing: credit where credit is due. <i>Nature.</i> 2014;508(7496):312-313.<a href="https://doi.org/10.1038/508312a"><span class="Hyperlink CharOverride-6"> doi:10.1038/508312a</span></a></p>
<sup>1</sup>Northwestern University, US, mohammad.hosseini@northwestern.edu; <sup>2</sup>University of Pennsylvania, US; <sup>3</sup>University College Cork, Ireland; <sup>4</sup>Lawrence Berkeley National Laboratory, US; <sup>5</sup>National Academy of Sciences, US; <sup>6</sup>PLOS, US; <sup>7</sup>University of Michigan, US; <sup>8</sup>University of Split, Croatia; <sup>9</sup>Nature, UK; <sup>10</sup>Duke University, US; <sup>11</sup>University of Minnesota, US.</p>
<h4>Conflict of Interest Disclosures </h4>
<p>The authors declare no conflicting interests. Véronique Kiermer and Ana Marušić are members of the Peer Review Congress Advisory Board but were not involved in the review or decision for this abstract.</p>
<h4>Funding/Support </h4>
<p>This work is funded by the National Academy of Sciences (NAS). Mohammad Hosseini is funded by the National Institutes of Health’s (NIH) National Center for Advancing Translational Sciences (UM1TR005121).</p>
<h4>Role of the Funder/Sponsor </h4>
<p>The funders have not played a role in the design, analysis, decision to publish, or preparation of the manuscript, and views expressed here do not represent the views and opinions of the NAS, NIH, or US government.</p>


Integration of Credit and Accountability Principles in Authorship Policies of Science Journals and Research Institutions

<p>Stephan D. Fihn,<sup>1,3</sup> Roy H. Perlis,<sup>2,3</sup> Jacob Kendall-Taylor,<sup>3</sup> Annette Flanagin<sup>3</sup></p>
<h4>Objective</h4>
<p> The major challenges for editors of journals include attracting submissions of high-quality manuscripts and recruiting reviewers.<sup>1</sup> In light of a dearth of relevant research, we sought to ascertain the association between publishing manuscripts as an author and participation in peer review for the same journal.</p>
<h4>Design</h4>
<p> We conducted a cross-sectional study using data extracted about authors and reviewers for <i>JAMA Network Open</i>, a high-volume open access general medical journal, for the years 2019 to 2023, including number of articles submitted and published as first or corresponding author during this period, numbers of requests to review and response (submitted, declined, no response), and quality of review submitted as determined by reviewing editor on a 5-point Likert scale (excellent = 1 to poor = 5). Data were analyzed using R 4.4.2.</p>
<h4>Results</h4>
<p> Among 38,683 invited reviewers, 30,082 (77.8%) submitted no manuscripts; 4014 (10.6%) submitted ≥1 article as first or corresponding author but had no published articles; and 4497 (11.6%) submitted ≥1 article as first or corresponding author and had ≥1 publication. <b>Table 25-1097</b> summarizes review requests to these individuals and compares their rates of accepting reviews and ratings among those who returned reviews. The mean (SD) number of invitations was greatest among published reviewers (5.1 [9.5]), followed by those with submissions but no publications (3.4 [6.1]), and then individuals with no submissions (2.7 [5.9]) (<i>P</i> &lt; .007 by analysis of variance). Reviewers with publications were significantly more likely to accept invitations and significantly less likely to ignore invitations compared with the other 2 groups (<i>P</i> &lt; .01 by analysis of variance and post hoc pairwise test). The absolute difference in proportion of accepted invitations was 26.2% (95% CI, 24.8%-27.6%; <i>P</i> &lt; .001) between published and never-submitted reviewers and 11.6% (95% CI, 9.7%-13.5%; <i>P</i> &lt; .001) between published and nonpublished submitters. Among those who returned reviews, quality (mean rating) was significantly higher among reviewers with publications (absolute difference from nonsubmitters: 0.25; 95% CI, −0.29 to −0.20; <i>P</i> &lt; .001). Differences between submitters without publications (mean difference: 0.03) and never-submitters (mean difference: 2.5%) were minimal for both rating metrics (<i>P</i> &gt; .50).</p>
<a href="/wp-content/uploads/2025_conference/tables/prc25-1097t1.png"> <img loading="lazy" src="/wp-content/uploads/2025_conference/tables/prc25-1097t1.png"> </a>
<h4>Conclusions </h4>
<p>Authors who published in <i>JAMA Network Open </i>were significantly more likely to accept an invitation to review even though they received nearly twice as many requests as those who did not publish. Moreover, the quality of their reviews was significantly greater. Individuals who submitted manuscripts but did not publish were intermediate between these 2 groups in terms of accepting invitations to reviews and quality of reviews. These results suggest that investigators may be willing to commit more of their academic effort to journals that publish their work. Limitations of this analysis include the inability to determine the relative timing of publication versus review; restriction of the analysis to a single high-volume journal; exclusion of co-authors; and limitation to a 4-year time frame.</p>
<h4>Reference</h4>
<p>1. Zupanc GKH. “It is becoming increasingly difficult to find reviewers”—myths and facts about peer review. J Comp Physiol A Neuroethol Sens Neural Behav Physiol. 2024;210(1):1-5. doi:10.1007/s00359-023-01642-w</p>
<sup>1</sup>Department of Medicine, University Washington, Seattle, WA, US; steve.fihn@jamanetwork.org; <sup>2</sup>Department of Psychiatry, Massachusetts General Hospital and Harvard Medical School, Boston, MA, US; <sup>3</sup>JAMA Network, Chicago, IL, US.</p>
<h4>Conflict of Interest Disclosures</h4>
<p> Stephan D. Fihn (SDF) and Roy H. Perlis (RHP) are paid as consultants for work as editors of JAMA Network Open (SDF and RHP) and JAMA+ AI (RHP). Jacob Kendall-Taylor and Annette Flanagin are on the editorial staff of the JAMA Network. Annette Flanagin is a member of the Peer Review Congress Advisory Board but was not involved in the review or decision for this abstract. None of the authors have any other conflicts of interest to disclose.</p>
<h4>Funding/Support </h4>
<p>The authors received no direct funding for this work apart from that described above. </p>


Next from The 10th Peer Review Congress

Accuracy and Precision of a Neural Network Author Name Disambiguator

Similar lecture

Perception of Open Science Practices on Reproducibility Among Reviewers of Grant Proposals at Research Funding Organizations