United States

&lt;p&gt;Peiling Wang,&lt;sup&gt;1&lt;/sup&gt; Dietmar Wolfram,&lt;sup&gt;2&lt;/sup&gt; Scott Shumate&lt;sup&gt;3&lt;/sup&gt;&lt;/p&gt;
&lt;h4&gt;Objective &lt;/h4&gt;
&lt;p&gt;Open peer review processes promote transparency and accountability. As more publishers adopt open peer review,&lt;sup&gt;1&lt;/sup&gt; accessible peer reviews become valuable corpora to develop AI tools for enhancing peer review processes.&lt;sup&gt;2&lt;/sup&gt; The main purpose of peer review as stated in the Guidelines for Reviewers&lt;sup&gt;3&lt;/sup&gt; is, “First, reviews provide constructive advice and recommendations to the authors on how their paper can be improved.” How can we determine the constructiveness of peer review reports? This exploratory study analyzed 11,321 peer review reports of the first version of 4100 manuscripts submitted to, and published in, &lt;i&gt;PeerJ&lt;/i&gt; to identify constructive comments. Specifically, this dataset includes first reviews from November 11, 2012, to November 24, 2019. This study examines 3 research questions: (1) To what extent do reviewers provide constructive comments? (2) To what extent do reviewers use hedging to soften criticisms or recommendations? (3) Are there differences in constructive comments between signed and anonymous reviews?&lt;/p&gt;
&lt;h4&gt;Design &lt;/h4&gt;
&lt;p&gt;The gpt-4o-2024-08-06 model from OpenAI API was used to analyze reviews at the sentence level to identify criticisms or supportive comments about different aspects of the research, including study purpose, hypotheses, methods, results, discussion, and conclusions. A detailed system prompt (&lt;a href=&quot;https://zenodo.org/records/15464322&quot;&gt;https://zenodo.org/records/15464322&lt;/a&gt;) was used with a temperature setting of 0. The model was instructed to categorize each sentence into a structured JSON output specifying section, aspects, evaluation, tone, and mode. Use of hedging tone includes phrases such as “I do not think the investigation was ‘rigorous’” or “has the potential to be much better and more relevant.” Building on the GPT analysis, we derived the following variables to measure review feature aspects, including (1) Pc, the percentage of critical sentences; (2) Ps, the percentage of supportive sentences; and (3) Ph, the percentage of hedging sentences. The constructive index (CI), considering both criticisms and supportive comments, is the percentage of sentences containing research comments.&lt;/p&gt;
&lt;h4&gt;Results &lt;/h4&gt;
&lt;p&gt;Of the 11,321 peer review reports, 7071 (62.5%) were anonymous and 4250 (37.5%) were signed (&lt;b&gt;Table 25-1055&lt;/b&gt;). Few reports were supportive (overall median of 9%). Higher levels (27% of anonymous reviews and 25% of signed reviews) contained criticisms of the research. A similar level of hedging was observed. Significant differences were observed between the 2 groups; signed reviews showed significantly fewer criticisms and higher levels of hedging.&lt;/p&gt;

![](https://assets.underline.io/markdown_image/1/image/c3c1db9d57bb2102d48d254f10bb4a4e.png)

&lt;h4&gt;Conclusions &lt;/h4&gt;
&lt;p&gt;We measured the extent of research-related comments in peer review reports that authors opted to publish. Anonymous reviewers tend to be more critical. The use of hedging in reviews could obscure clear suggestions on how to improve the manuscript. This research represents an initial effort towards a model for an AI-assisted peer review system. Further analysis should fine-tune the AI algorithms for analysis of reviewer comments. In addition, authors’ perspectives on what constitutes constructive feedback warrant further study. This research had limitations. Review histories were not available for rejected manuscripts or for articles where authors opted out of publishing reviews.&lt;/p&gt;
&lt;h4&gt;References&lt;/h4&gt;
&lt;p&gt;1. Wolfram D, Wang P, Hembree A, Park H. Open peer review: promoting transparency in open science. &lt;i&gt;Scientometrics. &lt;/i&gt;2020;125:1033-1051. doi:10.1007/s11192-020-03488-4&lt;/p&gt;
&lt;p&gt;2. Wolfram D, Wang P, Abuzahra F. An exploration of referees’ comments published in open peer review journals: the characteristics of review language and the association between review scrutiny and citations. &lt;i&gt;Res Eval. &lt;/i&gt;2021;30(3):314-322&lt;i&gt;. &lt;/i&gt;doi:10.1093/reseval/rvab005&lt;/p&gt;
&lt;p&gt;3. Technical Community on Real-Time Systems (TCRTS) of the IEEE Computer Society. Guidelines for reviewers. Accessed on May 29, 2025. &lt;a href=&quot;https://cmte.ieee.org/tcrts/guidelines-for-reviewers&quot;&gt;&lt;span class=&quot;Hyperlink CharOverride-6&quot;&gt;https://cmte.ieee.org/tcrts/guidelines-for-reviewers&lt;/span&gt;&lt;/a&gt;&lt;/p&gt;
&lt;sup&gt;1&lt;/sup&gt;University of Tennessee, Knoxville, US, peilingw@utk.edu; &lt;sup&gt;2&lt;/sup&gt;University of Wisconsin, Milwaukee, US; &lt;sup&gt;3&lt;/sup&gt;Austin Peay State University, Clarksville, TN, US.&lt;/p&gt;
&lt;h4&gt;Conflict of Interest Disclosures&lt;/h4&gt;
&lt;p&gt; None reported.&lt;/p&gt;


The 10th Peer Review Congress

An AI-assisted Analysis of Published PeerJ Open Peer Reviews

peer review process and models

open science

peer review

The Tenth Peer Review Congress featured 3 days for presentations of new research into peer review and all aspects of scientific publication, bias, quality of reporting, and information access and dissemination.

This year’s meeting was hybrid, and all plenary sessions were livestreamed with opportunities to view presentations and engage with participants in person and virtually during and after the meeting. All recordings are now available on Underline for viewing.

<div style="position:relative;padding-top:max(60%,324px);width:100%;height:0;"><iframe style="position:absolute;border:none;width:100%;height:100%;left:0;top:0;" src="https://online.fliphtml5.com/ebtyf/mget/"  seamless="seamless" scrolling="no" frameborder="0" allowtransparency="true" allowfullscreen="true" ></iframe></div>


#### Why? 
Our aim is to encourage research into the quality and credibility of peer review and scientific publication, to establish the evidence base on which scientists can improve the conduct, reporting, and dissemination of scientific research.

#### Who? 
Participants include editors and publishers of scientific peer-reviewed journals, researchers, funders, bibliometric and informatics experts, information innovators, librarians, journalists, policymakers, ethicists, scientific information disseminators, and anyone interested in the progress of the scientific information enterprise and the quality of scientific evidence.

The Congress embraces a wide range of disciplines, including biomedicine, health and life sciences, applied sciences, basic sciences, physical and chemical sciences, mathematics, computer sciences, engineering, economics, and social sciences as well as new and emerging disciplines.

#### What? 
The Congress provides a forum for the presentation and discussion of new research into the quality and credibility of editorial and funding peer review, scientific and scholarly publication, research methods and reporting practices, identification and management of biases, transparency, reproducibility, access, publication and dissemination metrics, identification and management of threats to the integrity of science, and advances in information exchange.

**Follow us**

[![](https://assets.underline.io/markdown_image/1/image/662a22766b9788ed3766cb23cd156e32.png)](https://x.com/peerrevcongress)  [![](https://assets.underline.io/markdown_image/1/image/bb105afb1773a01fa169aa97ca1af49a.png)](https://www.youtube.com/channel/UCsG_0nKi3_Zfyv0yRPdxjWQ)  [![](https://assets.underline.io/markdown_image/1/image/034e1e70a194a9a4178fdf470ef796db.png)](https://bsky.app/profile/peerreviewcongress.bsky.social)



You need to log in with the email address you registered with. 



Please log in to explore this event.

Registration is required to attend this event, please [here](https://peerreviewcongress.org/registration-information/) to secure your ticket. Please check your spam/promotional inbox too.



Please register!

The Tenth Peer Review Congress features 3 days for presentations of new research into peer review and all aspects of scientific publication, bias, quality of reporting, and information access and dissemination.

<p>Peiling Wang,<sup>1</sup> Dietmar Wolfram,<sup>2</sup> Scott Shumate<sup>3</sup></p>
<h4>Objective </h4>
<p>Open peer review processes promote transparency and accountability. As more publishers adopt open peer review,<sup>1</sup> accessible peer reviews become valuable corpora to develop AI tools for enhancing peer review processes.<sup>2</sup> The main purpose of peer review as stated in the Guidelines for Reviewers<sup>3</sup> is, “First, reviews provide constructive advice and recommendations to the authors on how their paper can be improved.” How can we determine the constructiveness of peer review reports? This exploratory study analyzed 11,321 peer review reports of the first version of 4100 manuscripts submitted to, and published in, <i>PeerJ</i> to identify constructive comments. Specifically, this dataset includes first reviews from November 11, 2012, to November 24, 2019. This study examines 3 research questions: (1) To what extent do reviewers provide constructive comments? (2) To what extent do reviewers use hedging to soften criticisms or recommendations? (3) Are there differences in constructive comments between signed and anonymous reviews?</p>
<h4>Design </h4>
<p>The gpt-4o-2024-08-06 model from OpenAI API was used to analyze reviews at the sentence level to identify criticisms or supportive comments about different aspects of the research, including study purpose, hypotheses, methods, results, discussion, and conclusions. A detailed system prompt (<a href="https://zenodo.org/records/15464322">https://zenodo.org/records/15464322</a>) was used with a temperature setting of 0. The model was instructed to categorize each sentence into a structured JSON output specifying section, aspects, evaluation, tone, and mode. Use of hedging tone includes phrases such as “I do not think the investigation was ‘rigorous’” or “has the potential to be much better and more relevant.” Building on the GPT analysis, we derived the following variables to measure review feature aspects, including (1) Pc, the percentage of critical sentences; (2) Ps, the percentage of supportive sentences; and (3) Ph, the percentage of hedging sentences. The constructive index (CI), considering both criticisms and supportive comments, is the percentage of sentences containing research comments.</p>
<h4>Results </h4>
<p>Of the 11,321 peer review reports, 7071 (62.5%) were anonymous and 4250 (37.5%) were signed (<b>Table 25-1055</b>). Few reports were supportive (overall median of 9%). Higher levels (27% of anonymous reviews and 25% of signed reviews) contained criticisms of the research. A similar level of hedging was observed. Significant differences were observed between the 2 groups; signed reviews showed significantly fewer criticisms and higher levels of hedging.</p>

![](https://assets.underline.io/markdown_image/1/image/c3c1db9d57bb2102d48d254f10bb4a4e.png)

<h4>Conclusions </h4>
<p>We measured the extent of research-related comments in peer review reports that authors opted to publish. Anonymous reviewers tend to be more critical. The use of hedging in reviews could obscure clear suggestions on how to improve the manuscript. This research represents an initial effort towards a model for an AI-assisted peer review system. Further analysis should fine-tune the AI algorithms for analysis of reviewer comments. In addition, authors’ perspectives on what constitutes constructive feedback warrant further study. This research had limitations. Review histories were not available for rejected manuscripts or for articles where authors opted out of publishing reviews.</p>
<h4>References</h4>
<p>1. Wolfram D, Wang P, Hembree A, Park H. Open peer review: promoting transparency in open science. <i>Scientometrics. </i>2020;125:1033-1051. doi:10.1007/s11192-020-03488-4</p>
<p>2. Wolfram D, Wang P, Abuzahra F. An exploration of referees’ comments published in open peer review journals: the characteristics of review language and the association between review scrutiny and citations. <i>Res Eval. </i>2021;30(3):314-322<i>. </i>doi:10.1093/reseval/rvab005</p>
<p>3. Technical Community on Real-Time Systems (TCRTS) of the IEEE Computer Society. Guidelines for reviewers. Accessed on May 29, 2025. <a href="https://cmte.ieee.org/tcrts/guidelines-for-reviewers"><span class="Hyperlink CharOverride-6">https://cmte.ieee.org/tcrts/guidelines-for-reviewers</span></a></p>
<sup>1</sup>University of Tennessee, Knoxville, US, peilingw@utk.edu; <sup>2</sup>University of Wisconsin, Milwaukee, US; <sup>3</sup>Austin Peay State University, Clarksville, TN, US.</p>
<h4>Conflict of Interest Disclosures</h4>
<p> None reported.</p>


poster

<p>Chirag J. Patel,<sup>1</sup> Arjun K. Manrai,<sup>1</sup> Randall J. Ellis,<sup>1</sup> John P. A. Ioannidis<sup>2</sup></p>
<h4>Background</h4>
<p> Large language models (LLMs) are increasingly used to synthesize biomedical evidence, yet it is unknown whether their inferred exposure-phenotype relationships mirror observational findings.<sup>1,2</sup> </p>
<h4>Objective</h4>
<p> To quantify concordance of exposure-phenotype empirical associations using National Health and Nutrition Examination Survey (NHANES) data with predictions from an LLM (ChatGPT 4o [OpenAI]).</p>
<h4>Methods</h4>
<p> We computed 115,056 exposure-phenotype partial correlations among 560 chemical, dietary, and clinical exposures and 268 phenotypes in 80,000 US adults (1999-2018 NHANES).<sup>3</sup> We then sampled 1500 pairs: 500 significant after Bonferroni correction, 500 significant only after false-discovery rate (FDR) correction, and 500 not significant—to fit within the LLM prompt window. An OpenAI-o1-aug24 model (temperature = 0) acting as a prompted “epidemiologist” was asked to predict the sign, magnitude, and <i>P</i> value of each pair and rated the evidence generated as high, trending, or borderline. Verifiability of LLM-cited references was assessed by estimating κ values and Spearman correlation.</p>
<h4>Results</h4>
<p> The LLM labeled 23 pairs as high evidence, 244 as trending evidence, and 1233 as borderline evidence. Sign concordance with NHANES was 73% for high-confidence pairs (κ = 0.49), 65% for Bonferroni pairs (κ = 0.25), 43% for FDR-only pairs (κ = 0.09), and 12% for borderline pairs (κ approximately 0) (<b>Figure 25-1123</b>). Random sign assignment yielded a κ of 0.0001. Concordance for correlation magnitudes was modest (Spearman ρ = 0.21). Permutation tests confirmed no greater-than-chance performance on nonsense data. Mechanistic explanations dominated high-confidence pairs; 41% of cited references (615 of 1500) were unverifiable.</p>

![](https://assets.underline.io/markdown_image/1/image/ca7fd3c645a90192139dbf1d062ea8bc.png)

<h4>Conclusions</h4>
<p> LLMs captured directionality for well-established exposure-phenotype associations but faltered on borderline signals, highlighting both the promise and current limits of LLM-based evidence synthesis in exposomic epidemiology.</p>
<h4>References</h4>
<p>1. Ioannidis JPA, Loy EY, Poulton R, Chia KS. Researching genetic versus nongenetic determinants of disease: a comparison and proposed unification. <i>Sci Transl Med.</i> 2009;1(7):7ps8. doi:10.1126/scitranslmed.3000247</p>
<p>2. Patel CJ, Ioannidis JPA. Studying the elusive environment in large scale. <i>JAMA</i>. 2014;311(21):2173-2174. doi:10.1001/jama.2014.4129</p>
<p>3. Patel CJ, Rehkopf DH, Leppert JT, et al. Systematic evaluation of environmental and behavioural factors associated with all-cause mortality in the United States National Health and Nutrition Examination Survey. <i>Int J Epidemiol</i>. 2013;42(6):1795-1810. doi:10.1093/ije/dyt208</p>
<sup>1</sup>Department of Biomedical Informatics, Harvard Medical School, Boston, MA, US, chirag@hms.harvard.edu; <sup>2</sup>Department of Medicine, Epidemiology and Population Health, Stanford University School of Medicine, Stanford, CA, US.</p>
<h4>Conflict of Interest Disclosures</h4>
<p> John P. A. Ioannidis is a member of the Peer Review Congress Advisory Board but was not involved in the review or decision for this abstract.</p>
<h4>Funding/Support</h4>
<p> This study was funded by grants R01ES032470 and U24ES036819 from the National Institutes of Environmental Health Sciences and grant R01DK137993 from the National Institute of Diabetes and Digestive and Kidney Diseases.</p>
<h4>Role of Funder/Sponsor</h4>
<p> The sponsors had no role in this study.</p>


Comparing Observational Exposure-Phenotype Correlations with Large Language Model Predictions

<p>Nicola Di Girolamo,<sup>1,2</sup> Reint Meursinge Reynders,<sup>3,4</sup> Ugo Di Girolamo<sup>5</sup></p>
<h4>Objective </h4>
<p>Consolidated Standard of Reporting of Trials (CONSORT) checklists help authors to adhere to reporting standards and allow editors and peer reviewers to ensure that critical information is reported in a randomized controlled trial (RCT) at a glance.<sup>1</sup> We pragmatically assessed the ability of 3 artificial intelligence (AI) large language model (LLM) chatbots provided with minimal instructions to properly complete a CONSORT checklist from manuscripts reporting RCTs before peer review in 2025.</p>
<h4>Design </h4>
<p>This cross-sectional study was registered on the Open Science Framework.<sup>2</sup> PDFs of manuscripts reporting RCTs before peer review that were consecutively published in <i>The BMJ</i> were extracted in reverse-chronological order for a total of 50 manuscripts. <i>The BMJ</i> granted permission for this study. In February 2025, each PDF was uploaded exactly as submitted by the authors to 3 LLMs (ChatGPT-4o, Gemini Advanced 2.0 Flash, and Claude 3.5 Sonnet), which were instructed to complete the CONSORT checklist in a table format,<sup>3</sup> including the reporting of the 37 CONSORT items and extraction of the text where the item was reported. The number of iterations and total time (seconds) required to obtain a visually acceptable table were recorded. Sensitivity, specificity, and accuracy for each LLM were calculated with human manual extraction as the reference standard. Manual extraction was performed by 1 operator (N.D.) with expertise in RCT methods. Generalized linear mixed models were built to evaluate the effect of PDF characteristics (number of pages, file size, and number of words) on the disagreements between AI and human assessment.</p>
<h4>Results </h4>
<p>Of the 1850 CONSORT items, ChatGPT provided decisions for 1776 (96.0%); 1159 of these were in agreement with human extraction (overall accuracy, 65.3% [95% CI, 63.0%-67.5%]; sensitivity, 61.3% [95% CI, 55.0%-67.3%]; specificity, 65.9% [95% CI, 63.5%-68.3%]). Gemini provided decisions for 1629 items (88.0%), of which 1521 were in agreement with human extraction (accuracy, 93.4% [95% CI, 92.1%-94.5%]; sensitivity, 97.6% [95% CI, 94.8%-99.1%]; specificity, 92.6% [95% CI, 91.1%-94.0%]). Claude provided decisions for 1480 items (80.0%), of which 1375 were in agreement with human extraction (accuracy, 92.9% [95% CI, 91.5%- 94.2%]; sensitivity, 56.1% [95% CI, 49.3%-62.8%]; specificity, 99.4% [95% CI, 98.8%-99.7%]). Accuracy of LLMs was particularly low for certain CONSORT items (<b>Figure 25-1152</b>). There were no significant associations between PDF characteristics and AI and human disagreements. Trial PDFs were a mean of 49 pages and 17,268 words. LLMs were able to compile visually acceptable CONSORT checklists on the first iteration in 76.7% (115 of 150) attempts: ChatGPT for 40 RCTs, Gemini for 36 RCTs, and Claude for 39 RCTs. Generation of a CONSORT checklist required a mean of 75 seconds, 38 seconds, and 47 seconds for ChatGPT, Gemini, and Claude, respectively (range, 16-249 seconds).</p>

![](https://assets.underline.io/markdown_image/1/image/39e42ee902f1b80f6a363027b6df1d1e.png)

<h4>Conclusions </h4>
<p>In early 2025, the LLMs Gemini and Claude, but not ChatGPT, were able to generate accurate CONSORT checklists from PDFs of manuscripts reporting RCTs before peer review in over 90% of cases, given minimal instructions and often in less than 1 minute.</p>
<h4>References<a id="_idTextAnchor008"></a><a id="_idTextAnchor009"></a></h4>
<p>1. Moher D, Jones A, Lepage L; CONSORT Group (Consolidated Standards for Reporting of Trials). Use of the CONSORT statement and quality of reports of randomized trials: a comparative before-and-after evaluation. <i>JAMA</i>. 2001;285(15):1992-1995. doi:10.1001/jama.285.15.1992</p>
<p>2. Di Girolamo N, Meursinge Reynders R, Di Girolamo U. Comparison of different AI large language models for extraction of CONSORT items from submitted randomized controlled trials before peer-review. Open Science Framework. Registered February 13, 2025. Accessed July 5, 2025. <a href="https://osf.io/5v6h2"><span class="Hyperlink CharOverride-6">https://osf.io/5v6h2</span></a></p>
<p>3. Schulz KF, Altman DG, Moher D; CONSORT Group. CONSORT 2010 Statement: updated guidelines for reporting parallel group randomised trials. <i>BMJ</i>. 2010;340:c332. doi:10.1136/bmj.c332</p>
<sup>1</sup>Department of Clinical Sciences, College of Veterinary Medicine, Cornell University, Ithaca, NY, US, nd374@cornell.edu; <sup>2</sup>Journal of Small Animal Practice, British Small Animal Veterinary Association, Gloucestershire, UK; <sup>3</sup>Department of Oral and Maxillofacial Surgery, Amsterdam University Medical Center, University of Amsterdam, Amsterdam, the Netherlands; <sup>4</sup>Private Practice of Orthodontics, Milan, Italy; <sup>5</sup>Compass, New York, NY, US.</p>
<h4>Conflict of Interest Disclosures </h4>
<p>Nicola Di Girolamo is editor in chief of 2 peer-reviewed journals, one published by Elsevier and one by Wiley. No other disclosures were reported.</p>


Pragmatic Assessment of Different AI Large Language Models for Extraction of CONSORT Items From Randomized Controlled Trials Before Peer Review

<p>Josh Major,<sup>1</sup> Kurt Mahnken,<sup>1</sup> Alec Young,<sup>1</sup> Cameron O’Brien,<sup>1</sup> Andrew V. Tran,<sup>1</sup> Patrick Crotty,<sup>1</sup> Alicia Ito Ford,<sup>1,2</sup> Matt Vassar<sup>1,2</sup></p>
<h4>Objective</h4>
<p> The integration of artificial intelligence (AI) in scientific research presents both opportunities and challenges, particularly concerning transparency, ethics, and reproducibility.<sup>1</sup> This study aimed to evaluate how leading orthopedic and sports medicine (OSM) journals address 2 distinct outcomes: (1) the extent to which AI is permitted in research and publication and (2) the degree to which OSM journals are endorsing AI-specific reporting guidelines (RGs). The primary outcome was to assess the presence of AI-related policies, including requirements for AI disclosure, acceptance or prohibition of AI-generated images and content, and AI’s role in manuscript writing and authorship. Secondary outcomes examined the endorsement of AI-specific RGs and the relationship between AI policies and Journal Impact Factor.</p>
<h4>Design</h4>
<p> We conducted a cross-sectional analysis of the top 100 OSM journals ranked by the 2023 SCImago Journal Rank (SJR) indicator. Eligibility criteria included actively publishing peer-reviewed clinical journals with publicly accessible Instructions for Authors in English language. On September 29, 2024, 2 investigators independently screened, extracted, and reconciled data in a masked, duplicate manner. Data were extracted from journals’ Instructions for Authors pages and affiliated publisher websites where indicated. If a journal had no mention of AI policies or AI-specific RGs, the editorial teams were contacted directly to inquire about these gaps. Biserial correlation analyses in R version 4.4.1 (The R Foundation) and RStudio (Posit) examined AI policy relationships with Journal Impact Factor and SJR.</p>
<h4>Results</h4>
<p> The initial search yielded 319 journals, from which the top 100 were analyzed. Of these, 94 mentioned AI in their Instructions for Authors, with all 94 requiring disclosure of AI use and prohibiting AI as an author (<b>Table 25-1175</b>). Additionally, AI-assisted manuscript writing was permitted in all cases, while 87% allowed AI-assisted content generation and 64% permitted AI-generated images. Despite this widespread recognition of AI policies, AI-specific RGs were endorsed in only 1% of journals, recommending the use of the Checklist for Artificial Intelligence in Medical Imaging (CLAIM) RG. No significant associations were observed between Journal Impact Factor and the presence of AI-related editorial policies. Additionally, no differences were found among the AI policies in the 4 quartiles of the OSM journals in this study.</p>
<a href="/wp-content/uploads/2025_conference/tables/prc25-1175t1.png"> <img loading="lazy" src="/wp-content/uploads/2025_conference/tables/prc25-1175t1.png"> </a>
<h4>Conclusions</h4>
<p> While some AI use in research is widely acknowledged by OSM journals, the lack of consistency in policies regarding AI-generated images and content suggest an area for improvement to standardize how AI can be applied in these journals. Furthermore, the lack of endorsement for AI-specific RGs suggests a critical gap in ensuring the transparency and methodological rigor of AI-integrated research. To address these issues, we recommend that OSM journals establish clear AI policies and endorse AI-specific RGs. The International Committee of Medical Journal Editors, Committee on Publication Ethics, and World Association of Medical Editors have taken initial steps; however, implementation may be difficult.<sup>2</sup> Implementing these measures will bridge existing gaps, promote transparency, and improve research quality in scientific publishing.</p>
<h4>References</h4>
<p>1. Prasana P, Mandal PK, Hussain D, et al. Intersection of orthopaedics and artificial intelligence: a review. <i>SSR Inst Int J Life Sci</i>. 2024;10(3):5544-5552. doi:10.21276/ssr-iijls.2024.10.3.21</p>
<p>2. Simera I, Moher D, Hoey J, Schulz KF, Altman DG. The EQUATOR Network and reporting guidelines: helping to achieve high standards in reporting health research studies. <i>Maturitas</i>. 2009;63(1):4-6. doi:10.1016/j.maturitas.2009.03.011</p>
<sup>1</sup>Office of Medical Student Research, Oklahoma State University Center for Health Sciences, Tulsa, OK, US, youngalec9.r@gmail.com; <sup>2</sup>Department of Psychiatry and Behavioral Sciences, Oklahoma State University Center for Health Sciences, Tulsa, OK, US.</p>
<h4>Conflict of Interest Disclosures</h4>
<p> Alicia Ito Ford reports receipt of funding from the Center for Integrative Research on Childhood Adversity, the Oklahoma Shared Clinical and Translational Resources, and internal grants from Oklahoma State University and Oklahoma State University Center for Health Sciences outside of the present work. Matt Vassar reports receipt of funding from the National Institute on Drug Abuse, the National Institute on Alcohol Abuse and Alcoholism, the US Office of Research Integrity, Oklahoma Center for Advancement of Science and Technology, and internal grants from Oklahoma State University Center for Health Sciences outside of the present work. No other disclosures were reported.</p>
<h4>Additional Information</h4>
<p> Andrew V. Tran is a co–corresponding author (andrewtranresearch@gmail.com).</p>


Artificial Intelligence Editorial Policies and Reporting Standards in Orthopedic and Sports Medicine Journals

<p>Heather Goodell,<sup>1</sup> Christine Beaty,<sup>1</sup> Jonathan Schultz,<sup>1</sup> Shilpi Mehra,<sup>2</sup> Chirag Jay Patel<sup>3</sup></p>
<h4>Objective</h4>
<p> The increase in abstract submissions to major scientific conferences requires efficient and reliable methods to ensure compliance with research integrity standards, which are essential for maintaining the credibility of science. Traditional review methods may not fully capture key issues, requiring a structured approach to evaluate submissions before final acceptance. This study presents a review process that uses artificial intelligence (AI) with human validation to assess the integrity of abstracts submitted to the American Heart Association (AHA) Scientific Sessions.</p>
<h4>Design</h4>
<p> An integrity evaluation process was applied by combining AI-driven analysis with human review of abstracts submitted for the 2024 AHA Scientific Sessions. An AI tool (Paperpal Preflight for Editorial Desk) was used to identify research integrity issues, including AI-generated text, author expertise misalignment, and reference accuracy. Subject matter experts then reviewed abstracts flagged for validation. AI checks categorized abstracts into <i>passed</i>, <i>warning</i>, or <i>critical</i> levels, with those not marked as <i>passed</i> requiring further human review.</p>
<h4>Results</h4>
<p> Among the 8477 submitted abstracts that were analyzed, 42 were flagged with a warning for integrity concerns (AI-generated text, author expertise misalignment, and reference accuracy); human reviewers cleared 3 of these. Further analysis revealed that 167 authors submitted 15 or more abstracts, 63 authors submitted 20 or more, and 13 authors submitted more than 30 abstracts. A total of 2624 abstracts had at least 1 author with 10 or more submissions, and 1438 abstracts had at least 1 author with 15 or more submissions. The number of submissions was considered when assessing research integrity concerns. Analysis of author submission counts helped flag questionable cases for closer review and the identification of potentially suspect activities.</p>
<h4>Conclusions</h4>
<p> The hybrid AI-human review process for abstract submissions was demonstrated to be an efficient and accurate evaluation method that should be scalable. By addressing concerns related to author submission volume and content errors, this approach can strengthen conference proceedings and support broader efforts to enhance research quality and trust in scientific dissemination. Future efforts will focus on refining and customizing AI models and optimizing reviewer workflows to further enhance this process.</p>
<sup>1</sup>American Heart Association, Dallas, TX, US; <sup>2</sup>Cactus Communications Pvt Ltd, Mumbai, India; <sup>3</sup>Cactus Communications Inc, Princeton, NJ, US; chirag.patel@cactusglobal.com.</p>
<h4>Conflict of Interest Disclosures</h4>
<p> Heather Goodell, Christine Beaty, and Jonathan Schultz are employed by the American Heart Association. Shilpi Mehra and Chirag Jay Patel are employed by Cactus Communications, which owns Paperpal Preflight for Editorial Desk. No other disclosures were reported.</p>


Enhancing Research Integrity in Abstract Submissions With a Hybrid AI-Human Post-Review Process

<p>Ting Gao,<sup>1</sup> Xiuyuan Hao<sup>2</sup></p>
<h4>Objective</h4>
<p> Authorship confers credit and has important academic, social, and financial implications.<sup>1</sup> Although certain journals, such as <i>JAMA</i>, enforce strict authorship policies, the recognition of equal contributions by 2 or more co–first and/or co–corresponding authors is increasingly common in biomedical publishing.<sup>2,3</sup> Both the <i>Chinese Medical Journal</i> (<i>CMJ</i>) and <i>JAMA</i> are peer-reviewed general medical journals covering all major medical disciplines. This study evaluated the prevalence of co–first and co–corresponding authors in <i>CMJ</i> and <i>JAMA</i>.</p>
<h4>Design</h4>
<p> From January 1 to January 21, 2025, we examined all articles published in <i>CMJ</i> and <i>JAMA</i> in 2024, excluding News, Editors’ Notes, Corrigendum/Correction, Humanities, and miscellaneous sections, to assess the frequency of co–first and co–corresponding authorship. Other characteristics examined included article type, subject, and geographic location of corresponding authors. A multivariate logistic regression model was used to identify characteristics associated with co–first or co–corresponding authors.</p>
<h4>Results</h4>
<p> Of the 473 articles published in <i>CMJ</i> in 2024, 233 (49.3%) included co–first authors and 180 (38.1%) included co–corresponding authors. <i>JAMA</i> published 1011 articles in 2024; among them, 42 (4.2%) had co–first authors and 18 (1.8%) had co–corresponding authors. Original articles most frequently featured co–first and co–corresponding authors in both journals (<b>Table 25-0882</b>). In <i>CMJ</i>, gastroenterology was the most frequent subject associated with both forms of shared authorship. In <i>JAMA</i>, public health had the highest number of co–first authors, while cardiology had the most co–first and co–corresponding authors. In <i>CMJ</i>, most corresponding authors of articles with co–first or co–corresponding authorship were based in China. Logistic regression analysis revealed that article type, subject, and corresponding author’s geographic location were not significantly associated with shared authorship status in <i>CMJ</i>. By contrast, in <i>JAMA</i>, article type was significantly associated with both co–first and co–corresponding authorship, and corresponding author’s geographic location was significantly associated with co–corresponding authorship. <i>CMJ</i> showed a higher prevalence of co–first (χ² = 431.3; <i>P</i> &lt;.001) and co–corresponding (χ² = 363.6; <i>P</i> &lt; .001) authorship than <i>JAMA</i>.</p>

![](https://assets.underline.io/markdown_image/1/image/432ce7332177690f74ec1e349e31502a.png)

<h4>Conclusions</h4>
<p> This study demonstrates a significant difference in the prevalence of co–first and co–corresponding authorship between <i>CMJ</i> and <i>JAMA</i>.<i> </i>China and the United States are among the leading countries in scientific output. Compared with US-based authors, Chinese authors were more likely to designate co–first and co–corresponding authors. This trend may reflect the academic, social, and economic significance of shared authorship roles within the Chinese research evaluation system. While journals such as <i>JAMA</i> impose restrictions on shared authorship, increasing multidisciplinary collaborations and the rise of team science have led others, such as <i>CMJ</i>, to adopt more flexible policies. Establishing a standardized framework for managing co–first and co–corresponding authorship is needed.</p>
<h4>References</h4>
<p>1. International Committee of Medical Journal Editors. Recommendations for the conduct, reporting, editing, and publication of scholarly work in medical journals. Updated May 2023. Accessed January 18, 2025. <a href="https://www.icmje.org/recommendations/"><span class="Hyperlink CharOverride-6">https://www.icmje.org/recommendations/</span></a></p>
<p>2. Aakhus E, Mitra N, Lautenbach E, Joffe S. Gender and byline placement of co-first authors in clinical and basic science journals with high impact factors. <i>JAMA</i>. 2018;319(6): 610-611. doi:10.1001/jama.2017.18672</p>
<p>3. Hosseini M. Equal co-authorship practices: Review and recommendations. <i>Sci Eng Ethics</i>. 2020;26(3):1133-1148. doi:10.1007/s11948-020-00183-8</p>
<sup>1</sup>Editorial Department, <i>Chinese Medical Journal</i>, Chinese Medical Association Publishing House, Beijing, China; <sup>2</sup>Editorial Department, <i>Chinese Medical Journal</i>, Chinese Medical Association Publishing House, Beijing, China, haoxiuyuan@163.com.</p>
<h4>Conflict of Interest Disclosures</h4>
<p> None reported.</p>


Co-First authors and Co-Corresponding Authors in the Chinese Medical Journal and JAMA

<p>Cecilia Munguti,<sup>1</sup> James Kiilu,<sup>1</sup> James Kigera,<sup>1</sup> Michael Mwachiro<sup>1</sup></p>
<h4>Objective</h4>
<p> The International Committee of Medical Journal Editors (ICMJE) authorship criteria ensure appropriate credit for significant research contributions, while the Contributor Role Taxonomy (CRediT) framework offers a detailed account of individual scholarly contributions. Despite these frameworks, honorary authorship remains widespread in health sciences,<sup>1</sup> and surveys from Africa have documented its prevalence and the difficulty of applying authorship standards.<sup>2</sup> This study evaluated the fulfillment of ICMJE authorship criteria and CRediT contributions in manuscripts submitted between 2017 and 2023.</p>
<h4>Design</h4>
<p> This retrospective study analyzed the fulfillment of ICMJE authorship criteria and author-reported CRediT contributor roles for manuscripts submitted to <i>Annals of African Surgery</i>, a journal that endorses the ICMJE authorship criteria, between 2017 and 2023. The evaluated ICMJE criteria included criteria 1 (intellectual input) and 2 (manuscript drafting or critical revision) because these are the only author-level items captured as discrete checkboxes in the submission system. Criteria 3 and 4 were not assessed because they are captured as a collective attestation in a signed mandatory licence agreement required at submission. Data were retrieved from the journal’s online manuscript management system. CRediT roles were scored using a weighted system: 3 points for lead, 2 for equal, and 1 for supporting contributions across 14 roles, with a maximum possible score of 42 converted to a percentage. Descriptive statistics summarized ICMJE compliance and CRediT role distribution.</p>
<h4>Results</h4>
<p> A total of 1619 authors and 448 manuscripts with complete authorship data were evaluated. Among the authors, 1027 (63.4%) met both ICMJE criteria 1 and 2, 460 (28.4%) met only 1 criterion, and 132 (8.2%) met none. Of the 1619 authors, 709 (43.8%) were supervisors; among these supervisor authors, 523 (73.8%) met both criteria, 124 (17.5%) met only 1 criterion, and 62 (8.7%) failed to meet any. A total of 9610 CRediT contributor roles were analyzed. On weighted CRediT scores, 275 (17%) authors had a contribution score above 50%, while 971 (60%) scored 25% or less. Among 709 supervisors, 241 (34%) had scores above 50% and 269 (38%) scored 25% or less. Funding acquisition was credited to 296 authors (18.3%). Among the supervisor authors, 248 (35.0%) reported funding acquisition, accounting for 83.8% of all authors who claimed that role.</p>
<h4>Conclusions</h4>
<p> Using ICMJE criteria, supervisor involvement appeared to be limited, suggesting a high prevalence of guest authorship. However, weighted CRediT scores indicated that supervisors often contributed significantly, more so than other coauthors, highlighting a disconnect between traditional authorship criteria and actual contributions. Although limited to 1 journal, this study demonstrates the value of integrating authorship criteria audits with CRediT data, while noting the potential for checkbox behavior by submitting authors who may not grasp the significance of their declarations.</p>
<h4>References</h4>
<p>1. Meursinge Reynders RA, ter Riet G, Di Girolamo N, Cavagnetto D, Malički M. Honorary authorship is highly prevalent in health sciences: systematic review and meta-analysis of surveys. <i>Sci Rep</i>. 2024;14:4385. doi:10.1038/s41598-024-31526-2</p>
<p>2. Breet E, Botha J, Horn L, Swartz L. Academic and scientific authorship practices: a survey among South African researchers. <i>J Empir Res Hum Res Ethics</i>. 2018;13(5):412-420. doi:10.1177/1556264618789253</p>
<sup>1</sup><i>Annals of African Surgery</i>, Nairobi, Kenya, ceciliamunguti@annalsofafricansurgery.com.</p>
<h4>Conflict of Interest Disclosures</h4>
<p> James Kigera is a member of the Peer Review Congress Advisory Board but was not involved in the review or decision for this abstract. No other disclosures were reported.</p>
<h4>Additional Information</h4>
<p> James Kiilu is a co–corresponding author (jameskiilu@annalsofafricansurgery.com).</p>


Authorship and Contributorship Criteria and Practices at the Annals of African Surgery

<p>Mohammad Hosseini,<sup>1</sup> Sofie Elaine Adams,<sup>2</sup> Yensi Flores,<sup>3</sup> Kathleen Hall Jamieson,<sup>2</sup> Joerg Heber,<sup>4</sup> Jennifer Heimberg,<sup>5</sup><span lang="ar-SA"> Véronique Kiermer,</span><sup>6</sup> Arthur Lupia,<sup>7</sup> Ana Marušić,<sup>8</sup> Beau Nielsen,<sup>5</sup> Magdalena Skipper,<sup>9</sup> Geeta Swamy,<sup>10</sup> Susan Wolf<sup>11</sup></p>
<h4>Objective </h4>
<p>Although authorship is a cornerstone of the reward system of science, it is often fraught with disagreements and questionable practices.<sup>1</sup> A working group (including authors of this abstract and additional experts) was convened in January 2024 by the National Academy of Sciences (NAS) Strategic Council on Research Excellence, Integrity, and Trust to examine the impact of previous recommendations on institutional and journal authorship policies.<sup>1,2</sup></p>
<h4>Design </h4>
<p>Our analysis consisted of 2 parts completed in 2024 and 2025. First, we examined authorship policies of the AAAS, AGU, APS, <i>BMJ</i>, <i>Cell</i>, ICMJE, IEEE, JACS, <i>Nature</i>, <i>NEJM</i>, PLOS, <i>PNAS</i>, and SAGE (represented in the group members who authored the NAS 2018 recommendation<sup>2</sup>) to examine the extent to which they had adapted policies. This review captured trends regarding authorship guidelines, use of ORCIDs, and the CRediT taxonomy. Second, we searched the websites of all R1 (very high research activity) institutions (search terms <i>authorship</i>, <i>authorship guidelines</i>, <i>research guidelines</i>, and <i>research integrity</i>) in the US (n = 146) to determine whether credit and accountability or responsibility were mentioned in authorship guidelines and, if so, whether they were cast as related concepts. We then checked whether guidelines included links to external instructions (eg, ICMJE) and encouraged meetings to discuss authorship. Both phases were followed by brainstorming sessions among working group members.</p>
<h4>Results </h4>
<p>Of the examined journal groups, 5 adopted ICMJE guidelines, 2 referenced the NAS 2018 recommendations, and others had different recommendations. Seven groups recommended using the CRediT taxonomy, but only 3 required ORCIDs for all authors. Of the 146 R1 institutions, 59 included a credit-accountability link in their guidelines, but in only 38 cases did the credit-accountability link appear at the top of the guideline. Guidelines of 60 institutions suggested authors meet proactively to discuss authorship.</p>
<h4>Conclusions </h4>
<p>Anchoring authorship in credit (give credit where it is due) and accountability (authors are accountable for the integrity of the work) principles<sup>2</sup> readily explains why questionable practices (eg, gifts, ghost writing, and coercive authorship) are problematic and helps determine when authorship credit is warranted. Providing a clear description of contributions reinforces the application of these principles and fosters transparency as an important complementary principle.<sup>3</sup> While journals should choose authorship guidelines that suit their context, encouraging the use of ORCID and taxonomies such as CRediT are beneficial. Aligning institutional guidelines with the credit-accountability framework offers an opportunity to improve authorship practices. We plan to engage with institutional leaders to explore ways to support these improvements (eg, by offering educational materials or courses, providing mechanisms to resolve conflicts, and offering a safe environment for discussing authorship).</p>
<h4>References</h4>
<p>1. National Academies of Sciences, Engineering, and Medicine. <i>Fostering Integrity in Research</i>. 2017; National Academies Press.<a href="https://doi.org/10.17226/21896"><span class="Hyperlink CharOverride-6"> doi:10.17226/21896</span></a></p>
<p>2. McNutt MK, Bradford M, Drazen JM, et al. Transparency in authors’ contributions and responsibilities to promote integrity in scientific publication. <i>Proc Natl Acad Sci U S A</i>. 2018;115(11):2557-2560.<a href="https://doi.org/10.1073/pnas.1715374115"><span class="Hyperlink CharOverride-6"> doi:10.1073/pnas.1715374115</span></a></p>
<p>3. Allen L, Scott J, Brand A, Hlava M, Altman M. Publishing: credit where credit is due. <i>Nature.</i> 2014;508(7496):312-313.<a href="https://doi.org/10.1038/508312a"><span class="Hyperlink CharOverride-6"> doi:10.1038/508312a</span></a></p>
<sup>1</sup>Northwestern University, US, mohammad.hosseini@northwestern.edu; <sup>2</sup>University of Pennsylvania, US; <sup>3</sup>University College Cork, Ireland; <sup>4</sup>Lawrence Berkeley National Laboratory, US; <sup>5</sup>National Academy of Sciences, US; <sup>6</sup>PLOS, US; <sup>7</sup>University of Michigan, US; <sup>8</sup>University of Split, Croatia; <sup>9</sup>Nature, UK; <sup>10</sup>Duke University, US; <sup>11</sup>University of Minnesota, US.</p>
<h4>Conflict of Interest Disclosures </h4>
<p>The authors declare no conflicting interests. Véronique Kiermer and Ana Marušić are members of the Peer Review Congress Advisory Board but were not involved in the review or decision for this abstract.</p>
<h4>Funding/Support </h4>
<p>This work is funded by the National Academy of Sciences (NAS). Mohammad Hosseini is funded by the National Institutes of Health’s (NIH) National Center for Advancing Translational Sciences (UM1TR005121).</p>
<h4>Role of the Funder/Sponsor </h4>
<p>The funders have not played a role in the design, analysis, decision to publish, or preparation of the manuscript, and views expressed here do not represent the views and opinions of the NAS, NIH, or US government.</p>


Integration of Credit and Accountability Principles in Authorship Policies of Science Journals and Research Institutions

<p>Stephan D. Fihn,<sup>1,3</sup> Roy H. Perlis,<sup>2,3</sup> Jacob Kendall-Taylor,<sup>3</sup> Annette Flanagin<sup>3</sup></p>
<h4>Objective</h4>
<p> The major challenges for editors of journals include attracting submissions of high-quality manuscripts and recruiting reviewers.<sup>1</sup> In light of a dearth of relevant research, we sought to ascertain the association between publishing manuscripts as an author and participation in peer review for the same journal.</p>
<h4>Design</h4>
<p> We conducted a cross-sectional study using data extracted about authors and reviewers for <i>JAMA Network Open</i>, a high-volume open access general medical journal, for the years 2019 to 2023, including number of articles submitted and published as first or corresponding author during this period, numbers of requests to review and response (submitted, declined, no response), and quality of review submitted as determined by reviewing editor on a 5-point Likert scale (excellent = 1 to poor = 5). Data were analyzed using R 4.4.2.</p>
<h4>Results</h4>
<p> Among 38,683 invited reviewers, 30,082 (77.8%) submitted no manuscripts; 4014 (10.6%) submitted ≥1 article as first or corresponding author but had no published articles; and 4497 (11.6%) submitted ≥1 article as first or corresponding author and had ≥1 publication. <b>Table 25-1097</b> summarizes review requests to these individuals and compares their rates of accepting reviews and ratings among those who returned reviews. The mean (SD) number of invitations was greatest among published reviewers (5.1 [9.5]), followed by those with submissions but no publications (3.4 [6.1]), and then individuals with no submissions (2.7 [5.9]) (<i>P</i> &lt; .007 by analysis of variance). Reviewers with publications were significantly more likely to accept invitations and significantly less likely to ignore invitations compared with the other 2 groups (<i>P</i> &lt; .01 by analysis of variance and post hoc pairwise test). The absolute difference in proportion of accepted invitations was 26.2% (95% CI, 24.8%-27.6%; <i>P</i> &lt; .001) between published and never-submitted reviewers and 11.6% (95% CI, 9.7%-13.5%; <i>P</i> &lt; .001) between published and nonpublished submitters. Among those who returned reviews, quality (mean rating) was significantly higher among reviewers with publications (absolute difference from nonsubmitters: 0.25; 95% CI, −0.29 to −0.20; <i>P</i> &lt; .001). Differences between submitters without publications (mean difference: 0.03) and never-submitters (mean difference: 2.5%) were minimal for both rating metrics (<i>P</i> &gt; .50).</p>
<a href="/wp-content/uploads/2025_conference/tables/prc25-1097t1.png"> <img loading="lazy" src="/wp-content/uploads/2025_conference/tables/prc25-1097t1.png"> </a>
<h4>Conclusions </h4>
<p>Authors who published in <i>JAMA Network Open </i>were significantly more likely to accept an invitation to review even though they received nearly twice as many requests as those who did not publish. Moreover, the quality of their reviews was significantly greater. Individuals who submitted manuscripts but did not publish were intermediate between these 2 groups in terms of accepting invitations to reviews and quality of reviews. These results suggest that investigators may be willing to commit more of their academic effort to journals that publish their work. Limitations of this analysis include the inability to determine the relative timing of publication versus review; restriction of the analysis to a single high-volume journal; exclusion of co-authors; and limitation to a 4-year time frame.</p>
<h4>Reference</h4>
<p>1. Zupanc GKH. “It is becoming increasingly difficult to find reviewers”—myths and facts about peer review. J Comp Physiol A Neuroethol Sens Neural Behav Physiol. 2024;210(1):1-5. doi:10.1007/s00359-023-01642-w</p>
<sup>1</sup>Department of Medicine, University Washington, Seattle, WA, US; steve.fihn@jamanetwork.org; <sup>2</sup>Department of Psychiatry, Massachusetts General Hospital and Harvard Medical School, Boston, MA, US; <sup>3</sup>JAMA Network, Chicago, IL, US.</p>
<h4>Conflict of Interest Disclosures</h4>
<p> Stephan D. Fihn (SDF) and Roy H. Perlis (RHP) are paid as consultants for work as editors of JAMA Network Open (SDF and RHP) and JAMA+ AI (RHP). Jacob Kendall-Taylor and Annette Flanagin are on the editorial staff of the JAMA Network. Annette Flanagin is a member of the Peer Review Congress Advisory Board but was not involved in the review or decision for this abstract. None of the authors have any other conflicts of interest to disclose.</p>
<h4>Funding/Support </h4>
<p>The authors received no direct funding for this work apart from that described above. </p>


Authors Who Publish in a Journal and Likelihood to Serve as Reviewers

<p>Sidney Xiang,<sup>1</sup> Daniel M. Romero,<sup>1,2,3</sup> Misha Teplitskiy<sup>1</sup></p>
<h4>Objective</h4>
<p> Prior literature on interdisciplinary research evaluation does not include prepublication manuscripts or account for differential evaluation over multiple dimensions of interdisciplinarity. Based on management science literature, we introduce 2 novel dimensions: topic interdisciplinarity, which may incur evaluation penalties by cutting across disciplinary standards and threatening symbolic boundaries, and knowledge-base interdisciplinarity, which may incur benefits by combining larger pools of nonredundant information. Evaluations may also depend on how these dimensions align with each other and the audience. Our study tested these hypotheses using a large-scale dataset of journal submissions.</p>
<h4>Design</h4>
<p> We performed a case-control study using administrative data from the Institute of Physics Publishing, a major STEM publisher. The data included 128,950 manuscripts (accepted and rejected) submitted to 62 physical science journals from 2018 to 2022. We calculated topic interdisciplinarity with concept tags of the manuscript’s associated OpenAlex record<sup>1</sup> and knowledge-base interdisciplinarity with concept tags of the manuscript’s references. For each type, we quantified interdisciplinarity using the integration index,<sup>2</sup> ranging from 0 to 1. Logistic regression was used to model the relationship between both types of interdisciplinarity and 2 evaluation outcomes: final decision and review positivity, controlling for journal, manuscript type, year, number of authors, number of authors’ prior citations, number of authors’ prior publications, and number of references. We retained observations with both interdisciplinarities and all covariates.</p>
<h4>Results</h4>
<p> The marginal effect of a 1-SD increase in knowledge-base interdisciplinarity was a 0.9 (0.7) percentage point higher acceptance (positive review) probability, while a 1-SD increase in topic interdisciplinarity corresponded to a 1.2 (0.4) percentage point lower acceptance (positive review) probability. Interactions revealed that high knowledge-base interdisciplinarity attenuated the negative relationship between topic interdisciplinarity and acceptance. We did not see a symmetric effect when high knowledge-base interdisciplinarity was accompanied by high topic interdisciplinarity. When disaggregating model results by journal interdisciplinarity according to the publisher’s 2022 product catalogue,<sup>3</sup> journals categorized by the publisher as interdisciplinary or multisubject had positive associations between both types of interdisciplinarity and evaluation outcomes, whereas monodisciplinary journals had negative associations. Our results were robust to the inclusion and transformation of manuscript and team covariates.</p>

![](https://assets.underline.io/markdown_image/1/image/3f8d7212732299f660aee5d48fd7f1cb.png)

<h4>Conclusions</h4>
<p> Our findings indicate that different dimensions of interdisciplinarity have different relationships to evaluation and that, for successful publication, authors must attend to alignment between types of interdisciplinarity within the manuscript and between the manuscript and journal audience. Limitations include the scope of journals in our dataset (STEM only) and the reliance on matching submissions to OpenAlex records.</p>
<h4>References</h4>
<p>1. Priem J, Piwowar H, Orr R. OpenAlex: a fully-open index of scholarly works, authors, venues, institutions, and concepts. arXiv. Preprint posted online May 4, 2022. doi:10.48550/ARXIV.2205.01833</p>
<p>2. Porter AL, Cohen AS, David Roessner J, Perreault M. Measuring researcher interdisciplinarity. Scientometrics. 2007;72(1):117-147. doi:10.1007/s11192-007-1700-5</p>
<p>3. IOP Publishing. Product catalogue 2022. <a href="https://ioppublishing.org/wp-content/uploads/2022/03/IOPP-Catalogue-2022.pdf"><span class="Hyperlink CharOverride-6">https://ioppublishing.org/wp-content/uploads/2022/03/IOPP-Catalogue-2022.pdf</span></a></p>
<sup>1</sup>University of Michigan School of Information, Ann Arbor, MI, US, tepl@umich.edu; <sup>2</sup>University of Michigan Department of Electrical Engineering and Computer Science, Ann Arbor, MI, US; <sup>3</sup>University of Michigan Center for the Study of Complex Systems, Ann Arbor, MI, US.</p>
<h4>Conflict of Interest Disclosures</h4>
<p> None reported.</p>


Topic and Knowledge-base Interdisciplinarity in Manuscripts Submitted to Physical Science Journals vs Editorial Decision and Reviewer Positivity

<p>Neil Millar,<sup>1</sup> Brian Budgell<sup>2</sup></p>
<h4>Objective</h4>
<p> Promotional language (hereafter, <i>hype</i>) in scientific writing has increased significantly over the past 40 years.<sup>1</sup> Analyses of grant proposals have found the incidence of hype to be positively associated with funding success<sup>2</sup>; however, evidence of causality is still lacking. Furthermore, an underpowered pilot study suggested that hype may bias clinicians’ perceptions and evaluations of evidence.<sup>3</sup> This study tested the hypothesis that students’ appraisal of biomedical research abstracts is influenced by hype.</p>
<h4>Design</h4>
<p> A double-blind randomized controlled trial was conducted to assess readers’ evaluation of hyped vs nonhyped abstracts. Four structured abstracts were selected from leading spinal care journals, controlling for length and absence of hype. Hyped abstracts were modified to include 6 common hype terms, such as “carefully designed” to highlight methodological rigor, “this is the first study to” to emphasize novelty, “convincing evidence” to amplify results, and “conducted by an experienced radiologist” to emphasize researcher competence. Inclusion criteria were that participants were second-year or higher chiropractic students recruited from institutions in the UK, US, and Canada. A blinded researcher randomly selected folders for students to evaluate. Participants rated abstracts on a 10-point scale, using 4 criteria: implementation likelihood, rigor, novelty, and researcher competence. A linear mixed-effects model was used to assess whether evaluations were influenced by hype, accounting for variability across participants and abstracts.</p>
<h4>Results</h4>
<p> This ongoing study began on April 2, 2024. As of January 2025, 36 students at 3 institutions had completed the rating task. Mixed-effects modeling, including interactions between condition and criterion, showed no significant differences between hyped and original abstracts across all evaluation criteria (estimate, 0.067; <i>P</i> &gt; .05) (<b>Figure 25-0927</b>). While novelty received consistently lower ratings (estimate, −1.479; <i>P</i> &lt; .001), the influence of hype did not vary significantly by criterion, as interaction terms were nonsignificant (eg, condition × novelty interaction [estimate, 0.517; <i>P</i> &gt; .05]), indicating minimal measurable effect of promotional language.</p>

![](https://assets.underline.io/markdown_image/1/image/ed072eece70a4477cfa47ac6438bf2df.png)

<h4>Conclusions </h4>
<p>Within this experimental paradigm, hype did not appear to bias readers’ evaluation of the scientific merit of the literature. These findings are discussed in relation to evidence-based medicine and bias on the part of researchers, reviewers, editors, and other stakeholders. Limitations of the study include the small sample size reported to date and restriction of the experimental texts to abstracts.</p>
<h4>References</h4>
<p>1. Millar N, Batalo B, Budgell B. Trends in the use of promotional language (hype) in National Institutes of Health funding opportunity announcements, 1992-2020. <i>JAMA Netw Open</i>. 2022;5(11):e2243221. doi:10.1001/jamanetworkopen.2022.43221</p>
<p>2. Qiu HS, Peng H, Fosse HB, Woodruff TK, Uzzi B. Use of promotional language in grant applications and grant success. <i>JAMA Netw Open</i>. 2024;7(12):e2448696. doi:10.1001/jamanetworkopen.2024.48696</p>
<p>3. Millar N, Budgell B. Impact of hype on clinicians’ evaluation of trials—a pilot study. <i>J Can Chiropr Assoc</i>. 2023;67(1):38-49.</p>
<sup>1</sup>Faculty of Engineering, Information and Systems, University of Tsukuba, Tsukuba, Ibaraki, Japan; <sup>2</sup>Canadian Memorial Chiropractic College, Toronto, Ontario, Canada, bbudgell@cmcc.ca.</p>
<h4>Conflict of Interest Disclosures </h4>
<p>None reported.</p>
<h4>Funding/Support </h4>
<p>This study was supported by grant 21K02919 from the Japan Society for the Promotion of Science.</p>
<h4>Role of the Funder/Sponsor </h4>
<p>The funding organization had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the abstract; and decision to submit the abstract for presentation.</p>


Next from The 10th Peer Review Congress

Comparing Observational Exposure-Phenotype Correlations with Large Language Model Predictions

Similar lecture

Peer Reviews of Peer Reviews: A Randomized Controlled Trial and Other Assessments