United States

&lt;p&gt;Stephan B. Bruns,&lt;sup&gt;1&lt;/sup&gt; Helmut Herwartz,&lt;sup&gt;2&lt;/sup&gt; John P. A. Ioannidis,&lt;sup&gt;3&lt;/sup&gt; Chris-Gabriel Islam,&lt;sup&gt;2&lt;/sup&gt; Fabian H. C. Raters&lt;sup&gt;2&lt;/sup&gt;&lt;/p&gt;
&lt;h4&gt;Objective &lt;/h4&gt;
&lt;p&gt;The economics literature predominantly reports statistical values in data tables. This study developed a tool for extraction of statistical values in tables to identify reporting errors and to analyze the association between journal data and code availability policies and article characteristics.&lt;/p&gt;
&lt;h4&gt;Design &lt;/h4&gt;
&lt;p&gt;DORIS (Diagnosis of Reporting Errors in Scraped Tables)&lt;sup&gt;1,2&lt;/sup&gt; automatically extracts statistical values (eg, coefficients and SEs) from tables via web scraping in R to extract tables from HTML articles and text mining in Python to interpret data. This analysis included the top 50 economics journals that provided articles in HTML from 1998 to 2016. Tests were divided into 2 categories: main (first 3 table rows from main tables) and nonmain (all other rows from main tables and other tables). We used a staggered difference-in-differences design to assess the association between data and code availability policies and article outcomes&lt;sup&gt;3&lt;/sup&gt;: reporting errors using a dummy for inconsistency altering significance level; statistical significance proxied by &lt;i&gt;z&lt;/i&gt;-value, presence of visual indicators of statistical significance, and type of statistical value reported; logarithmized number of tests per article, tables with tests, and a dummy for appendix position; and citations measured by the logarithmized citation count for the first 7 years.&lt;/p&gt;
&lt;h4&gt;Results &lt;/h4&gt;
&lt;p&gt;The study included 578,132 statistical tests (median: 27 per table) from 15,725 tables (median: 4 per article) in 3746 articles (median: 76 per journal) that were published in 31 journals. Based on a sample of 3068 statistical tests, DORIS had a false discovery rate of 1.2%. Analysis revealed that 547 of 3677 articles (14.8%) had at least 1 reporting error in main tests altering the statistical significance of findings. The errors were prevalently oriented toward lowering &lt;i&gt;P&lt;/i&gt; values and claiming statistical significance. Assessment of data and code availability policies included 21 journals and 535,838 tests. These policies were associated with slightly less emphasis on statistical significance (smaller z-values: −0.16 [90% CI, −0.33 to 0], &lt;i&gt;P&lt;/i&gt; = .10; fewer statistical significance symbols: −4 [90% CI, −6 to −2] percentage points [pp]; &lt;i&gt;P&lt;/i&gt; = .004; fewer tests reported with a focus on significance (&lt;i&gt;t&lt;/i&gt;, &lt;i&gt;z&lt;/i&gt;, or &lt;i&gt;P &lt;/i&gt;values instead of SEs and CIs): −8 [90% CI, −13 to −3] pp; &lt;i&gt;P&lt;/i&gt; = .004), more rigor in reporting (more tests: 11% [90% CI, −1% to 22%]; &lt;i&gt;P&lt;/i&gt; = .12; more tables: 15% [90% CI, 7% to 23%]; &lt;i&gt;P&lt;/i&gt; = .002; more appendix tables: 4 [90% CI, 2 to 7] pp; &lt;i&gt;P&lt;/i&gt; = .009), and fewer reporting errors in nonmain tests (−0.2 [90% CI, −0.4 to 0] pp; &lt;i&gt;P&lt;/i&gt; = .08), but there was no difference in citations (−9% [90% CI, −21% to 4%]; &lt;i&gt;P&lt;/i&gt; = .25).&lt;/p&gt;
&lt;h4&gt;Conclusions &lt;/h4&gt;
&lt;p&gt;DORIS may be used in the peer review process in the economics literature to improve article quality and to generate large scale data for future meta-research. Data and code availability policies may help improve the reliability of published economics research.&lt;/p&gt;
&lt;h4&gt;References&lt;/h4&gt;
&lt;p&gt;1. Better Papers. Home page. Accessed June 12, 2025. &lt;a href=&quot;https://betterpapers.org/%23sec-faqs&quot;&gt;&lt;span class=&quot;Hyperlink CharOverride-6&quot;&gt;https://betterpapers.org/#sec-faqs&lt;/span&gt;&lt;/a&gt;&lt;/p&gt;
&lt;p&gt;2. Bruns SB, Herwartz H, Ioannidis JPA, Islam C-G, Raters FHC. Statistical reporting errors in economics. &lt;i&gt;MetaArXiv&lt;/i&gt;. Preprint posted online September 01, 2023. doi:10.31222/osf.io/mbx62&lt;/p&gt;
&lt;p&gt;3. Borusyak, K, Jaravel, X, Spiess, J. Revisiting event-study designs: robust and efficient estimation. &lt;i&gt;Rev Econ Studies&lt;/i&gt;. 2024;91(6):3253-3285. doi:10.1093/restud/rdae007&lt;/p&gt;
&lt;sup&gt;1&lt;/sup&gt;Hasselt University, Hasselt, Belgium, stephan.b.bruns@gmail.com; &lt;sup&gt;2&lt;/sup&gt;Georg August University Göttingen, Göttingen, Germany; &lt;sup&gt;3&lt;/sup&gt;Meta-Research Innovation Center at Stanford (METRICS), Stanford, CA, US.&lt;/p&gt;
&lt;h4&gt;Conflict of Interest Disclosures&lt;/h4&gt;
&lt;p&gt; John P. A. Ioannidis is a member of the Peer Review Congress Advisory Board but was not involved in the review or decision for this abstract. No other disclosures were reported.&lt;/p&gt;
&lt;h4&gt;Funding/Support&lt;/h4&gt;
&lt;p&gt; Funding was received from the German Research Foundation (DFG) under the project “Replications in Empirical Economics: Necessity, Incentives and Impact,” with follow up project “Selective Reporting and the Evolving Research Landscape in Economics” (project number: 405039391) and from Hasselt University within the framework of BOF BILA (BOF21BL08).&lt;/p&gt;
&lt;h4&gt;Role of Funder/Sponsor&lt;/h4&gt;
&lt;p&gt; DFG and BOF BILA funding helped in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the abstract; and decision to submit the abstract for presentation.&lt;/p&gt;


The 10th Peer Review Congress

Automated Interpretation of Statistical Tables to Assess Reporting Errors and Associations With Open Science Policies in Economic Journals

errors and corrections

publication

open science

bias

The Tenth Peer Review Congress featured 3 days for presentations of new research into peer review and all aspects of scientific publication, bias, quality of reporting, and information access and dissemination.

This year’s meeting was hybrid, and all plenary sessions were livestreamed with opportunities to view presentations and engage with participants in person and virtually during and after the meeting. All recordings are now available on Underline for viewing.

<div style="position:relative;padding-top:max(60%,324px);width:100%;height:0;"><iframe style="position:absolute;border:none;width:100%;height:100%;left:0;top:0;" src="https://online.fliphtml5.com/ebtyf/mget/"  seamless="seamless" scrolling="no" frameborder="0" allowtransparency="true" allowfullscreen="true" ></iframe></div>


#### Why? 
Our aim is to encourage research into the quality and credibility of peer review and scientific publication, to establish the evidence base on which scientists can improve the conduct, reporting, and dissemination of scientific research.

#### Who? 
Participants include editors and publishers of scientific peer-reviewed journals, researchers, funders, bibliometric and informatics experts, information innovators, librarians, journalists, policymakers, ethicists, scientific information disseminators, and anyone interested in the progress of the scientific information enterprise and the quality of scientific evidence.

The Congress embraces a wide range of disciplines, including biomedicine, health and life sciences, applied sciences, basic sciences, physical and chemical sciences, mathematics, computer sciences, engineering, economics, and social sciences as well as new and emerging disciplines.

#### What? 
The Congress provides a forum for the presentation and discussion of new research into the quality and credibility of editorial and funding peer review, scientific and scholarly publication, research methods and reporting practices, identification and management of biases, transparency, reproducibility, access, publication and dissemination metrics, identification and management of threats to the integrity of science, and advances in information exchange.

**Follow us**

[![](https://assets.underline.io/markdown_image/1/image/662a22766b9788ed3766cb23cd156e32.png)](https://x.com/peerrevcongress)  [![](https://assets.underline.io/markdown_image/1/image/bb105afb1773a01fa169aa97ca1af49a.png)](https://www.youtube.com/channel/UCsG_0nKi3_Zfyv0yRPdxjWQ)  [![](https://assets.underline.io/markdown_image/1/image/034e1e70a194a9a4178fdf470ef796db.png)](https://bsky.app/profile/peerreviewcongress.bsky.social)



You need to log in with the email address you registered with. 



Please log in to explore this event.

Registration is required to attend this event, please [here](https://peerreviewcongress.org/registration-information/) to secure your ticket. Please check your spam/promotional inbox too.



Please register!

The Tenth Peer Review Congress features 3 days for presentations of new research into peer review and all aspects of scientific publication, bias, quality of reporting, and information access and dissemination.

<p>Stephan B. Bruns,<sup>1</sup> Helmut Herwartz,<sup>2</sup> John P. A. Ioannidis,<sup>3</sup> Chris-Gabriel Islam,<sup>2</sup> Fabian H. C. Raters<sup>2</sup></p>
<h4>Objective </h4>
<p>The economics literature predominantly reports statistical values in data tables. This study developed a tool for extraction of statistical values in tables to identify reporting errors and to analyze the association between journal data and code availability policies and article characteristics.</p>
<h4>Design </h4>
<p>DORIS (Diagnosis of Reporting Errors in Scraped Tables)<sup>1,2</sup> automatically extracts statistical values (eg, coefficients and SEs) from tables via web scraping in R to extract tables from HTML articles and text mining in Python to interpret data. This analysis included the top 50 economics journals that provided articles in HTML from 1998 to 2016. Tests were divided into 2 categories: main (first 3 table rows from main tables) and nonmain (all other rows from main tables and other tables). We used a staggered difference-in-differences design to assess the association between data and code availability policies and article outcomes<sup>3</sup>: reporting errors using a dummy for inconsistency altering significance level; statistical significance proxied by <i>z</i>-value, presence of visual indicators of statistical significance, and type of statistical value reported; logarithmized number of tests per article, tables with tests, and a dummy for appendix position; and citations measured by the logarithmized citation count for the first 7 years.</p>
<h4>Results </h4>
<p>The study included 578,132 statistical tests (median: 27 per table) from 15,725 tables (median: 4 per article) in 3746 articles (median: 76 per journal) that were published in 31 journals. Based on a sample of 3068 statistical tests, DORIS had a false discovery rate of 1.2%. Analysis revealed that 547 of 3677 articles (14.8%) had at least 1 reporting error in main tests altering the statistical significance of findings. The errors were prevalently oriented toward lowering <i>P</i> values and claiming statistical significance. Assessment of data and code availability policies included 21 journals and 535,838 tests. These policies were associated with slightly less emphasis on statistical significance (smaller z-values: −0.16 [90% CI, −0.33 to 0], <i>P</i> = .10; fewer statistical significance symbols: −4 [90% CI, −6 to −2] percentage points [pp]; <i>P</i> = .004; fewer tests reported with a focus on significance (<i>t</i>, <i>z</i>, or <i>P </i>values instead of SEs and CIs): −8 [90% CI, −13 to −3] pp; <i>P</i> = .004), more rigor in reporting (more tests: 11% [90% CI, −1% to 22%]; <i>P</i> = .12; more tables: 15% [90% CI, 7% to 23%]; <i>P</i> = .002; more appendix tables: 4 [90% CI, 2 to 7] pp; <i>P</i> = .009), and fewer reporting errors in nonmain tests (−0.2 [90% CI, −0.4 to 0] pp; <i>P</i> = .08), but there was no difference in citations (−9% [90% CI, −21% to 4%]; <i>P</i> = .25).</p>
<h4>Conclusions </h4>
<p>DORIS may be used in the peer review process in the economics literature to improve article quality and to generate large scale data for future meta-research. Data and code availability policies may help improve the reliability of published economics research.</p>
<h4>References</h4>
<p>1. Better Papers. Home page. Accessed June 12, 2025. <a href="https://betterpapers.org/%23sec-faqs"><span class="Hyperlink CharOverride-6">https://betterpapers.org/#sec-faqs</span></a></p>
<p>2. Bruns SB, Herwartz H, Ioannidis JPA, Islam C-G, Raters FHC. Statistical reporting errors in economics. <i>MetaArXiv</i>. Preprint posted online September 01, 2023. doi:10.31222/osf.io/mbx62</p>
<p>3. Borusyak, K, Jaravel, X, Spiess, J. Revisiting event-study designs: robust and efficient estimation. <i>Rev Econ Studies</i>. 2024;91(6):3253-3285. doi:10.1093/restud/rdae007</p>
<sup>1</sup>Hasselt University, Hasselt, Belgium, stephan.b.bruns@gmail.com; <sup>2</sup>Georg August University Göttingen, Göttingen, Germany; <sup>3</sup>Meta-Research Innovation Center at Stanford (METRICS), Stanford, CA, US.</p>
<h4>Conflict of Interest Disclosures</h4>
<p> John P. A. Ioannidis is a member of the Peer Review Congress Advisory Board but was not involved in the review or decision for this abstract. No other disclosures were reported.</p>
<h4>Funding/Support</h4>
<p> Funding was received from the German Research Foundation (DFG) under the project “Replications in Empirical Economics: Necessity, Incentives and Impact,” with follow up project “Selective Reporting and the Evolving Research Landscape in Economics” (project number: 405039391) and from Hasselt University within the framework of BOF BILA (BOF21BL08).</p>
<h4>Role of Funder/Sponsor</h4>
<p> DFG and BOF BILA funding helped in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the abstract; and decision to submit the abstract for presentation.</p>


poster

<p>Constant Vinatier,<sup>1</sup> Ayu Putu Madri Dewi,<sup>2</sup> Gwénaël Dumont,<sup>1</sup>Tracey Weissgerber,<sup>3</sup>Vladislav Nachev,<sup>3</sup>Gowri Gopalakrishna,<sup>2,3,4</sup> Maud Scheidecker,<sup>1</sup> François-Joseph Arnault,<sup>1</sup> Nicholas J. DeVito,<sup>5</sup> Guillaume Freyermuth,<sup>6</sup> Mathieu Acher,<sup>6,10</sup> Gauthier Le Bartz Lyan,<sup>6</sup> Inge Stegeman,<sup>7,8</sup>Mariska M. G. Leeflang,<sup>2</sup>F. Naudet<sup>9,10</sup></p>
<h4>Objective </h4>
<p>Despite open science policies in major biomedical journals, adherence remains uncertain. This study evaluated automated tools, from regular expressions to large language models (LLMs), for assessing core open science practices in leading biomedical journals.</p>
<h4>Design </h4>
<p>We retrospectively assessed research articles from a sample of 10 major generalist medical journals (<i>Annals of-Internal Medicine</i>, <i>BMJ</i>, <i>BMC Medicine</i>, <i>Canadian Medical Association Journal</i> [<i>CMAJ</i>], <i>JAMA</i>, <i>JAMA</i> <i>Network Open</i>, <i>Lancet</i>, <i>Nature Medicine</i>, <i>New England Journal of Medicine</i>, and <i>PLoS Medicine</i>) from 2020 to 2023. Articles were retrieved via PubMed using a Peer Review of Electronic Search Strategies (PRESS) search strategy. The database comprised random samples of 103 randomized controlled trials (RCTs), 98 meta-analyses (MAs), and 111 other research articles (RAs). We evaluated 13 open science practices, including study registration, data sharing, and protocol sharing (open access or upon request). Each article was evaluated by 2 independent raters, with any disagreements resolved by a third rater. Seven different automated tools—rtransparent, oddpub, ctRegistries, ContriBot, DataSeer, SciScore, and an LLM (Llama 3-70B)—were used. Diagnostic accuracies were estimated using sensitivities, specificities, F1 scores, and LR+ and LR-.</p>
<h4>Results </h4>
<p>Manual extraction in the 312 articles identified registration in 98% (101/103) of RCTs, 69% (68/98) of MAs, and 18% (20/111) of RAs. Open data were present in 6% (6/103) of RCTs, 36% (35/98) of MAs, and 13% (15/111) of RAs and accessible upon request in 78% (80/103), 41% (40/98), and 59% (66/111), respectively. Protocols were openly available in 84% (87/103) of RCTs, 64% (63/98) of MAs, and 20% (22/111) of RAs and accessible upon request in 5% (5/103), 3% (3/98), and 3% (3/111), respectively. The accuracy of automated tools varied depending on the practice evaluated, with F1 scores ranging from 1.00 (Conflict of Interest statement, rtransparent) to 0.16 (SciScore, registration). For study registration, a simple tool using regular expressions, such as rtransparent, demonstrated good sensitivity (77%; 95% CI, 70%-83%) and high specificity (93%; 95% CI, 88%-97%). Data sharing detection remained challenging; for instance, rtransparent detects data sharing with a sensitivity of 74% (95% CI, 68%-80%) and a specificity of 59% (95% CI, 46%-70%). Different diagnostic accuracies were observed depending on the type of research and the journal, likely due to different formatting standards. All results are shown in <b>Table 25-1025</b>. Limitations include the declarative nature of some practices (eg, data sharing).</p>

![](https://assets.underline.io/markdown_image/1/image/f6bf4287ea9763381f2096129dbea4e8.png)

<h4>Conclusions </h4>
<p>Our study provides a detailed description of core open science practices across leading biomedical journals. It also highlights current challenges regarding the accuracy of automated tools in detecting these practices. While these tools likely provide valuable insights into overall practices, it is crucial to remain aware of the potential ranking biases introduced by these tools, as well as their limitations in providing detailed feedback for individual studies.</p>
<sup>1</sup>Univ Rennes, Inserm, EHESP, Irset (Institut de recherche en santé, environnement et travail), UMR_S 1085, Rennes, France, constant.vinatier1@gmail.com; <sup>2</sup>Department of Epidemiology and Data Science, Amsterdam University Medical Centers, Amsterdam, the Netherlands; <sup>3</sup>QUEST Center for Responsible Research, Berlin Institute of Health at Charité–Universitätsmedizin Berlin, Berlin, Germany; <sup>4</sup>Department of Epidemiology, Faculty of Health, Medicine, and Life Sciences, Maastricht University, Maastricht, the Netherlands; <sup>5</sup>Bennett Institute for Applied Data Science, Nuffield Department of Primary Care Health Sciences, University of Oxford, Oxford, UK; <sup>6</sup>Univ Rennes, IRISA, Inria, CNRS, Rennes, France; <sup>7</sup>Department of Otorhinolaryngology and Head and Neck Surgery, University Medical Center Utrecht, Utrecht, the Netherlands; <sup>8</sup>Brain Center, University Medical Center Utrecht, Utrecht, the Netherlands; <sup>9</sup>Univ Rennes, CHU Rennes, Inserm, EHESP, Irset (Institut de recherche en santé, environnement et travail), UMR_S 1085, Rennes, France; <sup>10</sup>Institut Universitaire de France (IUF), France.</p>
<h4>Conflict of Interest Disclosures</h4>
<p> None reported.</p>
<h4>Funding/Support</h4>
<p> As part of the OSIRIS project, this work was supported by the European Union’s Horizon Europe Research and Innovation Program under grant agreement number 101094725. Constant Vinatier, Ayu Putu Madri Dewi, Gowri Gopalakrishna, Nicholas J. DeVito, Inge Stegeman, Mariska M. G. Leeflang, and <span lang="fi-FI">F. Naudet</span> are members of this project.</p>
<h4>Role of the Funder/Sponsor</h4>
<p> The funder had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.</p>


Detection of Open Science Practices in Major Medical Journals: A Survey and Diagnostic Accuracy of Automatic Tools Using Sensitivity and Specificity

<p>Luka Ursić,<sup>1,2</sup> Nensi Bralić,<sup>1</sup> Giovanni Spitale,<sup>2</sup> Federico Germani,<sup>2</sup> Ana Marušić<sup>1</sup></p>
<h4>Objective</h4>
<p> Divisive discourse observed in news coverage or social media posts by researchers and the general public during the H1N1 and COVID-19 pandemics contributed to polarization and politicization, rapid shifts in sentiment, and concerning rates of misinformation.<sup>1-3</sup> We aimed to study the sentiment and natural language processing (NLP) analysis of editorial material in medical journals discussing the health care system response to the 2 pandemics.</p>
<h4>Design</h4>
<p> Using a systematic review design, we searched Web of Science and PubMed up to May 22, 2024, for editorials, viewpoints, and similar opinion pieces discussing the following segments of the health care system response to the 2 pandemics: nonpharmaceutical interventions; misinformation or disinformation; health care resource allocation and management; health care system preparedness; mandates, policies, and guidance related to research, education, or new technologies; and lessons learned. We limited our search to 2009 to 2014 for the H1N1 pandemic and 2019 to 2024 for the COVID-19 pandemic. Three researchers screened the titles, abstracts, and full texts of the retrieved records for eligibility in pairs, resolving discrepancies through discussion. We extracted the text from the included records’ PDFs using Python and performed a sentiment analysis using the Language Inquiry and Word Count 2022 software. We also used NLP approaches (flat lemmatization, convoluted lemmatization, rule-based autocoding, and cosine similarity) to determine the most frequently used lemmas in the 2 groups of editorials and explore topics based on their co-occurrence. We conducted inferential analyses at the level of the full sample and across specific categories (ie, stratified by pandemic, author country, and context of response). While we could not set a hypothesis for the exploratory NLP analysis, we hypothesized that in terms of sentiment, the editorials published during the COVID-19 pandemic would have higher scores for negative tone and emotion, personal and person-centered language, certitude, and all-or-none thinking and lower scores for analytical thinking.</p>
<h4>Results </h4>
<p>Following the screening process (<b>Figure 25-0979</b>), we included 2954 editorials in the final dataset. We performed a preliminary analysis of 200 editorials: 25 for the H1N1 pandemic and 175 for the COVID-19 pandemic. In contrast to our initial hypothesis, the editorials for the H1N1 pandemic had higher values for negative tone (median, 2.21 [IQR, 1.74-3.07] vs 1.69 [IQR, 1.22-2.31]; <i>P</i> = .007) and certitude (median, 0.39 [IQR, 0.16-0.66] vs 0.23 [IQR, 0.12-0.37]; <i>P</i> = .01). There were no differences in negative emotion, analytical thinking, or all-or-none thinking.</p>

![](https://assets.underline.io/markdown_image/1/image/d658c3c3e26c29b586c4a178cb5264eb.png)

<h4>Conclusions </h4>
<p>In this preliminary analysis, the editorials for the H1N1 pandemic contained more words connoting negative tone and certitude than those for the COVID-19 pandemic. These findings suggest that the researchers authoring these editorials expressed a more negative sentiment toward the public health response to the H1N1 pandemic and that they were seemingly more certain about their position.</p>
<h4>References</h4>
<p>1. Schmidt H. Pandemics and politics: analyzing the politicization and polarization of pandemic-related reporting. <i>Newsp Res J</i>. 2023;44(1):26-52. doi:10.1177/07395329221095850</p>
<p>2. Lwin MO, Lu J, Sheldenkar A, et al. Global sentiments surrounding the COVID-19 pandemic on Twitter: analysis of Twitter trends. <i>JMIR Public Health Surveill</i>. 2020;6(2):e19447. doi:10.2196/19447</p>
<p>3. Sule S, DaCosta MC, DeCou E, Gilson C, Wallace K, Goff SL. Communication of COVID-19 misinformation on social media by physicians in the US. <i>JAMA Netw Open</i>. 2023;6(8):e2328928. doi:10.1001/jamanetworkopen.2023.28928</p>
<sup>1</sup>Department of Research in Biomedicine and Health, University of Split School of Medicine, Split, Croatia, luka.ursic@mefst.hr; <sup>2</sup>Institute of Biomedical Ethics and History of Medicine, University of Zürich, Zürich, Switzerland.</p>
<h4>Conflict of Interest Disclosures</h4>
<p> Ana Marušić is a member of the Peer Review Congress Advisory Board but was not involved in the review or decision for this abstract. No other disclosures were reported.</p>
<h4>Funding/Support </h4>
<p>This study was funded by the Croatian Science Foundation under grant agreement IP-2019-04-4882 and the University of Zürich Institute of Biomedical Ethics and History of Medicine under a Stehr-Boldt fellowship given to Luka Ursić.</p>
<h4>Role of the Funder/Sponsor</h4>
<p> The funders had no role in the design of this study, its execution, analyses, interpretation of the data, or decision to submit results.</p>


Analysis of Editorials on the Response to the H1N1 and COVID-19 Pandemics

<p>Caitlin J. Bakker,<sup>1,2</sup> Erin E. Reardon,<sup>3</sup> Sarah Jane Brown,<sup>4</sup> Nicole Theis-Mahon,<sup>4</sup> Sara Schroter,<sup>5,6</sup> Lex Bouter,<sup>7,8</sup> Maurice P. Zeegers<sup>2</sup></p>
<h4>Objective </h4>
<p>The scientific community responded rapidly to COVID-19, producing over 200,000 publications in 1 year.<sup>1</sup> This speed brought challenges, including a higher retraction rate.<sup>2</sup> While retraction helps correct the scientific record, retracted status may be inconsistently presented and notices may be incomplete.<sup>3</sup> During a pandemic, inconsistency and incompleteness can have immediate and long-term health impacts. We evaluated retraction presentation consistency and notice completeness for COVID-19 vs non–COVID-19 publications. We describe reasons for retraction and time from publication to retraction.</p>
<h4>Design </h4>
<p>In March 2023, we retrieved retracted publications categorized as research articles or clinical studies in the subject area of public health and safety from Retraction Watch. A previous study focused on all retracted publications in this period<sup>3</sup>; this is a sub-study comparing COVID-19 with non–COVID-19 publications. Between April 28 and June 6, 2023, we assessed consistency in 11 databases (Academia.edu, CINAHL, Embase.com, Ovid Embase, Ovid Medline, PubMed, ResearchGate, SciHub, Scopus, Web of Science, and publisher websites) using 12 criteria from the International Committee of Medical Journal Editors and National Library of Medicine and notice completeness using 17 criteria from Retraction Watch and the Committee on Publication Ethics. Criteria were scored 0 if missing or 1 if present; partial scores were assigned when evaluating multicomponent criteria, such as bidirectional links. To ensure consistent scoring, a random subset of 21% (92 of 441) of retracted publications were independently reviewed by 2 researchers. Each researcher extracted data using Qualtrics forms, and scoring discrepancies were resolved by consensus. Following this calibration phase, remaining publications were extracted by a single reviewer. Kruskal-Wallis tests assessed differences in scores between COVID-19 and non–COVID-19 publications. </p>
<h4>Results</h4>
<p> Of 441 publications, 47 were about COVID-19 and 394 were not. COVID-19 publications were published between 2019 and 2022, while non-COVID 19 publications were published between 1978 and 2022. COVID-19 publications were most frequently retracted due to concerns about reliability of data or results (15 [31.9%]) compared with plagiarism (82 [20.8%]) for non–COVID-19 publications. The median time between publication and retraction was 120 (IQR, 15-196) days for COVID-19 publications and 326 (IQR, 124-789) days for non–COVID-19 publications (<i>P</i> &lt; .001). Across 11 databases, 41.2% (110 of 267) of records retrieved for retracted COVID-19 publications were marked as retracted compared with 47.6% (1225 of 2574) for non–COVID-19 publications. There was no statistically significant difference between consistency or completeness scores for COVID-19 vs non–COVID-19 retracted publications (<b>Table 25-1062</b>). No publications met all criteria.</p>

![](https://assets.underline.io/markdown_image/1/image/edc419e41ec09f04154b15739dea0699.png)

<h4>Conclusions </h4>
<p>Incomplete and inconsistent information poses challenges for researchers and practitioners, undermining trust in scientific literature. We found no association between publications being about COVID-19 and the consistency or completeness of retraction information; however, publications about COVID-19 appeared to be retracted more quickly.</p>
<h4>References</h4>
<p>1. Shimray SR. Research done wrong: a comprehensive investigation of retracted publications in COVID-19. <i>Account Res</i>. 2022;30(7):393-406. doi:10.1080/08989621.2021.2014327</p>
<p>2. Yeo-Teh NSL, Tang BL. An alarming retraction rate for scientific publications on coronavirus disease 2019 (COVID-19). <i>Account Res</i>. 2020;28(1):47-53. doi:10.1080/08989621.2020.1782203</p>
<p>3. Bakker CJ, Reardon EE, Brown SJ, et al. Identification of retracted publications and completeness of retraction notices in public health. <i>J Clin Epidemiol</i>. 2024;173:111427. doi:10.1016/j.jclinepi.2024.111427</p>
<sup>1</sup>University of Regina, Regina, Saskatchewan, Canada, caitlin.bakker@uregina.ca; <sup>2</sup>Maastricht University, Maastricht, the Netherlands; <sup>3</sup>Emory University, Atlanta, GA, US; <sup>4</sup>University of Minnesota, Minneapolis, MN, US; <sup>5</sup><i>BMJ</i>, London, UK; <sup>6</sup>London School of Hygiene and Tropical Medicine, London, UK; <sup>7</sup>Amsterdam University Medical Center, Amsterdam, the Netherlands; <sup>8</sup>Vrije Universiteit Amsterdam, Amsterdam, the Netherlands.</p>
<h4>Conflict of Interest Disclosures</h4>
<p> Caitlin J. Bakker is cochair of the National Information Standards Organization Communication of Retractions, Removals and Expressions of Concern Standing Committee. Lex Bouter is a member of the Peer Review Congress Advisory Board but was not involved in the review or decision for this abstract. No other disclosures were reported.</p>
<h4>Funding/Support</h4>
<p> This research is part of an ongoing PhD collaboration between The BMJ (British Medical Journal) and the team Meta-Research at Maastricht University (UM) on the responsible conduct of publishing scientific research. The BMJ is published by BMJ Group, a wholly owned subsidiary of the British Medical Association. UM is a public legal entity in the Netherlands. This study is part of Caitlin Bakker’s self-funded BMJ/UM PhD. No exchange of funds has taken place for this research project.</p>
<h4>Role of the Funder/Sponsor</h4>
<p> The authors are wholly responsible for the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the abstract; and decision to submit the abstract for presentation.</p>
<h4>Disclaimer</h4>
<p> All authors express their own opinions and not necessarily that of their employers.</p>


Consistency and Completeness of Retractions in Public Health Research on COVID-19

<p>Annie Whamond,<sup>1</sup> Adrian G. Barnett,<sup>2</sup> Jennifer A. Byrne<sup>1,3</sup></p>
<h4>Objective</h4>
<p> Paper mills are unethical organizations that provide low-value or fraudulent content to client authors.<sup>1</sup> Although image manipulation and incorrect reagent detection tools are available,<sup>1</sup> these approaches can require reviewers and readers to learn new skills. Scaled manuscript production through templates may also result in paper mill articles showing other common features. We aimed to help readers identify possible indicators of paper mill support by simply scanning article text and references. We focused on discussion sections for this study.</p>
<h4>Design</h4>
<p> We used an exploratory, cross-sectional design<sup>2</sup> to develop a descriptive analysis of discussion text and references in high Impact Factor (IF) cancer journals, defined as IF of 7 or higher for journal categories oncology, biochemistry and molecular biology, or cell biology. We downloaded the Retraction Watch database on October 16, 2024, and filtered for retraction reason <i>paper mill</i>, subject <i>cancer</i>, and journals on our high IF list. To identify comparison article cohorts, we randomly sampled original cancer articles in (1) the same 8 journals as retracted articles or (2) 8 independent journals with no paper mill retractions and high IF maintained for 20 years or longer. For each article, we recorded the percentages of references first cited in the introduction, methods, results, or discussion section. Focusing on discussion sections, we classified each sentence as providing background, summary, comparison, interpretation or implication, limitation, or future direction<sup>3</sup> and recorded whether individual discussion sentences cited new references. </p>
<h4>Results </h4>
<p>We found 22 retracted paper mill articles from 8 journals published between June 21, 2016, and June 16, 2022. Both comparison groups (50 articles each) were restricted to articles published between January 1, 2016, and October 31, 2024. Articles in all 3 groups included similar numbers of total references per article (median [IQR]: 42 [36-46] retracted articles; 47 [36-55] articles in same journals; 51 [45-60] articles in independent journals) and percentages of discussion sentences per article (median [IQR]: 34 [30-38] retracted articles; 40 [30-48] same journals; 34 [26-43] independent journals). Analyses of discussion sections indicated that retracted paper mill articles included higher percentages of references that were first cited in the discussion and higher percentages of discussion sentences that described background information and cited new references (<b>Figure 25-1022</b>).</p>

![](https://assets.underline.io/markdown_image/1/image/d20ae00ed6b6328fe12ed01f027e066e.png)

<h4>Conclusions</h4>
<p> Our analyses suggest that some discussion sections in retracted paper mill cancer research articles reiterate background information that is supported by new and possibly superfluous references. While recognizing that genuine studies also cite new references in discussion sections, superficial and redundant second introductions in discussion sections could help readers identify potentially problematic articles in high IF cancer journals, particularly when combined with other features of paper mill support.<sup>1</sup></p>
<h4>References</h4>
<p>1. Byrne JA, Abalkina A, Akinduro-Aje O, et al. A call for research to address the threat of paper mills. <i>PLoS Biol</i>. 2024;22(11):e3002931. doi:10.1371/journal.pbio.3002931</p>
<p>2. Kesmodel US. Cross-sectional studies—what are they good for? <i>Acta Obstet Gynecol Scand</i>. 2018;97(4):388-393. doi:10.1111/aogs.13331</p>
<p>3. Toronto CE, Remington R. Discussion and conclusion. In: <i>A Step-by-Step Guide to Conducting an Integrative Review</i>. Toronto CE, Remington R, eds. Springer International; 2020:71-84.</p>
<sup>1</sup>School of Medical Sciences, Faculty of Medicine and Health, University of Sydney, Sydney, New South Wales, Australia, jennifer.byrne@health.nsw.gov.au; <sup>2</sup>School of Public Health and Social Work, Queensland University of Technology, Kelvin Grove, Queensland, Australia; <sup>3</sup>NSW Health Statewide Biobank, NSW Health Pathology, Camperdown, New South Wales, Australia.</p>
<h4>Conflict of Interest Disclosures</h4>
<p> None reported.</p>
<h4>Funding/Support</h4>
<p> Adrian G. Barnett and Jennifer A. Byrne acknowledge grant funding from the National Health and Medical Research Council of Australia (ideas grant APP2029249). This grant supports Annie Whamond’s PhD candidature.</p>
<h4>Role of Funder/Sponsor</h4>
<p> The funding body played no role in the study design, data collection, management, analysis, or interpretation, and will play no role in the writing of any report, or the decision to submit the report for publication.</p>


Analysis of Cancer Research Discussion Text and References in High-Impact Factor Journals for Possible Indicators of Paper Mills

<p>Alexandre Clausse,<sup>1</sup> Guillaume Cabanac,<sup>1,2</sup> Pascal Cuxac,<sup>3</sup> Cyril Labbé<sup>4</sup></p>
<h4>Objective</h4>
<p> The Problematic Paper Screener (PPS) features 10 detectors to identify problematic articles in the scientific record.<sup>1</sup> The detectors search through metadata, references, or textual contents. The tortured phrases<sup>2</sup> detector uses a list of more than 7000 identified suspect expressions (<b>Table 25-1159</b>) to query a database (Dimensions; Digital Science) with approximately 130 million scientific articles.<sup>3</sup> The PPS displays the screening results on a public website. In this qualitative study, we introduce an alternative screening algorithm based on regular expressions (regex) and benchmark its effectiveness. Publishers are welcome to use this stand-alone algorithm that does not require a subscription to the database that we used. </p>

![](https://assets.underline.io/markdown_image/1/image/8b16462ed041c6b4f82712aec9be879f.png)

<h4>Design</h4>
<p> As of May 2025, we designed an algorithm that uses regex based on the PPS fingerprints list in the database to capture different forms of tortured phrases, such as “man-made consciousness,” “profound learning AND deep learning,” and “128 pieces”~5’ [128 bits] (ie, a tortured phrase with components in a 5-word sliding window). This approach is independent from search engines, as it matches fingerprints against the textual content of each article, ignoring their metadata, figure and table contents (including captions), and references. We benchmarked this new approach on the Hindawi extensible markup language (XML) corpus, which contains several hundred articles with tortured phrases, focusing on articles published from 2020 to 2022—the period with the most PPS-flagged articles (n <i>= </i>3400).</p>
<h4>Results</h4>
<p> The regex-based algorithm flagged 2455 problematic articles, with 1948 also flagged by PPS, yielding a 58% overlap. After removing duplicates, 401 articles flagged by the PPS were missing from the Hindawi XML corpus, amounting to 3371 problematic articles in 139 different journals. The evaluation is ongoing; we analyzed the top 200 results and found 48 false-positive results due to database indexing issues related to special characters and figures and PPS querying issues. We also found 52 false-negative results due to fingerprints list issues and because we excluded figures, tables, and references. Overall, we obtained 100 true-positive results, as both detectors (ie, PPS and regex-based algorithm) extracted expressions on the fingerprints list from the same articles.</p>
<h4>Conclusions</h4>
<p> The regex-based algorithm yielded results comparable to the current PPS screening process. Other configurations should be tested, including figures, tables, and references in the screening, as tortured phrases can also appear there. We invite publishers to use this regex-based approach to screen incoming manuscripts. We noticed several querying issues with the database that we used, and some fingerprints need to be redesigned. We will investigate further the validation assessment, as there are still 3351 articles to be reassessed. This research could contribute to improving the regex and updating the PPS fingerprints list, while offering detailed feedback and bug reports to the database developer.</p>
<h4>References</h4>
<p>1. Cabanac G, Labbé C, Magazinov A. The ‘Problematic Paper Screener’ automatically selects suspect publications for post-publication (re)assessment. <i>arXiv</i>. Preprint posted online October 7, 2022. doi:10.48550/arXiv.2210.04895</p>
<p>2. Cabanac G, Labbé C, Magazinov A. Tortured phrases: A dubious writing style emerging in science—evidence of critical issues affecting established journals. <i>arXiv</i>. Preprint posted online July 12, 2021. doi:10.48550/arXiv.2107.06751</p>
<p>3. Herzog C, Hook D, Konkiel S. Bringing down barriers between scientometricians and data. <i>Quantitative Science Studies</i>. 2020;1:387-395. doi:10.1162/qss_a_00020</p>
<sup>1</sup>Université de Toulouse, IRIT (UMR 5505), Toulouse, France, alexandre.clausse@irit.fr; <sup>2</sup>Institut Universitaire de France, Paris, France; <sup>3</sup>INIST CNRS, UAR 67, Vandoeuvre-lès-Nancy, France; <sup>4</sup>Université Grenoble Alpes, CNRS, Grenoble INP, LIG UMR 5217, Grenoble, France.</p>
<h4>Conflicts of Interest Disclosures </h4>
<p>Guillaume Cabanac is the administrator of the Problematic Paper Screener, a public platform that uses metadata from Digital Science and PubPeer via no-cost agreements. Guillaume Cabanac and Cyril Labbé have been in touch with most of the major publishers and their integrity officers, offering pro bono consulting regarding detection tools to various actors in the field, including Clear Skies, Morressier, River Valley, Signals, and STM.</p>
<h4>Funding/Support </h4>
<p>Alexandre Clausse, Guillaume Cabanac, and Cyril Labbé received funding from the European Research Council. Guillaume Cabanac received funding from the Institut Universitaire de France.</p>


Screening articles for tortured phrases with a regex-based detector

<p>Melissa L. Rethlefsen,<sup>1</sup> Carrie Price,<sup>2</sup> Sara Schroter<sup>3</sup></p>
<h4>Objective</h4>
<p> To explore the perspectives of librarians and information specialists (LISs) who participated in a randomized controlled trial of the effect of LIS involvement on reporting quality of systematic reviews (SRs).<sup>1</sup></p>
<h4>Design</h4>
<p> We surveyed LISs who completed a peer review of an SR in a trial conducted in <i>BMJ</i>, <i>BMJ Open</i>, and <i>BMJ Medicine</i> from January 3, 2023, to January 2, 2024. LISs were not told they were peer reviewing manuscripts as part of a trial but were sent invitations to review as part of the usual process. The questionnaire sought to understand their experience, what aspects of manuscripts they focused on, perceived impact on editorial decision-making and authors’ revisions, and willingness to peer review again. To better understand factors that might impact decisions to review again, we contacted 27 respondents to participate in a focus group concentrating on facilitators and barriers to peer reviewing SRs. </p>
<h4>Results</h4>
<p> Of the 88 LISs invited to participate in the survey, 70 (79.5%) responded. Most respondents had 6 or more years of experience as an LIS (67/70; 95.7%) and advising researchers on doing SRs (55/70; 78.6%) and had peer reviewed for a journal prior to the study (57/70; 81.4%). Most focused on the search and SR methods when reviewing but also commented on aspects such as research question formulation, plagiarism, and study results and conclusions. Two-thirds (44/66; 66.7%) believed they impacted editors’ decision-making and 59.1% (39/66) believed they impacted the authors’ revisions. Only 3 factors were considered extremely or very likely to impact their decision to review again: their schedule and/or lack of time, review turnaround time, and their sense of professional duty (<b>Table 24-0830</b>). Seventeen of 27 invited LISs (63.0%) participated in a focus group. Time was the primary barrier identified in the focus groups, followed by a sense of intimidation. LISs reported being motivated by feeling valued by editors, the enjoyment of peer reviewing, the desire to improve SR quality, and peer review as a learning experience. Several expressed surprise and delight at being asked to peer review for the journals.</p>

![](https://assets.underline.io/markdown_image/1/image/c26ed0c4df287a1d953b680e49fbcda6.png)

<h4>Conclusions</h4>
<p> A select sample of highly engaged LIS respondents believed they made a difference through their peer reviews and said they were very likely to agree to peer review in the future. LISs may be an underused peer reviewing resource with methodological experience that can help editors make decisions and improve the quality of SRs.</p>
<h4>Reference</h4>
<p>1. Rethlefsen ML, Schroter S, Bouter LM, et al. Improving peer review of systematic reviews and related review types by involving librarians and information specialists as methodological peer reviewers: a randomised controlled trial. <i>BMJ Evid Based Med</i>. Published online March 11, 2025. doi:10.1136/bmjebm-2024-113527</p>
<sup>1</sup>Health Sciences Library &amp; Informatics Center, University of New Mexico, Albuquerque, NM, US, mlrethlefsen@gmail.com; <sup>2</sup>ToxStrategies, A Blue Ridge Life Sciences Company, Baltimore, MD, US; <sup>3</sup>BMJ Publishing Group, London, UK.</p>
<h4>Conflict of Interest Disclosures </h4>
<p>This study follows up on a study that was conducted as part of Melissa L. Rethlefsen’s self-funded PhD project registered at Maastricht University, the Netherlands, in collaboration with BMJ Publishing Group. Carrie Price was employed at the National Institutes of Health Library at the time of this research. Sara Schroter is a full-time employee of BMJ Publishing Group but is not involved in decision-making on individual research submissions.</p>


Librarian and Information Specialist Perceptions of Peer Reviewing Systematic Reviews

<p><span lang="fr-FR">Abdelghani Maddi,¹ Ahmad Yaman Abdin,²</span><sup>,</sup><span lang="ar-SA">³ Francesco De Pretis⁴</span><sup>,</sup><span lang="fr-FR">⁵</span></p>

<h4>Objective</h4>
<p> To evaluate, using an empirically calibrated, agent-based model (ABM), whether a structured reviewer-training intervention is associated with improved peer review quality and reviewer engagement and reduced positive outcome bias across single- and multijournal settings. Existing randomized and quasi-experimental trials of peer review interventions are limited in scope and effect size,¹ and evolutionary models suggest competition among journals shapes reviewer effort.²</p>
<h4>Design </h4>
<p>We built an ABM populated by authors, reviewers, and editors. Calibration used the PeerRead International Conference on Learning Representations (ICLR)&#160;2017-2019 corpus (8151 reviews, 4905 submissions).³ Simulations followed the STRESS-ABM recommendations. Single-journal scenarios assumed 2500 submissions per year, while multijournal portfolios comprised 5 titles handling 10,000 combined submissions. The intervention increased the share of reviewers completing a certified online training module from 0% to 40%. Main outcomes were (1) review quality score (1-10 ordinal), (2) reviewer engagement (report length, number of words), and (3) positive outcome bias (difference in acceptance probability between high- and low-significance manuscripts). Each scenario used 10,000 Monte-Carlo replications; 95% confidence intervals were obtained with percentile bootstrap.</p>
<h4>Results </h4>
<p>In the single-journal baseline, mean review quality score was 5.52 (95%&#160;CI,&#160;5.49-5.55), and mean reviewer engagement was 604&#160;words (95% CI, 598-610); positive outcome bias was 0.33 (95% CI, 0.32-0.35) (<b>Table 25-0961</b>). With training, mean scores increased to 5.80 (95% CI, 5.77-5.83; Δ&#160;+0.28), engagement to 682&#160;words (95% CI, 675-689; Δ&#160;+78 words), and bias decreased to 0.26 (95% CI, 0.25-0.28; Δ&#160;-0.07). Multijournal results were similar (score, 5.54&#160;vs&#160;5.82; engagement, 607&#160;vs&#160;685&#160;words; bias, 0.34&#160;vs&#160;0.27). Across 10,000 replications per arm, each difference was significant at P &lt; .001.</p>

![](https://assets.underline.io/markdown_image/1/image/bbc8009dbf735846f879b676a4babc28.png)

<h4>Conclusions</h4>
<p> A pragmatic reviewer training package was associated with modest but consistent improvements in review quality, reviewer engagement, and reduction of positive outcome bias in both single- and multijournal simulations. These data support further real-world trials of low-cost training interventions.</p>
<h4>References</h4>
<p>1. Heim A, Ravaud P, Baron G, Boutron I. Designs of trials assessing interventions to improve the peer review process: a vignette-based survey. <i>BMC Med</i>. 2018;16(1):191. doi:10.1186/s12916-018-1167-7</p>
<p>2. Radzvilas M, De Pretis F, Peden W, Tortoli D, Osimani B. Incentives for research effort: an evolutionary model of publication markets with double-blind and open review. <i>Comput Econ</i>. 2023;61(4):1433-1476. doi:10.1007/s10614-022-10250-w</p>
<p>3. Kang D, Ammar W, Dalvi B, et al. A dataset of peer reviews (PeerRead): collection, insights, and NLP applications. In: <i>Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics (HLT)</i>. Association for Computational Linguistics; 2018;1. doi:10.18653/v1/N18-1149</p>
<sup>1</sup>Sorbonne Université, CNRS, Groupe d’Étude des Méthodes de l’Analyse Sociologique de la Sorbonne, GEMASS, Paris, France; ²Division of Pharmasophy, School of Pharmacy, Saarland University, Saarbrücken, Germany; ³Division of Bioorganic Chemistry, School of Pharmacy, Saarland University, Saarbrücken, Germany; ⁴Department of Environmental and Occupational Health, School of Public Health, Indiana University Bloomington, Bloomington, IN, US, francesco.depretis@unimore.it; ⁵Department of Communication and Economics, University of Modena and Reggio Emilia, Reggio Emilia, Italy.</p>
<h4>Conflict of Interest Disclosures</h4>
<p> None reported.</p>
<h4>Funding/Support</h4>
<p> This research was partially supported by the French Research Agency ANR via the OPENIT project (projet AAPG 2024, Agence Nationale de la Recherche).</p>
<h4>Role of the Funder/Sponsor</h4>
<p> The funding organizations had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the abstract; and decision to submit the abstract for presentation.</p>
<h4>Additional Information</h4>
<p> GPT-o3 (OpenAI) was used for code and writing and editing on January 31, 2025. All authors take responsibility for the integrity of the content.</p>


An Agent-Based Modeling Approach for Evaluating Interventions to Optimize Peer Review

<p>Cyril Labbé,<sup>1</sup> <span lang="fr-FR">Gilles Hubert,</span><sup>2</sup><span lang="fr-FR"> </span>Wendeline Swart,<sup>2</sup> <span lang="fr-FR">Guillaume Cabanac</span><sup>2,3</sup></p>
<h4>Objective </h4>
<p>Paper mills are well documented, and the existence of review mills has been suggested.<sup>1</sup> Our aim was to test tools and methods that would report evidence of such mills. We studied 4 datasets of peer review reports and found evidence for template-based peer review report practices.</p>
<h4>Design </h4>
<p>Attempting to maximize diversity, we collected 4 datasets gathering 148,159 peer review reports. We used webscraped peer review reports from MDPI journals (11 journals from 2018-2025: 47,593 articles and 122,831 reports), <i>BMJ</i> (97 articles from 2021-2024: 308 reports), <i>PeerJ</i> journals listed in the Multidisciplinary Open Peer Review Dataset (MOPRD)<sup>2</sup> (7 journals from 2015-2022: 6292 articles and 12,959 reports), as well as the available dataset NLPeer<sup>3</sup> (12,061 reports from 5 conferences in NLP: years 2016, 2017, 2020, and 2022, and F1000Research platform: year 2022). We investigated only plain text reports (excluding attached files) considering “round 1” only. We computed statistics on report length, common sequences of terms (CST), and similarity measures between reports.</p>
<h4>Results</h4>
<p> Depending on the dataset, the mean report length ranged from 251 to 530 words, and the median from 197 to 433 words. Reports with fewer than 20 words (from 1 to 2825, depending on the dataset) were mostly coming from resubmissions inaccurately reported as a round 1 submission. Regarding CST, excluding stop words (respectively including stop words), 8% to 12% (respectively 11.16% to 24.72%) of reports shared common sequences of at least 10 terms. This represented from 15% to 28% (CST excluding stop words) of articles having at least 1 such report. Regarding similarity, approximately 0.5% to 1.3% of reports were highly similar to another report (ie, cosine similarity excluding stop words greater than 0.75). This represented 0.8% to 3% of articles having at least 1 such report. After having automatically highlighted common CST in reports sharing more than 100 words, we analyzed them qualitatively in order to identify potential templates used by reviewers to write their reports. We were able to identify very generic chunks that are reused from report to report, sometimes by different identifiable reviewers.</p>
<h4>Conclusions</h4>
<p> Our results showed evidence of template-based peer review reports practices. We observed that the extent of these practices varies from one dataset to another. This might be due to reviewers’ practices, discrepancies in dataset sizes, differences in scientific fields, or to journals’, editors’, or publishers’ habits. More openly available review report datasets would help further characterize and understand this phenomenon.</p>
<h4>References</h4>
<p>1. Oviedo-García MÁ. The review mills, not just (self-)plagiarism in review reports, but a step further. <i>Scientometrics</i>. 2024;129:5805-5813. doi:10.1007/s11192-024-05125-w</p>
<p>2. Lin J, Song J, Zhou Z, Chen Y, Shi X. MOPRD: a multidisciplinary open peer review dataset. Neural Comput Applications. 2023;35:24191-24206. https://doi.org/10.1007/s00521-023-08891-5</p>
<p>3. Dycke N, Kuznetsov I, Gurevych I. NLPeer: a unified resource for the computational study of peer review. <i>arXiv</i>. Preprint posted online November 12, 2022.doi:10.18653/v1/2023.acl-long.277</p>
<sup>1</sup>Univ. Grenoble Alpes, CNRS, Grenoble INP, LIG (Laboratoire d’Informatique de Grenoble LIG - équipe SIGMA), Domaine Universitaire de Saint-Martin-d’Hères, Grenoble, France, cyril.labbe@imag.fr; <sup>2</sup>IRIT, Université de Toulouse, CNRS, Toulouse INP, UT, Laboratoire IRIT - équipe IRIS, Toulouse, France; <sup>3</sup>Institut Universitaire de France (IUF), Paris, France.</p>
<h4>Conflict of Interest Disclosures</h4>
<p> None reported.</p>
<h4>Funding/Support </h4>
<p>Cyril Labbé and Guillaume Cabanac acknowledge the NanoBubbles project (<a href="https://nanobubbles.hypotheses.org/"><span class="Hyperlink CharOverride-6">https://nanobubbles.hypotheses.org/</span></a>) that has received Synergy grant funding from the European Research Council (ERC) within the European Union’s Horizon 2020 program, grant agreement number 951393. Guillaume Cabanac received funding from the Institut Universitaire de France (IUF).</p>
<h4>Additional Information</h4>
<p> Cyril Labbé and Gilles Hubert (gilles.hubert@irit.fr) are co–corresponding authors.</p>


Evidence of Use of Template-Based Peer-Review Reports and Concern About Review Mills

<p>Markus K Heinemann,<sup>1</sup> Andreas Boening,<sup>2</sup> Kazunori Okabe,<sup>3</sup> Jessica Bogensberger,<sup>4</sup> Zulfugar Timur Taghiyev<sup>2</sup></p>
<h4>Objective</h4>
<p> Double-anonymized peer review is thought to enhance objectivity but causes additional work for editorial staff.<sup>1-3</sup> The aim of this study was to evaluate its efficiency in a midsized cardiothoracic surgical journal.</p>
<h4>Design</h4>
<p> In July 2017, the journal introduced anonymization, performed by a secretary during the first editorial office check, and introduced custom questions to the review form asking reviewers if they had an idea about the origin of a manuscript despite its anonymization and, if so, why. Descriptive statistics were used for analysis, and a 2-proportion <i>z</i> test was utilized as applicable.</p>
<h4>Results</h4>
<p> Between July 2017 and November 2024, a total of 1735 manuscripts were amenable for anonymization and sent for peer review to a mean of 2 reviewers. For 525 manuscripts (30.3%), at least 1 reviewer had an idea about its origin for at least 1 of 4 potential reasons (multiple entries possible; n = 597): faulty anonymization (232), references (172), familiar with work (146), and characteristic language (47). In 82 manuscripts (15.6%), more than 1 reason was given by at least 1 reviewer. Forty guesses (7.6%) were incorrect for the following reasons: misled by impressions of being familiar with the work (20), references (9), faulty anonymization (6), and characteristic language (5). Multiple reviewers guessed for the same reason in 8 cases and for different reasons in 32 cases. One of the reviewers was incorrect in 5 manuscripts; all reviewers guessed correctly in 35. Among 649 accepted manuscripts (37.4% of the total manuscripts), 223 (34.3%) were deanonymized by reviewers, whereas 426 stayed anonymized (65.6%). The 2-proportion <i>z</i> test demonstrated higher acceptance rates for anonymized compared with deanonymized manuscripts (426 vs 223; <i>z</i> = −11.27).</p>
<h4>Conclusions</h4>
<p> Perfect anonymization remains challenging, if not impossible, because hints identifying origins can be hidden throughout a manuscript. Roughly 30% of manuscripts were correctly deanonymized by the reviewers of an admittedly limited specialty surgical community. Even though the results seem to support different editorial dispositions of anonymized vs deanonymized manuscripts, editors must be aware of the confined reliability of anonymization and weigh it against the additional administrative workload.</p>
<h4>References</h4>
<p>1. Parmanne P, Laajava J, Järvinen N, Harju T, Marttunen M, Saloheimo P. Peer reviewers’ willingness to review, their recommendations and quality of reviews after the Finnish Medical Journal switched from single-blind to double-blind peer review. Res Integr Peer Rev. 2023;8:14. doi:10.1186/s41073-023-00140-6</p>
<p>2. Ucci MA, D’Antonio F, Berghella V. Double- vs single-blind peer review effect on acceptance rates: a systematic review and meta-analysis of randomized trials. <i>Am J Obstet Gynecol MFM</i>. 2022;4(4):100645. doi:10.1016/j.ajogmf.2022.100645</p>
<p>3. Kmietowicz, Z. Double blind peer reviews are fairer and more objective, say academics. <i>BMJ</i>. 2008;336(7638):241. doi:10.1136/bmj.39476.357280.DB</p>
<sup>1</sup>German Society for Thoracic and Cardiovascular Surgery (DGTHG), Germany, heinemann@uni-mainz.de; <sup>2</sup>Universitaetsklinik Giessen, Germany; <sup>3</sup>Bell Land General Hospital, Osaka, Japan; <sup>4</sup>Thieme Publishers, Stuttgart, Germany.</p>
<h4>Conflict of Interest Disclosures </h4>
<p>Jessica Bogensberger is an employee of Thieme Publishers, the publishing house of the journal investigated.</p>


Efficiency of Author Anonymization in Peer Review

<p>Yurong Fei-Bloom,<sup>1</sup> Matthew Welch<sup>2</sup></p>
<h4>Objective </h4>
<p><i>Molecular Biology of the Cell</i> (<i>MBoC</i>) serves as our community cell science journal for members of the American Society for Cell Biology (ASCB) and cell scientists worldwide. ASCB members contribute high-quality research manuscripts to MBoC, enhancing its reputation as a platform for enduring work. Nearly half of the articles published in <i>MBoC</i> come from ASCB members, and most submissions from members are accepted. In our annual surveys, ASCB members expressed great value in having a community-led society journal where they can publish their research, which ensures that funds used to publish remain within our community to support meetings, awards, mentorship, and professional training. The objective of this study was to examine the implications of guaranteed peer review for submissions from ASCB members. </p>
<h4>Design</h4>
<p> Starting in January 2024, <i>MBoC </i>guaranteed that all newly submitted research manuscripts with current ASCB members as corresponding authors would undergo the peer review process.<sup>1</sup> It is important to note that guaranteed peer review did not imply guaranteed acceptance; editorial standards<sup>2</sup> remained the same for all submissions, whether from ASCB members or non-members. Editors were aware of which manuscripts were submitted by members, but reviewers were not aware of the member status. The Editor in Chief carefully assessed each nonmember submission to determine whether it should be desk rejected or moved forward to the peer-review stage. The peer-review process was overseen by dedicated monitoring editors with subject matter expertise. This structure enabled these editors to manage the peer review effectively, fostering a fair and unbiased system that upholds the integrity of the review for all submissions, including those from members.</p>
<h4>Results</h4>
<p> <i>MBoC</i> received 120 submissions (new or revisions) from ASCB members in 2023 and 204 in 2024, which is a 70% increase from the previous year. The total submissions rose by 13%, from 373 in 2023 to 421 in 2024. In 2023, the journal accepted 180 manuscripts, including 102 (57%) from members. In 2024, it accepted 165 manuscripts, with 140 (85%) from members. In 2023, the journal rejected 180 manuscripts, including 12 from members. In 2024, it rejected 233 manuscripts, including 12 from members. The rejection rate of the member manuscripts was 9% in 2023 and 6% in 2024, suggesting that our members consistently submit high-quality work.<i> </i>The average time from submission to the first decision for peer-reviewed articles was 34 days, 2 days fewer than in 2023 (36 days). </p>
<h4>Conclusions</h4>
<p><i> </i>Findings of<i> </i>this study suggest that guaranteed peer review increased submissions from ASCB members while maintaining the quality and timing of the peer review process. The data also suggest that the community values straightforward and guaranteed access to constructive peer feedback.<sup>3</sup> </p>
<h4>References</h4>
<p>1. Fei-Bloom Y, Welch M. New ASCB Member Benefit—Guaranteed Peer Review in MBoC. ASCB Member News, Publishing, Society News. December 22, 2023. Accessed July 15, 2025. <a href="https://www.ascb.org/society-news/new-member-benefit-guaranteed-peer-review-in-mboc/">https://www.ascb.org/society-news/new-member-benefit-guaranteed-peer-review-in-mboc/</a></p>
<p>2. Baum B. Reviewing papers as you would like your papers to be reviewed. <i>Mol Biol Cell</i>. 2019;30:3013-3014. doi.org/10.1091/mbc.E19-05-0273</p>
<p>3. Drubin DG. Any jackass can trash a manuscript, but it takes good scholarship to create one (how <i>MBoC</i> promotes civil and constructive peer review). <i>Mol Biol Cell</i>. 2011;22:525-527. doi.org/10.1091/mbc.e11-01-0002</p>
<sup>1</sup>American Society for Cell Biology, Rockville, MD, US, yfeibloom@ascb.org; <sup>2</sup>Department of Molecular and Cell Biology, University of California, Berkeley, Berkeley, CA, US.</p>
<h4>Conflict of Interest Disclosures </h4>
<p>None reported.</p>


Next from The 10th Peer Review Congress

Detection of Open Science Practices in Major Medical Journals: A Survey and Diagnostic Accuracy of Automatic Tools Using Sensitivity and Specificity

Similar lecture

Effectiveness of Preregistration in Psychology