United States

&lt;p&gt;Luka Ursić,&lt;sup&gt;1,2&lt;/sup&gt; Nensi Bralić,&lt;sup&gt;1&lt;/sup&gt; Giovanni Spitale,&lt;sup&gt;2&lt;/sup&gt; Federico Germani,&lt;sup&gt;2&lt;/sup&gt; Ana Marušić&lt;sup&gt;1&lt;/sup&gt;&lt;/p&gt;
&lt;h4&gt;Objective&lt;/h4&gt;
&lt;p&gt; Divisive discourse observed in news coverage or social media posts by researchers and the general public during the H1N1 and COVID-19 pandemics contributed to polarization and politicization, rapid shifts in sentiment, and concerning rates of misinformation.&lt;sup&gt;1-3&lt;/sup&gt; We aimed to study the sentiment and natural language processing (NLP) analysis of editorial material in medical journals discussing the health care system response to the 2 pandemics.&lt;/p&gt;
&lt;h4&gt;Design&lt;/h4&gt;
&lt;p&gt; Using a systematic review design, we searched Web of Science and PubMed up to May 22, 2024, for editorials, viewpoints, and similar opinion pieces discussing the following segments of the health care system response to the 2 pandemics: nonpharmaceutical interventions; misinformation or disinformation; health care resource allocation and management; health care system preparedness; mandates, policies, and guidance related to research, education, or new technologies; and lessons learned. We limited our search to 2009 to 2014 for the H1N1 pandemic and 2019 to 2024 for the COVID-19 pandemic. Three researchers screened the titles, abstracts, and full texts of the retrieved records for eligibility in pairs, resolving discrepancies through discussion. We extracted the text from the included records’ PDFs using Python and performed a sentiment analysis using the Language Inquiry and Word Count 2022 software. We also used NLP approaches (flat lemmatization, convoluted lemmatization, rule-based autocoding, and cosine similarity) to determine the most frequently used lemmas in the 2 groups of editorials and explore topics based on their co-occurrence. We conducted inferential analyses at the level of the full sample and across specific categories (ie, stratified by pandemic, author country, and context of response). While we could not set a hypothesis for the exploratory NLP analysis, we hypothesized that in terms of sentiment, the editorials published during the COVID-19 pandemic would have higher scores for negative tone and emotion, personal and person-centered language, certitude, and all-or-none thinking and lower scores for analytical thinking.&lt;/p&gt;
&lt;h4&gt;Results &lt;/h4&gt;
&lt;p&gt;Following the screening process (&lt;b&gt;Figure 25-0979&lt;/b&gt;), we included 2954 editorials in the final dataset. We performed a preliminary analysis of 200 editorials: 25 for the H1N1 pandemic and 175 for the COVID-19 pandemic. In contrast to our initial hypothesis, the editorials for the H1N1 pandemic had higher values for negative tone (median, 2.21 [IQR, 1.74-3.07] vs 1.69 [IQR, 1.22-2.31]; &lt;i&gt;P&lt;/i&gt; = .007) and certitude (median, 0.39 [IQR, 0.16-0.66] vs 0.23 [IQR, 0.12-0.37]; &lt;i&gt;P&lt;/i&gt; = .01). There were no differences in negative emotion, analytical thinking, or all-or-none thinking.&lt;/p&gt;

![](https://assets.underline.io/markdown_image/1/image/d658c3c3e26c29b586c4a178cb5264eb.png)

&lt;h4&gt;Conclusions &lt;/h4&gt;
&lt;p&gt;In this preliminary analysis, the editorials for the H1N1 pandemic contained more words connoting negative tone and certitude than those for the COVID-19 pandemic. These findings suggest that the researchers authoring these editorials expressed a more negative sentiment toward the public health response to the H1N1 pandemic and that they were seemingly more certain about their position.&lt;/p&gt;
&lt;h4&gt;References&lt;/h4&gt;
&lt;p&gt;1. Schmidt H. Pandemics and politics: analyzing the politicization and polarization of pandemic-related reporting. &lt;i&gt;Newsp Res J&lt;/i&gt;. 2023;44(1):26-52. doi:10.1177/07395329221095850&lt;/p&gt;
&lt;p&gt;2. Lwin MO, Lu J, Sheldenkar A, et al. Global sentiments surrounding the COVID-19 pandemic on Twitter: analysis of Twitter trends. &lt;i&gt;JMIR Public Health Surveill&lt;/i&gt;. 2020;6(2):e19447. doi:10.2196/19447&lt;/p&gt;
&lt;p&gt;3. Sule S, DaCosta MC, DeCou E, Gilson C, Wallace K, Goff SL. Communication of COVID-19 misinformation on social media by physicians in the US. &lt;i&gt;JAMA Netw Open&lt;/i&gt;. 2023;6(8):e2328928. doi:10.1001/jamanetworkopen.2023.28928&lt;/p&gt;
&lt;sup&gt;1&lt;/sup&gt;Department of Research in Biomedicine and Health, University of Split School of Medicine, Split, Croatia, luka.ursic@mefst.hr; &lt;sup&gt;2&lt;/sup&gt;Institute of Biomedical Ethics and History of Medicine, University of Zürich, Zürich, Switzerland.&lt;/p&gt;
&lt;h4&gt;Conflict of Interest Disclosures&lt;/h4&gt;
&lt;p&gt; Ana Marušić is a member of the Peer Review Congress Advisory Board but was not involved in the review or decision for this abstract. No other disclosures were reported.&lt;/p&gt;
&lt;h4&gt;Funding/Support &lt;/h4&gt;
&lt;p&gt;This study was funded by the Croatian Science Foundation under grant agreement IP-2019-04-4882 and the University of Zürich Institute of Biomedical Ethics and History of Medicine under a Stehr-Boldt fellowship given to Luka Ursić.&lt;/p&gt;
&lt;h4&gt;Role of the Funder/Sponsor&lt;/h4&gt;
&lt;p&gt; The funders had no role in the design of this study, its execution, analyses, interpretation of the data, or decision to submit results.&lt;/p&gt;


The 10th Peer Review Congress

Analysis of Editorials on the Response to the H1N1 and COVID-19 Pandemics

editorial freedom and integrity

pandemic science

research integrity

The Tenth Peer Review Congress featured 3 days for presentations of new research into peer review and all aspects of scientific publication, bias, quality of reporting, and information access and dissemination.

This year’s meeting was hybrid, and all plenary sessions were livestreamed with opportunities to view presentations and engage with participants in person and virtually during and after the meeting. All recordings are now available on Underline for viewing.

<div style="position:relative;padding-top:max(60%,324px);width:100%;height:0;"><iframe style="position:absolute;border:none;width:100%;height:100%;left:0;top:0;" src="https://online.fliphtml5.com/ebtyf/mget/"  seamless="seamless" scrolling="no" frameborder="0" allowtransparency="true" allowfullscreen="true" ></iframe></div>


#### Why? 
Our aim is to encourage research into the quality and credibility of peer review and scientific publication, to establish the evidence base on which scientists can improve the conduct, reporting, and dissemination of scientific research.

#### Who? 
Participants include editors and publishers of scientific peer-reviewed journals, researchers, funders, bibliometric and informatics experts, information innovators, librarians, journalists, policymakers, ethicists, scientific information disseminators, and anyone interested in the progress of the scientific information enterprise and the quality of scientific evidence.

The Congress embraces a wide range of disciplines, including biomedicine, health and life sciences, applied sciences, basic sciences, physical and chemical sciences, mathematics, computer sciences, engineering, economics, and social sciences as well as new and emerging disciplines.

#### What? 
The Congress provides a forum for the presentation and discussion of new research into the quality and credibility of editorial and funding peer review, scientific and scholarly publication, research methods and reporting practices, identification and management of biases, transparency, reproducibility, access, publication and dissemination metrics, identification and management of threats to the integrity of science, and advances in information exchange.

**Follow us**

[![](https://assets.underline.io/markdown_image/1/image/662a22766b9788ed3766cb23cd156e32.png)](https://x.com/peerrevcongress)  [![](https://assets.underline.io/markdown_image/1/image/bb105afb1773a01fa169aa97ca1af49a.png)](https://www.youtube.com/channel/UCsG_0nKi3_Zfyv0yRPdxjWQ)  [![](https://assets.underline.io/markdown_image/1/image/034e1e70a194a9a4178fdf470ef796db.png)](https://bsky.app/profile/peerreviewcongress.bsky.social)



You need to log in with the email address you registered with. 



Please log in to explore this event.

Registration is required to attend this event, please [here](https://peerreviewcongress.org/registration-information/) to secure your ticket. Please check your spam/promotional inbox too.



Please register!

The Tenth Peer Review Congress features 3 days for presentations of new research into peer review and all aspects of scientific publication, bias, quality of reporting, and information access and dissemination.

<p>Luka Ursić,<sup>1,2</sup> Nensi Bralić,<sup>1</sup> Giovanni Spitale,<sup>2</sup> Federico Germani,<sup>2</sup> Ana Marušić<sup>1</sup></p>
<h4>Objective</h4>
<p> Divisive discourse observed in news coverage or social media posts by researchers and the general public during the H1N1 and COVID-19 pandemics contributed to polarization and politicization, rapid shifts in sentiment, and concerning rates of misinformation.<sup>1-3</sup> We aimed to study the sentiment and natural language processing (NLP) analysis of editorial material in medical journals discussing the health care system response to the 2 pandemics.</p>
<h4>Design</h4>
<p> Using a systematic review design, we searched Web of Science and PubMed up to May 22, 2024, for editorials, viewpoints, and similar opinion pieces discussing the following segments of the health care system response to the 2 pandemics: nonpharmaceutical interventions; misinformation or disinformation; health care resource allocation and management; health care system preparedness; mandates, policies, and guidance related to research, education, or new technologies; and lessons learned. We limited our search to 2009 to 2014 for the H1N1 pandemic and 2019 to 2024 for the COVID-19 pandemic. Three researchers screened the titles, abstracts, and full texts of the retrieved records for eligibility in pairs, resolving discrepancies through discussion. We extracted the text from the included records’ PDFs using Python and performed a sentiment analysis using the Language Inquiry and Word Count 2022 software. We also used NLP approaches (flat lemmatization, convoluted lemmatization, rule-based autocoding, and cosine similarity) to determine the most frequently used lemmas in the 2 groups of editorials and explore topics based on their co-occurrence. We conducted inferential analyses at the level of the full sample and across specific categories (ie, stratified by pandemic, author country, and context of response). While we could not set a hypothesis for the exploratory NLP analysis, we hypothesized that in terms of sentiment, the editorials published during the COVID-19 pandemic would have higher scores for negative tone and emotion, personal and person-centered language, certitude, and all-or-none thinking and lower scores for analytical thinking.</p>
<h4>Results </h4>
<p>Following the screening process (<b>Figure 25-0979</b>), we included 2954 editorials in the final dataset. We performed a preliminary analysis of 200 editorials: 25 for the H1N1 pandemic and 175 for the COVID-19 pandemic. In contrast to our initial hypothesis, the editorials for the H1N1 pandemic had higher values for negative tone (median, 2.21 [IQR, 1.74-3.07] vs 1.69 [IQR, 1.22-2.31]; <i>P</i> = .007) and certitude (median, 0.39 [IQR, 0.16-0.66] vs 0.23 [IQR, 0.12-0.37]; <i>P</i> = .01). There were no differences in negative emotion, analytical thinking, or all-or-none thinking.</p>

![](https://assets.underline.io/markdown_image/1/image/d658c3c3e26c29b586c4a178cb5264eb.png)

<h4>Conclusions </h4>
<p>In this preliminary analysis, the editorials for the H1N1 pandemic contained more words connoting negative tone and certitude than those for the COVID-19 pandemic. These findings suggest that the researchers authoring these editorials expressed a more negative sentiment toward the public health response to the H1N1 pandemic and that they were seemingly more certain about their position.</p>
<h4>References</h4>
<p>1. Schmidt H. Pandemics and politics: analyzing the politicization and polarization of pandemic-related reporting. <i>Newsp Res J</i>. 2023;44(1):26-52. doi:10.1177/07395329221095850</p>
<p>2. Lwin MO, Lu J, Sheldenkar A, et al. Global sentiments surrounding the COVID-19 pandemic on Twitter: analysis of Twitter trends. <i>JMIR Public Health Surveill</i>. 2020;6(2):e19447. doi:10.2196/19447</p>
<p>3. Sule S, DaCosta MC, DeCou E, Gilson C, Wallace K, Goff SL. Communication of COVID-19 misinformation on social media by physicians in the US. <i>JAMA Netw Open</i>. 2023;6(8):e2328928. doi:10.1001/jamanetworkopen.2023.28928</p>
<sup>1</sup>Department of Research in Biomedicine and Health, University of Split School of Medicine, Split, Croatia, luka.ursic@mefst.hr; <sup>2</sup>Institute of Biomedical Ethics and History of Medicine, University of Zürich, Zürich, Switzerland.</p>
<h4>Conflict of Interest Disclosures</h4>
<p> Ana Marušić is a member of the Peer Review Congress Advisory Board but was not involved in the review or decision for this abstract. No other disclosures were reported.</p>
<h4>Funding/Support </h4>
<p>This study was funded by the Croatian Science Foundation under grant agreement IP-2019-04-4882 and the University of Zürich Institute of Biomedical Ethics and History of Medicine under a Stehr-Boldt fellowship given to Luka Ursić.</p>
<h4>Role of the Funder/Sponsor</h4>
<p> The funders had no role in the design of this study, its execution, analyses, interpretation of the data, or decision to submit results.</p>


poster

<p>Caitlin J. Bakker,<sup>1,2</sup> Erin E. Reardon,<sup>3</sup> Sarah Jane Brown,<sup>4</sup> Nicole Theis-Mahon,<sup>4</sup> Sara Schroter,<sup>5,6</sup> Lex Bouter,<sup>7,8</sup> Maurice P. Zeegers<sup>2</sup></p>
<h4>Objective </h4>
<p>The scientific community responded rapidly to COVID-19, producing over 200,000 publications in 1 year.<sup>1</sup> This speed brought challenges, including a higher retraction rate.<sup>2</sup> While retraction helps correct the scientific record, retracted status may be inconsistently presented and notices may be incomplete.<sup>3</sup> During a pandemic, inconsistency and incompleteness can have immediate and long-term health impacts. We evaluated retraction presentation consistency and notice completeness for COVID-19 vs non–COVID-19 publications. We describe reasons for retraction and time from publication to retraction.</p>
<h4>Design </h4>
<p>In March 2023, we retrieved retracted publications categorized as research articles or clinical studies in the subject area of public health and safety from Retraction Watch. A previous study focused on all retracted publications in this period<sup>3</sup>; this is a sub-study comparing COVID-19 with non–COVID-19 publications. Between April 28 and June 6, 2023, we assessed consistency in 11 databases (Academia.edu, CINAHL, Embase.com, Ovid Embase, Ovid Medline, PubMed, ResearchGate, SciHub, Scopus, Web of Science, and publisher websites) using 12 criteria from the International Committee of Medical Journal Editors and National Library of Medicine and notice completeness using 17 criteria from Retraction Watch and the Committee on Publication Ethics. Criteria were scored 0 if missing or 1 if present; partial scores were assigned when evaluating multicomponent criteria, such as bidirectional links. To ensure consistent scoring, a random subset of 21% (92 of 441) of retracted publications were independently reviewed by 2 researchers. Each researcher extracted data using Qualtrics forms, and scoring discrepancies were resolved by consensus. Following this calibration phase, remaining publications were extracted by a single reviewer. Kruskal-Wallis tests assessed differences in scores between COVID-19 and non–COVID-19 publications. </p>
<h4>Results</h4>
<p> Of 441 publications, 47 were about COVID-19 and 394 were not. COVID-19 publications were published between 2019 and 2022, while non-COVID 19 publications were published between 1978 and 2022. COVID-19 publications were most frequently retracted due to concerns about reliability of data or results (15 [31.9%]) compared with plagiarism (82 [20.8%]) for non–COVID-19 publications. The median time between publication and retraction was 120 (IQR, 15-196) days for COVID-19 publications and 326 (IQR, 124-789) days for non–COVID-19 publications (<i>P</i> &lt; .001). Across 11 databases, 41.2% (110 of 267) of records retrieved for retracted COVID-19 publications were marked as retracted compared with 47.6% (1225 of 2574) for non–COVID-19 publications. There was no statistically significant difference between consistency or completeness scores for COVID-19 vs non–COVID-19 retracted publications (<b>Table 25-1062</b>). No publications met all criteria.</p>

![](https://assets.underline.io/markdown_image/1/image/edc419e41ec09f04154b15739dea0699.png)

<h4>Conclusions </h4>
<p>Incomplete and inconsistent information poses challenges for researchers and practitioners, undermining trust in scientific literature. We found no association between publications being about COVID-19 and the consistency or completeness of retraction information; however, publications about COVID-19 appeared to be retracted more quickly.</p>
<h4>References</h4>
<p>1. Shimray SR. Research done wrong: a comprehensive investigation of retracted publications in COVID-19. <i>Account Res</i>. 2022;30(7):393-406. doi:10.1080/08989621.2021.2014327</p>
<p>2. Yeo-Teh NSL, Tang BL. An alarming retraction rate for scientific publications on coronavirus disease 2019 (COVID-19). <i>Account Res</i>. 2020;28(1):47-53. doi:10.1080/08989621.2020.1782203</p>
<p>3. Bakker CJ, Reardon EE, Brown SJ, et al. Identification of retracted publications and completeness of retraction notices in public health. <i>J Clin Epidemiol</i>. 2024;173:111427. doi:10.1016/j.jclinepi.2024.111427</p>
<sup>1</sup>University of Regina, Regina, Saskatchewan, Canada, caitlin.bakker@uregina.ca; <sup>2</sup>Maastricht University, Maastricht, the Netherlands; <sup>3</sup>Emory University, Atlanta, GA, US; <sup>4</sup>University of Minnesota, Minneapolis, MN, US; <sup>5</sup><i>BMJ</i>, London, UK; <sup>6</sup>London School of Hygiene and Tropical Medicine, London, UK; <sup>7</sup>Amsterdam University Medical Center, Amsterdam, the Netherlands; <sup>8</sup>Vrije Universiteit Amsterdam, Amsterdam, the Netherlands.</p>
<h4>Conflict of Interest Disclosures</h4>
<p> Caitlin J. Bakker is cochair of the National Information Standards Organization Communication of Retractions, Removals and Expressions of Concern Standing Committee. Lex Bouter is a member of the Peer Review Congress Advisory Board but was not involved in the review or decision for this abstract. No other disclosures were reported.</p>
<h4>Funding/Support</h4>
<p> This research is part of an ongoing PhD collaboration between The BMJ (British Medical Journal) and the team Meta-Research at Maastricht University (UM) on the responsible conduct of publishing scientific research. The BMJ is published by BMJ Group, a wholly owned subsidiary of the British Medical Association. UM is a public legal entity in the Netherlands. This study is part of Caitlin Bakker’s self-funded BMJ/UM PhD. No exchange of funds has taken place for this research project.</p>
<h4>Role of the Funder/Sponsor</h4>
<p> The authors are wholly responsible for the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the abstract; and decision to submit the abstract for presentation.</p>
<h4>Disclaimer</h4>
<p> All authors express their own opinions and not necessarily that of their employers.</p>


Consistency and Completeness of Retractions in Public Health Research on COVID-19

<p>Annie Whamond,<sup>1</sup> Adrian G. Barnett,<sup>2</sup> Jennifer A. Byrne<sup>1,3</sup></p>
<h4>Objective</h4>
<p> Paper mills are unethical organizations that provide low-value or fraudulent content to client authors.<sup>1</sup> Although image manipulation and incorrect reagent detection tools are available,<sup>1</sup> these approaches can require reviewers and readers to learn new skills. Scaled manuscript production through templates may also result in paper mill articles showing other common features. We aimed to help readers identify possible indicators of paper mill support by simply scanning article text and references. We focused on discussion sections for this study.</p>
<h4>Design</h4>
<p> We used an exploratory, cross-sectional design<sup>2</sup> to develop a descriptive analysis of discussion text and references in high Impact Factor (IF) cancer journals, defined as IF of 7 or higher for journal categories oncology, biochemistry and molecular biology, or cell biology. We downloaded the Retraction Watch database on October 16, 2024, and filtered for retraction reason <i>paper mill</i>, subject <i>cancer</i>, and journals on our high IF list. To identify comparison article cohorts, we randomly sampled original cancer articles in (1) the same 8 journals as retracted articles or (2) 8 independent journals with no paper mill retractions and high IF maintained for 20 years or longer. For each article, we recorded the percentages of references first cited in the introduction, methods, results, or discussion section. Focusing on discussion sections, we classified each sentence as providing background, summary, comparison, interpretation or implication, limitation, or future direction<sup>3</sup> and recorded whether individual discussion sentences cited new references. </p>
<h4>Results </h4>
<p>We found 22 retracted paper mill articles from 8 journals published between June 21, 2016, and June 16, 2022. Both comparison groups (50 articles each) were restricted to articles published between January 1, 2016, and October 31, 2024. Articles in all 3 groups included similar numbers of total references per article (median [IQR]: 42 [36-46] retracted articles; 47 [36-55] articles in same journals; 51 [45-60] articles in independent journals) and percentages of discussion sentences per article (median [IQR]: 34 [30-38] retracted articles; 40 [30-48] same journals; 34 [26-43] independent journals). Analyses of discussion sections indicated that retracted paper mill articles included higher percentages of references that were first cited in the discussion and higher percentages of discussion sentences that described background information and cited new references (<b>Figure 25-1022</b>).</p>

![](https://assets.underline.io/markdown_image/1/image/d20ae00ed6b6328fe12ed01f027e066e.png)

<h4>Conclusions</h4>
<p> Our analyses suggest that some discussion sections in retracted paper mill cancer research articles reiterate background information that is supported by new and possibly superfluous references. While recognizing that genuine studies also cite new references in discussion sections, superficial and redundant second introductions in discussion sections could help readers identify potentially problematic articles in high IF cancer journals, particularly when combined with other features of paper mill support.<sup>1</sup></p>
<h4>References</h4>
<p>1. Byrne JA, Abalkina A, Akinduro-Aje O, et al. A call for research to address the threat of paper mills. <i>PLoS Biol</i>. 2024;22(11):e3002931. doi:10.1371/journal.pbio.3002931</p>
<p>2. Kesmodel US. Cross-sectional studies—what are they good for? <i>Acta Obstet Gynecol Scand</i>. 2018;97(4):388-393. doi:10.1111/aogs.13331</p>
<p>3. Toronto CE, Remington R. Discussion and conclusion. In: <i>A Step-by-Step Guide to Conducting an Integrative Review</i>. Toronto CE, Remington R, eds. Springer International; 2020:71-84.</p>
<sup>1</sup>School of Medical Sciences, Faculty of Medicine and Health, University of Sydney, Sydney, New South Wales, Australia, jennifer.byrne@health.nsw.gov.au; <sup>2</sup>School of Public Health and Social Work, Queensland University of Technology, Kelvin Grove, Queensland, Australia; <sup>3</sup>NSW Health Statewide Biobank, NSW Health Pathology, Camperdown, New South Wales, Australia.</p>
<h4>Conflict of Interest Disclosures</h4>
<p> None reported.</p>
<h4>Funding/Support</h4>
<p> Adrian G. Barnett and Jennifer A. Byrne acknowledge grant funding from the National Health and Medical Research Council of Australia (ideas grant APP2029249). This grant supports Annie Whamond’s PhD candidature.</p>
<h4>Role of Funder/Sponsor</h4>
<p> The funding body played no role in the study design, data collection, management, analysis, or interpretation, and will play no role in the writing of any report, or the decision to submit the report for publication.</p>


Analysis of Cancer Research Discussion Text and References in High-Impact Factor Journals for Possible Indicators of Paper Mills

<p>Alexandre Clausse,<sup>1</sup> Guillaume Cabanac,<sup>1,2</sup> Pascal Cuxac,<sup>3</sup> Cyril Labbé<sup>4</sup></p>
<h4>Objective</h4>
<p> The Problematic Paper Screener (PPS) features 10 detectors to identify problematic articles in the scientific record.<sup>1</sup> The detectors search through metadata, references, or textual contents. The tortured phrases<sup>2</sup> detector uses a list of more than 7000 identified suspect expressions (<b>Table 25-1159</b>) to query a database (Dimensions; Digital Science) with approximately 130 million scientific articles.<sup>3</sup> The PPS displays the screening results on a public website. In this qualitative study, we introduce an alternative screening algorithm based on regular expressions (regex) and benchmark its effectiveness. Publishers are welcome to use this stand-alone algorithm that does not require a subscription to the database that we used. </p>

![](https://assets.underline.io/markdown_image/1/image/8b16462ed041c6b4f82712aec9be879f.png)

<h4>Design</h4>
<p> As of May 2025, we designed an algorithm that uses regex based on the PPS fingerprints list in the database to capture different forms of tortured phrases, such as “man-made consciousness,” “profound learning AND deep learning,” and “128 pieces”~5’ [128 bits] (ie, a tortured phrase with components in a 5-word sliding window). This approach is independent from search engines, as it matches fingerprints against the textual content of each article, ignoring their metadata, figure and table contents (including captions), and references. We benchmarked this new approach on the Hindawi extensible markup language (XML) corpus, which contains several hundred articles with tortured phrases, focusing on articles published from 2020 to 2022—the period with the most PPS-flagged articles (n <i>= </i>3400).</p>
<h4>Results</h4>
<p> The regex-based algorithm flagged 2455 problematic articles, with 1948 also flagged by PPS, yielding a 58% overlap. After removing duplicates, 401 articles flagged by the PPS were missing from the Hindawi XML corpus, amounting to 3371 problematic articles in 139 different journals. The evaluation is ongoing; we analyzed the top 200 results and found 48 false-positive results due to database indexing issues related to special characters and figures and PPS querying issues. We also found 52 false-negative results due to fingerprints list issues and because we excluded figures, tables, and references. Overall, we obtained 100 true-positive results, as both detectors (ie, PPS and regex-based algorithm) extracted expressions on the fingerprints list from the same articles.</p>
<h4>Conclusions</h4>
<p> The regex-based algorithm yielded results comparable to the current PPS screening process. Other configurations should be tested, including figures, tables, and references in the screening, as tortured phrases can also appear there. We invite publishers to use this regex-based approach to screen incoming manuscripts. We noticed several querying issues with the database that we used, and some fingerprints need to be redesigned. We will investigate further the validation assessment, as there are still 3351 articles to be reassessed. This research could contribute to improving the regex and updating the PPS fingerprints list, while offering detailed feedback and bug reports to the database developer.</p>
<h4>References</h4>
<p>1. Cabanac G, Labbé C, Magazinov A. The ‘Problematic Paper Screener’ automatically selects suspect publications for post-publication (re)assessment. <i>arXiv</i>. Preprint posted online October 7, 2022. doi:10.48550/arXiv.2210.04895</p>
<p>2. Cabanac G, Labbé C, Magazinov A. Tortured phrases: A dubious writing style emerging in science—evidence of critical issues affecting established journals. <i>arXiv</i>. Preprint posted online July 12, 2021. doi:10.48550/arXiv.2107.06751</p>
<p>3. Herzog C, Hook D, Konkiel S. Bringing down barriers between scientometricians and data. <i>Quantitative Science Studies</i>. 2020;1:387-395. doi:10.1162/qss_a_00020</p>
<sup>1</sup>Université de Toulouse, IRIT (UMR 5505), Toulouse, France, alexandre.clausse@irit.fr; <sup>2</sup>Institut Universitaire de France, Paris, France; <sup>3</sup>INIST CNRS, UAR 67, Vandoeuvre-lès-Nancy, France; <sup>4</sup>Université Grenoble Alpes, CNRS, Grenoble INP, LIG UMR 5217, Grenoble, France.</p>
<h4>Conflicts of Interest Disclosures </h4>
<p>Guillaume Cabanac is the administrator of the Problematic Paper Screener, a public platform that uses metadata from Digital Science and PubPeer via no-cost agreements. Guillaume Cabanac and Cyril Labbé have been in touch with most of the major publishers and their integrity officers, offering pro bono consulting regarding detection tools to various actors in the field, including Clear Skies, Morressier, River Valley, Signals, and STM.</p>
<h4>Funding/Support </h4>
<p>Alexandre Clausse, Guillaume Cabanac, and Cyril Labbé received funding from the European Research Council. Guillaume Cabanac received funding from the Institut Universitaire de France.</p>


Screening articles for tortured phrases with a regex-based detector

<p>Melissa L. Rethlefsen,<sup>1</sup> Carrie Price,<sup>2</sup> Sara Schroter<sup>3</sup></p>
<h4>Objective</h4>
<p> To explore the perspectives of librarians and information specialists (LISs) who participated in a randomized controlled trial of the effect of LIS involvement on reporting quality of systematic reviews (SRs).<sup>1</sup></p>
<h4>Design</h4>
<p> We surveyed LISs who completed a peer review of an SR in a trial conducted in <i>BMJ</i>, <i>BMJ Open</i>, and <i>BMJ Medicine</i> from January 3, 2023, to January 2, 2024. LISs were not told they were peer reviewing manuscripts as part of a trial but were sent invitations to review as part of the usual process. The questionnaire sought to understand their experience, what aspects of manuscripts they focused on, perceived impact on editorial decision-making and authors’ revisions, and willingness to peer review again. To better understand factors that might impact decisions to review again, we contacted 27 respondents to participate in a focus group concentrating on facilitators and barriers to peer reviewing SRs. </p>
<h4>Results</h4>
<p> Of the 88 LISs invited to participate in the survey, 70 (79.5%) responded. Most respondents had 6 or more years of experience as an LIS (67/70; 95.7%) and advising researchers on doing SRs (55/70; 78.6%) and had peer reviewed for a journal prior to the study (57/70; 81.4%). Most focused on the search and SR methods when reviewing but also commented on aspects such as research question formulation, plagiarism, and study results and conclusions. Two-thirds (44/66; 66.7%) believed they impacted editors’ decision-making and 59.1% (39/66) believed they impacted the authors’ revisions. Only 3 factors were considered extremely or very likely to impact their decision to review again: their schedule and/or lack of time, review turnaround time, and their sense of professional duty (<b>Table 24-0830</b>). Seventeen of 27 invited LISs (63.0%) participated in a focus group. Time was the primary barrier identified in the focus groups, followed by a sense of intimidation. LISs reported being motivated by feeling valued by editors, the enjoyment of peer reviewing, the desire to improve SR quality, and peer review as a learning experience. Several expressed surprise and delight at being asked to peer review for the journals.</p>

![](https://assets.underline.io/markdown_image/1/image/c26ed0c4df287a1d953b680e49fbcda6.png)

<h4>Conclusions</h4>
<p> A select sample of highly engaged LIS respondents believed they made a difference through their peer reviews and said they were very likely to agree to peer review in the future. LISs may be an underused peer reviewing resource with methodological experience that can help editors make decisions and improve the quality of SRs.</p>
<h4>Reference</h4>
<p>1. Rethlefsen ML, Schroter S, Bouter LM, et al. Improving peer review of systematic reviews and related review types by involving librarians and information specialists as methodological peer reviewers: a randomised controlled trial. <i>BMJ Evid Based Med</i>. Published online March 11, 2025. doi:10.1136/bmjebm-2024-113527</p>
<sup>1</sup>Health Sciences Library &amp; Informatics Center, University of New Mexico, Albuquerque, NM, US, mlrethlefsen@gmail.com; <sup>2</sup>ToxStrategies, A Blue Ridge Life Sciences Company, Baltimore, MD, US; <sup>3</sup>BMJ Publishing Group, London, UK.</p>
<h4>Conflict of Interest Disclosures </h4>
<p>This study follows up on a study that was conducted as part of Melissa L. Rethlefsen’s self-funded PhD project registered at Maastricht University, the Netherlands, in collaboration with BMJ Publishing Group. Carrie Price was employed at the National Institutes of Health Library at the time of this research. Sara Schroter is a full-time employee of BMJ Publishing Group but is not involved in decision-making on individual research submissions.</p>


Librarian and Information Specialist Perceptions of Peer Reviewing Systematic Reviews

<p><span lang="fr-FR">Abdelghani Maddi,¹ Ahmad Yaman Abdin,²</span><sup>,</sup><span lang="ar-SA">³ Francesco De Pretis⁴</span><sup>,</sup><span lang="fr-FR">⁵</span></p>

<h4>Objective</h4>
<p> To evaluate, using an empirically calibrated, agent-based model (ABM), whether a structured reviewer-training intervention is associated with improved peer review quality and reviewer engagement and reduced positive outcome bias across single- and multijournal settings. Existing randomized and quasi-experimental trials of peer review interventions are limited in scope and effect size,¹ and evolutionary models suggest competition among journals shapes reviewer effort.²</p>
<h4>Design </h4>
<p>We built an ABM populated by authors, reviewers, and editors. Calibration used the PeerRead International Conference on Learning Representations (ICLR)&#160;2017-2019 corpus (8151 reviews, 4905 submissions).³ Simulations followed the STRESS-ABM recommendations. Single-journal scenarios assumed 2500 submissions per year, while multijournal portfolios comprised 5 titles handling 10,000 combined submissions. The intervention increased the share of reviewers completing a certified online training module from 0% to 40%. Main outcomes were (1) review quality score (1-10 ordinal), (2) reviewer engagement (report length, number of words), and (3) positive outcome bias (difference in acceptance probability between high- and low-significance manuscripts). Each scenario used 10,000 Monte-Carlo replications; 95% confidence intervals were obtained with percentile bootstrap.</p>
<h4>Results </h4>
<p>In the single-journal baseline, mean review quality score was 5.52 (95%&#160;CI,&#160;5.49-5.55), and mean reviewer engagement was 604&#160;words (95% CI, 598-610); positive outcome bias was 0.33 (95% CI, 0.32-0.35) (<b>Table 25-0961</b>). With training, mean scores increased to 5.80 (95% CI, 5.77-5.83; Δ&#160;+0.28), engagement to 682&#160;words (95% CI, 675-689; Δ&#160;+78 words), and bias decreased to 0.26 (95% CI, 0.25-0.28; Δ&#160;-0.07). Multijournal results were similar (score, 5.54&#160;vs&#160;5.82; engagement, 607&#160;vs&#160;685&#160;words; bias, 0.34&#160;vs&#160;0.27). Across 10,000 replications per arm, each difference was significant at P &lt; .001.</p>

![](https://assets.underline.io/markdown_image/1/image/bbc8009dbf735846f879b676a4babc28.png)

<h4>Conclusions</h4>
<p> A pragmatic reviewer training package was associated with modest but consistent improvements in review quality, reviewer engagement, and reduction of positive outcome bias in both single- and multijournal simulations. These data support further real-world trials of low-cost training interventions.</p>
<h4>References</h4>
<p>1. Heim A, Ravaud P, Baron G, Boutron I. Designs of trials assessing interventions to improve the peer review process: a vignette-based survey. <i>BMC Med</i>. 2018;16(1):191. doi:10.1186/s12916-018-1167-7</p>
<p>2. Radzvilas M, De Pretis F, Peden W, Tortoli D, Osimani B. Incentives for research effort: an evolutionary model of publication markets with double-blind and open review. <i>Comput Econ</i>. 2023;61(4):1433-1476. doi:10.1007/s10614-022-10250-w</p>
<p>3. Kang D, Ammar W, Dalvi B, et al. A dataset of peer reviews (PeerRead): collection, insights, and NLP applications. In: <i>Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics (HLT)</i>. Association for Computational Linguistics; 2018;1. doi:10.18653/v1/N18-1149</p>
<sup>1</sup>Sorbonne Université, CNRS, Groupe d’Étude des Méthodes de l’Analyse Sociologique de la Sorbonne, GEMASS, Paris, France; ²Division of Pharmasophy, School of Pharmacy, Saarland University, Saarbrücken, Germany; ³Division of Bioorganic Chemistry, School of Pharmacy, Saarland University, Saarbrücken, Germany; ⁴Department of Environmental and Occupational Health, School of Public Health, Indiana University Bloomington, Bloomington, IN, US, francesco.depretis@unimore.it; ⁵Department of Communication and Economics, University of Modena and Reggio Emilia, Reggio Emilia, Italy.</p>
<h4>Conflict of Interest Disclosures</h4>
<p> None reported.</p>
<h4>Funding/Support</h4>
<p> This research was partially supported by the French Research Agency ANR via the OPENIT project (projet AAPG 2024, Agence Nationale de la Recherche).</p>
<h4>Role of the Funder/Sponsor</h4>
<p> The funding organizations had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the abstract; and decision to submit the abstract for presentation.</p>
<h4>Additional Information</h4>
<p> GPT-o3 (OpenAI) was used for code and writing and editing on January 31, 2025. All authors take responsibility for the integrity of the content.</p>


An Agent-Based Modeling Approach for Evaluating Interventions to Optimize Peer Review

<p>Cyril Labbé,<sup>1</sup> <span lang="fr-FR">Gilles Hubert,</span><sup>2</sup><span lang="fr-FR"> </span>Wendeline Swart,<sup>2</sup> <span lang="fr-FR">Guillaume Cabanac</span><sup>2,3</sup></p>
<h4>Objective </h4>
<p>Paper mills are well documented, and the existence of review mills has been suggested.<sup>1</sup> Our aim was to test tools and methods that would report evidence of such mills. We studied 4 datasets of peer review reports and found evidence for template-based peer review report practices.</p>
<h4>Design </h4>
<p>Attempting to maximize diversity, we collected 4 datasets gathering 148,159 peer review reports. We used webscraped peer review reports from MDPI journals (11 journals from 2018-2025: 47,593 articles and 122,831 reports), <i>BMJ</i> (97 articles from 2021-2024: 308 reports), <i>PeerJ</i> journals listed in the Multidisciplinary Open Peer Review Dataset (MOPRD)<sup>2</sup> (7 journals from 2015-2022: 6292 articles and 12,959 reports), as well as the available dataset NLPeer<sup>3</sup> (12,061 reports from 5 conferences in NLP: years 2016, 2017, 2020, and 2022, and F1000Research platform: year 2022). We investigated only plain text reports (excluding attached files) considering “round 1” only. We computed statistics on report length, common sequences of terms (CST), and similarity measures between reports.</p>
<h4>Results</h4>
<p> Depending on the dataset, the mean report length ranged from 251 to 530 words, and the median from 197 to 433 words. Reports with fewer than 20 words (from 1 to 2825, depending on the dataset) were mostly coming from resubmissions inaccurately reported as a round 1 submission. Regarding CST, excluding stop words (respectively including stop words), 8% to 12% (respectively 11.16% to 24.72%) of reports shared common sequences of at least 10 terms. This represented from 15% to 28% (CST excluding stop words) of articles having at least 1 such report. Regarding similarity, approximately 0.5% to 1.3% of reports were highly similar to another report (ie, cosine similarity excluding stop words greater than 0.75). This represented 0.8% to 3% of articles having at least 1 such report. After having automatically highlighted common CST in reports sharing more than 100 words, we analyzed them qualitatively in order to identify potential templates used by reviewers to write their reports. We were able to identify very generic chunks that are reused from report to report, sometimes by different identifiable reviewers.</p>
<h4>Conclusions</h4>
<p> Our results showed evidence of template-based peer review reports practices. We observed that the extent of these practices varies from one dataset to another. This might be due to reviewers’ practices, discrepancies in dataset sizes, differences in scientific fields, or to journals’, editors’, or publishers’ habits. More openly available review report datasets would help further characterize and understand this phenomenon.</p>
<h4>References</h4>
<p>1. Oviedo-García MÁ. The review mills, not just (self-)plagiarism in review reports, but a step further. <i>Scientometrics</i>. 2024;129:5805-5813. doi:10.1007/s11192-024-05125-w</p>
<p>2. Lin J, Song J, Zhou Z, Chen Y, Shi X. MOPRD: a multidisciplinary open peer review dataset. Neural Comput Applications. 2023;35:24191-24206. https://doi.org/10.1007/s00521-023-08891-5</p>
<p>3. Dycke N, Kuznetsov I, Gurevych I. NLPeer: a unified resource for the computational study of peer review. <i>arXiv</i>. Preprint posted online November 12, 2022.doi:10.18653/v1/2023.acl-long.277</p>
<sup>1</sup>Univ. Grenoble Alpes, CNRS, Grenoble INP, LIG (Laboratoire d’Informatique de Grenoble LIG - équipe SIGMA), Domaine Universitaire de Saint-Martin-d’Hères, Grenoble, France, cyril.labbe@imag.fr; <sup>2</sup>IRIT, Université de Toulouse, CNRS, Toulouse INP, UT, Laboratoire IRIT - équipe IRIS, Toulouse, France; <sup>3</sup>Institut Universitaire de France (IUF), Paris, France.</p>
<h4>Conflict of Interest Disclosures</h4>
<p> None reported.</p>
<h4>Funding/Support </h4>
<p>Cyril Labbé and Guillaume Cabanac acknowledge the NanoBubbles project (<a href="https://nanobubbles.hypotheses.org/"><span class="Hyperlink CharOverride-6">https://nanobubbles.hypotheses.org/</span></a>) that has received Synergy grant funding from the European Research Council (ERC) within the European Union’s Horizon 2020 program, grant agreement number 951393. Guillaume Cabanac received funding from the Institut Universitaire de France (IUF).</p>
<h4>Additional Information</h4>
<p> Cyril Labbé and Gilles Hubert (gilles.hubert@irit.fr) are co–corresponding authors.</p>


Evidence of Use of Template-Based Peer-Review Reports and Concern About Review Mills

<p>Markus K Heinemann,<sup>1</sup> Andreas Boening,<sup>2</sup> Kazunori Okabe,<sup>3</sup> Jessica Bogensberger,<sup>4</sup> Zulfugar Timur Taghiyev<sup>2</sup></p>
<h4>Objective</h4>
<p> Double-anonymized peer review is thought to enhance objectivity but causes additional work for editorial staff.<sup>1-3</sup> The aim of this study was to evaluate its efficiency in a midsized cardiothoracic surgical journal.</p>
<h4>Design</h4>
<p> In July 2017, the journal introduced anonymization, performed by a secretary during the first editorial office check, and introduced custom questions to the review form asking reviewers if they had an idea about the origin of a manuscript despite its anonymization and, if so, why. Descriptive statistics were used for analysis, and a 2-proportion <i>z</i> test was utilized as applicable.</p>
<h4>Results</h4>
<p> Between July 2017 and November 2024, a total of 1735 manuscripts were amenable for anonymization and sent for peer review to a mean of 2 reviewers. For 525 manuscripts (30.3%), at least 1 reviewer had an idea about its origin for at least 1 of 4 potential reasons (multiple entries possible; n = 597): faulty anonymization (232), references (172), familiar with work (146), and characteristic language (47). In 82 manuscripts (15.6%), more than 1 reason was given by at least 1 reviewer. Forty guesses (7.6%) were incorrect for the following reasons: misled by impressions of being familiar with the work (20), references (9), faulty anonymization (6), and characteristic language (5). Multiple reviewers guessed for the same reason in 8 cases and for different reasons in 32 cases. One of the reviewers was incorrect in 5 manuscripts; all reviewers guessed correctly in 35. Among 649 accepted manuscripts (37.4% of the total manuscripts), 223 (34.3%) were deanonymized by reviewers, whereas 426 stayed anonymized (65.6%). The 2-proportion <i>z</i> test demonstrated higher acceptance rates for anonymized compared with deanonymized manuscripts (426 vs 223; <i>z</i> = −11.27).</p>
<h4>Conclusions</h4>
<p> Perfect anonymization remains challenging, if not impossible, because hints identifying origins can be hidden throughout a manuscript. Roughly 30% of manuscripts were correctly deanonymized by the reviewers of an admittedly limited specialty surgical community. Even though the results seem to support different editorial dispositions of anonymized vs deanonymized manuscripts, editors must be aware of the confined reliability of anonymization and weigh it against the additional administrative workload.</p>
<h4>References</h4>
<p>1. Parmanne P, Laajava J, Järvinen N, Harju T, Marttunen M, Saloheimo P. Peer reviewers’ willingness to review, their recommendations and quality of reviews after the Finnish Medical Journal switched from single-blind to double-blind peer review. Res Integr Peer Rev. 2023;8:14. doi:10.1186/s41073-023-00140-6</p>
<p>2. Ucci MA, D’Antonio F, Berghella V. Double- vs single-blind peer review effect on acceptance rates: a systematic review and meta-analysis of randomized trials. <i>Am J Obstet Gynecol MFM</i>. 2022;4(4):100645. doi:10.1016/j.ajogmf.2022.100645</p>
<p>3. Kmietowicz, Z. Double blind peer reviews are fairer and more objective, say academics. <i>BMJ</i>. 2008;336(7638):241. doi:10.1136/bmj.39476.357280.DB</p>
<sup>1</sup>German Society for Thoracic and Cardiovascular Surgery (DGTHG), Germany, heinemann@uni-mainz.de; <sup>2</sup>Universitaetsklinik Giessen, Germany; <sup>3</sup>Bell Land General Hospital, Osaka, Japan; <sup>4</sup>Thieme Publishers, Stuttgart, Germany.</p>
<h4>Conflict of Interest Disclosures </h4>
<p>Jessica Bogensberger is an employee of Thieme Publishers, the publishing house of the journal investigated.</p>


Efficiency of Author Anonymization in Peer Review

<p>Yurong Fei-Bloom,<sup>1</sup> Matthew Welch<sup>2</sup></p>
<h4>Objective </h4>
<p><i>Molecular Biology of the Cell</i> (<i>MBoC</i>) serves as our community cell science journal for members of the American Society for Cell Biology (ASCB) and cell scientists worldwide. ASCB members contribute high-quality research manuscripts to MBoC, enhancing its reputation as a platform for enduring work. Nearly half of the articles published in <i>MBoC</i> come from ASCB members, and most submissions from members are accepted. In our annual surveys, ASCB members expressed great value in having a community-led society journal where they can publish their research, which ensures that funds used to publish remain within our community to support meetings, awards, mentorship, and professional training. The objective of this study was to examine the implications of guaranteed peer review for submissions from ASCB members. </p>
<h4>Design</h4>
<p> Starting in January 2024, <i>MBoC </i>guaranteed that all newly submitted research manuscripts with current ASCB members as corresponding authors would undergo the peer review process.<sup>1</sup> It is important to note that guaranteed peer review did not imply guaranteed acceptance; editorial standards<sup>2</sup> remained the same for all submissions, whether from ASCB members or non-members. Editors were aware of which manuscripts were submitted by members, but reviewers were not aware of the member status. The Editor in Chief carefully assessed each nonmember submission to determine whether it should be desk rejected or moved forward to the peer-review stage. The peer-review process was overseen by dedicated monitoring editors with subject matter expertise. This structure enabled these editors to manage the peer review effectively, fostering a fair and unbiased system that upholds the integrity of the review for all submissions, including those from members.</p>
<h4>Results</h4>
<p> <i>MBoC</i> received 120 submissions (new or revisions) from ASCB members in 2023 and 204 in 2024, which is a 70% increase from the previous year. The total submissions rose by 13%, from 373 in 2023 to 421 in 2024. In 2023, the journal accepted 180 manuscripts, including 102 (57%) from members. In 2024, it accepted 165 manuscripts, with 140 (85%) from members. In 2023, the journal rejected 180 manuscripts, including 12 from members. In 2024, it rejected 233 manuscripts, including 12 from members. The rejection rate of the member manuscripts was 9% in 2023 and 6% in 2024, suggesting that our members consistently submit high-quality work.<i> </i>The average time from submission to the first decision for peer-reviewed articles was 34 days, 2 days fewer than in 2023 (36 days). </p>
<h4>Conclusions</h4>
<p><i> </i>Findings of<i> </i>this study suggest that guaranteed peer review increased submissions from ASCB members while maintaining the quality and timing of the peer review process. The data also suggest that the community values straightforward and guaranteed access to constructive peer feedback.<sup>3</sup> </p>
<h4>References</h4>
<p>1. Fei-Bloom Y, Welch M. New ASCB Member Benefit—Guaranteed Peer Review in MBoC. ASCB Member News, Publishing, Society News. December 22, 2023. Accessed July 15, 2025. <a href="https://www.ascb.org/society-news/new-member-benefit-guaranteed-peer-review-in-mboc/">https://www.ascb.org/society-news/new-member-benefit-guaranteed-peer-review-in-mboc/</a></p>
<p>2. Baum B. Reviewing papers as you would like your papers to be reviewed. <i>Mol Biol Cell</i>. 2019;30:3013-3014. doi.org/10.1091/mbc.E19-05-0273</p>
<p>3. Drubin DG. Any jackass can trash a manuscript, but it takes good scholarship to create one (how <i>MBoC</i> promotes civil and constructive peer review). <i>Mol Biol Cell</i>. 2011;22:525-527. doi.org/10.1091/mbc.e11-01-0002</p>
<sup>1</sup>American Society for Cell Biology, Rockville, MD, US, yfeibloom@ascb.org; <sup>2</sup>Department of Molecular and Cell Biology, University of California, Berkeley, Berkeley, CA, US.</p>
<h4>Conflict of Interest Disclosures </h4>
<p>None reported.</p>


Manuscript Submissions Following Implementation of Guaranteed Peer Review

<p>Douglas K. Novins,<sup>1</sup> Mary K. Billingsley,<sup>2</sup> Robert R. Althoff<sup>3</sup> </p>
<h4>Objective</h4>
<p> For many journals in science and medicine, a large number of submitted manuscripts may be rejected by editors before peer review. Given the high rates of such “desk rejections,” it is important that editorial reviews minimize bias in this decision-making process. One potential source of bias is the editor having access to information regarding the authors of manuscripts, which may result in editorial decision being influenced by authors’ academic reputation as well as other author characteristics.<sup>1,2</sup> Unfortunately, a recent review was unable to identify any empirical studies of such “triple-blind” peer review.<sup>3</sup> In this pilot study, we masked author information in the initial stage of editorial review to explore whether this changes patterns in the articles that our journal sends out for peer review.</p>
<h4>Design</h4>
<p> The <i>Journal of the American Academy of Child and Adolescent Psychiatry</i> utilizes a 2-round editor in chief (EIC) review process during which the EIC is aware of author identity in both rounds of review. In the standard round 1 review, the EIC completes a high-level reading of the manuscript and the cover letter, checks for test recycling, and reviews study preregistration. In the standard round 2 review, the EIC completes a detailed read of the manuscript. In this preregistered pretest/posttest study (<a href="https://doi.org/10.17605/OSF.IO/TNYFX">https://doi.org/ 10.17605/OSF.IO/TNYFX</a>), we followed the above standard process in the pretest. In the posttest, the identity of the authors was masked in the round 1 review and tasks that unmask author identity, such as reading the cover leader, were moved to the round 2 review. </p>
<h4>Results</h4>
<p> A total of 95 manuscripts were included in the pretest and 102 were included in the posttest. In the masked process, significantly more manuscript decisions were made in round 2 (49/102) than in the unmasked process (26/95) (χ² = 8.91; 1 degree of freedom; <i>P</i> &lt; .01) (<b>Figure 25-1112</b>). Overall, EIC decisions (reject, transfer, assignment to an action editor) did not differ between the pretest and posttest (χ² = 2.48; 2 degrees of freedom; <i>P</i> = .29). </p>

![](https://assets.underline.io/markdown_image/1/image/aafe3983a7ba8c694206726665c8c41c.png)

<h4>Conclusions </h4>
<p>The shift of decisions from round 1 to round 2 in the posttest review process is consistent with the shift of key EIC tasks to the round 2 review, resulting in more manuscripts also receiving a detailed read of their manuscript than in the pretest. </p>
<h4>References</h4>
<p>1. Fox CW, Meyer J, Aimé E. Double-blind peer review affects reviewer ratings and editor decisions at an ecology journal. <i>Funct Ecol</i>. 2023;37(5):1144-1157. doi:10.1111/1365-2435.14259</p>
<p>2. Srivastava DS, Bernardino J, Marques AT, et al. Editors are biased too: an extension of Fox et al. (2023)’s analysis makes the case for triple-blind review. Funct Ecol. 2024;38(2):278-283. doi:10.1111/1365-2435.14483</p>
<p>3. Polnaszek BE, Mei J, Cheng C, et al. Triple-blind peer review in scientific publishing: a systematic review. <i>Am J Obstet Gynecol MFM</i>. 2024;6(4):101320. doi:10.1016/j.ajogmf.2024.101320</p>
<sup>1</sup><i>Journal of the American Academy of Child and Adolescent Psychiatry</i> (University of Colorado Anschutz Medical Campus); <sup>2</sup><i>Journal of the American Academy of Child and Adolescent Psychiatry</i> (American Academy of Child and Adolescent Psychiatry); <sup>3</sup><i>Journal of the American Academy of Child and Adolescent Psychiatry</i> (University of Vermont).</p>
<h4>Conflict of Interest Disclosures</h4>
<p> Douglas K. Novins reported receiving honorarium for serving as editor in chief for the <i>Journal of the American Academy of Child and Adolescent Psychiatry</i>, American Academy of Child and Adolescent Psychiatry and grant support from the National Institutes of Health. Robert R. Althoff reported receiving honorarium for serving as associate editor for the <i>Journal of the American Academy of Child and Adolescent Psychiatry</i> from the American Academy of Child and Adolescent Psychiatry; grant support from National Institutes of Health and Klingenstein Third Generation Foundation; and stock or Equity in WISER Systems, LLC.</p>
<h4>Acknowledgment</h4>
<p> We acknowledge the contributions of our journal’s senior editorial team in designing this study.</p>


Editor Initial Manuscript Review: A Masked Pilot Study

<p>Andrea Corvillon,<sup>1</sup> John Carpenter,<sup>1</sup> Nihar B. Shah<sup>2</sup></p>
<h4>Objective </h4>
<p>As the Atacama Large Millimeter/Submillimeter (ALMA) telescope, the largest radio telescope in the world, transitioned from panel-based to distributed peer review to manage increasing proposal volumes, new challenges emerged in aligning reviewer expertise with proposal content. Building on prior work highlighting the importance of reviewer-proposal match quality, this study evaluates a machine learning and optimization framework to improve assignment fairness and accuracy.</p>
<h4>Design</h4>
<p> The new assignment process has 2 steps: (1) using machine learning to measure similarity between a proposal’s topic and a reviewer’s expertise and (2) optimizing assignments based on a fairness metric. Proposal topics were inferred using latent Dirichlet allocation (LDA), trained on ALMA proposals submitted between 2012 and 2023; the topics were known based on a set of keywords selected by the principal investigators. Reviewer expertise was estimated using the same model trained on each reviewer’s past proposals. Both were represented as topic vectors, and cosine similarity was used to assess alignment. Each proposal was assigned to 10 reviewers, and each reviewer received 10 proposals. Assignments were optimized using the PeerReview4All algorithm,<sup>1</sup> which prioritizes proposals with the lowest similarity scores and least reviewer availability, ensuring fairer matches by improving assignments for the most disadvantaged proposals. This method was compared with ALMA’s approach from 2021 to 2022, which relied on direct keyword overlap. In 2021 and 2022, 1016 and 1087 reviewers assessed 1497 and 1729 proposals, resulting in 14,970 and 17,290 assignments, respectively.</p>
<h4>Results</h4>
<p> The new method was implemented in 2023, involving 1098 reviewers and 1635 proposals. Similarity scores were validated using survey data from cycles 2021 and 2022. On average, 89% of the reviewers rated their expertise. The median similarity of assignments in which reviewers identified themselves as experts was 0.35 compared with 0.04 for nonexpert assignments, confirming the metric’s reliability. Retrospective application of the similarity metric to cycles 2021 and 2022 using the same LDA model showed that the median similarity increased from 0.20 in 2022, the final cycle of the old algorithm, to 0.71 in 2023, the first cycle of the new algorithm, demonstrating a better alignment between reviewers and proposal assignments. Every cycle, on average, 87% of reviewers rated their expertise level on their assigned proposals. With the new algorithm, the percentage of reviewers identifying themselves as experts increased from 45% to 65%, while self-identified nonexperts decreased from 10% to 5%. This tendency continued in cycle 2024, when the new algorithm was also used (<b>Figure 25-1127</b>). Additionally, the new algorithm eliminated manual reassignments, reducing manual effort time from 3 to 5 days to nearly zero. These findings will be updated with analysis of new data from 2024 and 2025.</p>

![](https://assets.underline.io/markdown_image/1/image/1bbe2eac526f172fa408e3aeb5da040a.png)

<h4>Conclusions </h4>
<p>ALMA’s machine learning–based assignment framework improved reviewer-proposal alignment, increased expertise match rates, and eliminated manual reassignments.</p>
<h4>Reference</h4>
<p>1. Stelmakh I, Shah N, Singh A. PeerReview4All: fair and accurate reviewer assignment in peer review. <i>J Machine Learning Res</i>. 2021;22(163):1-66. Accessed July 16, 2025. <a href="https://jmlr.csail.mit.edu/papers/volume22/20-190/20-190.pdf"><span class="Hyperlink CharOverride-6">https://jmlr.csail.mit.edu/papers/volume22/20-190/20-190.pdf</span></a></p>
<sup>1</sup>Joint ALMA Observatory, Santiago, Chile, andrea.corvillon@alma.cl; <sup>2</sup>Carnegie Mellon University, Pittsburgh, PA, US.</p>
<h4>Conflict of Interest Disclosures</h4>
<p> Andrea Corvillon and John Carpenter are employed by the Joint ALMA Observatory (JAO), which is jointly managed by Associated Universities, Inc/National Radio Astronomy Observatory, the European Organization for Astronomy Research in the Southern Hemisphere (ESO), and the National Astronomical Observatory of Japan (NAOJ) on behalf of the ALMA partnership. Nihar B. Shah is employed by the Carnegie Mellon University and is a member of the Peer Review Congress Advisory Board but was not involved in the editorial review or decision for this abstract.</p>
<h4>Funding</h4>
<p> The study was funded by grant 1942124 from the National Science Foundation (NSF) to the JAO.</p>
<h4>Role of Funder/Sponsor</h4>
<p> This work is carried out as part of the duties of the authors at the Joint ALMA Observatory, which receives funding from the NSF through NRAO. The NSF did not contribute to the analysis presented in this contribution.</p>
<h4>Acknowledgments</h4>
<p> ALMA is a partnership of ESO (representing its member states), NSF (US), and the National Institute of Natural Sciences (Japan), together with the National Research Council of Canada (Canada), the National Science and Technology Council and the Institute of Astronomy &amp; Astrophysics, Academia Sinica (Taiwan), and the Korea Astronomy and Space Science Institute (Republic of Korea), in cooperation with the Republic of Chile.</p>


Next from The 10th Peer Review Congress

Consistency and Completeness of Retractions in Public Health Research on COVID-19

Similar lecture

Natural Language Processing to Assess the Role of Preprints in COVID-19 Policy Guidance