United States

&lt;p&gt;Cyril Labbé,&lt;sup&gt;1&lt;/sup&gt; &lt;span lang=&quot;fr-FR&quot;&gt;Gilles Hubert,&lt;/span&gt;&lt;sup&gt;2&lt;/sup&gt;&lt;span lang=&quot;fr-FR&quot;&gt; &lt;/span&gt;Wendeline Swart,&lt;sup&gt;2&lt;/sup&gt; &lt;span lang=&quot;fr-FR&quot;&gt;Guillaume Cabanac&lt;/span&gt;&lt;sup&gt;2,3&lt;/sup&gt;&lt;/p&gt;
&lt;h4&gt;Objective &lt;/h4&gt;
&lt;p&gt;Paper mills are well documented, and the existence of review mills has been suggested.&lt;sup&gt;1&lt;/sup&gt; Our aim was to test tools and methods that would report evidence of such mills. We studied 4 datasets of peer review reports and found evidence for template-based peer review report practices.&lt;/p&gt;
&lt;h4&gt;Design &lt;/h4&gt;
&lt;p&gt;Attempting to maximize diversity, we collected 4 datasets gathering 148,159 peer review reports. We used webscraped peer review reports from MDPI journals (11 journals from 2018-2025: 47,593 articles and 122,831 reports), &lt;i&gt;BMJ&lt;/i&gt; (97 articles from 2021-2024: 308 reports), &lt;i&gt;PeerJ&lt;/i&gt; journals listed in the Multidisciplinary Open Peer Review Dataset (MOPRD)&lt;sup&gt;2&lt;/sup&gt; (7 journals from 2015-2022: 6292 articles and 12,959 reports), as well as the available dataset NLPeer&lt;sup&gt;3&lt;/sup&gt; (12,061 reports from 5 conferences in NLP: years 2016, 2017, 2020, and 2022, and F1000Research platform: year 2022). We investigated only plain text reports (excluding attached files) considering “round 1” only. We computed statistics on report length, common sequences of terms (CST), and similarity measures between reports.&lt;/p&gt;
&lt;h4&gt;Results&lt;/h4&gt;
&lt;p&gt; Depending on the dataset, the mean report length ranged from 251 to 530 words, and the median from 197 to 433 words. Reports with fewer than 20 words (from 1 to 2825, depending on the dataset) were mostly coming from resubmissions inaccurately reported as a round 1 submission. Regarding CST, excluding stop words (respectively including stop words), 8% to 12% (respectively 11.16% to 24.72%) of reports shared common sequences of at least 10 terms. This represented from 15% to 28% (CST excluding stop words) of articles having at least 1 such report. Regarding similarity, approximately 0.5% to 1.3% of reports were highly similar to another report (ie, cosine similarity excluding stop words greater than 0.75). This represented 0.8% to 3% of articles having at least 1 such report. After having automatically highlighted common CST in reports sharing more than 100 words, we analyzed them qualitatively in order to identify potential templates used by reviewers to write their reports. We were able to identify very generic chunks that are reused from report to report, sometimes by different identifiable reviewers.&lt;/p&gt;
&lt;h4&gt;Conclusions&lt;/h4&gt;
&lt;p&gt; Our results showed evidence of template-based peer review reports practices. We observed that the extent of these practices varies from one dataset to another. This might be due to reviewers’ practices, discrepancies in dataset sizes, differences in scientific fields, or to journals’, editors’, or publishers’ habits. More openly available review report datasets would help further characterize and understand this phenomenon.&lt;/p&gt;
&lt;h4&gt;References&lt;/h4&gt;
&lt;p&gt;1. Oviedo-García MÁ. The review mills, not just (self-)plagiarism in review reports, but a step further. &lt;i&gt;Scientometrics&lt;/i&gt;. 2024;129:5805-5813. doi:10.1007/s11192-024-05125-w&lt;/p&gt;
&lt;p&gt;2. Lin J, Song J, Zhou Z, Chen Y, Shi X. MOPRD: a multidisciplinary open peer review dataset. Neural Comput Applications. 2023;35:24191-24206. https://doi.org/10.1007/s00521-023-08891-5&lt;/p&gt;
&lt;p&gt;3. Dycke N, Kuznetsov I, Gurevych I. NLPeer: a unified resource for the computational study of peer review. &lt;i&gt;arXiv&lt;/i&gt;. Preprint posted online November 12, 2022.doi:10.18653/v1/2023.acl-long.277&lt;/p&gt;
&lt;sup&gt;1&lt;/sup&gt;Univ. Grenoble Alpes, CNRS, Grenoble INP, LIG (Laboratoire d’Informatique de Grenoble LIG - équipe SIGMA), Domaine Universitaire de Saint-Martin-d’Hères, Grenoble, France, cyril.labbe@imag.fr; &lt;sup&gt;2&lt;/sup&gt;IRIT, Université de Toulouse, CNRS, Toulouse INP, UT, Laboratoire IRIT - équipe IRIS, Toulouse, France; &lt;sup&gt;3&lt;/sup&gt;Institut Universitaire de France (IUF), Paris, France.&lt;/p&gt;
&lt;h4&gt;Conflict of Interest Disclosures&lt;/h4&gt;
&lt;p&gt; None reported.&lt;/p&gt;
&lt;h4&gt;Funding/Support &lt;/h4&gt;
&lt;p&gt;Cyril Labbé and Guillaume Cabanac acknowledge the NanoBubbles project (&lt;a href=&quot;https://nanobubbles.hypotheses.org/&quot;&gt;&lt;span class=&quot;Hyperlink CharOverride-6&quot;&gt;https://nanobubbles.hypotheses.org/&lt;/span&gt;&lt;/a&gt;) that has received Synergy grant funding from the European Research Council (ERC) within the European Union’s Horizon 2020 program, grant agreement number 951393. Guillaume Cabanac received funding from the Institut Universitaire de France (IUF).&lt;/p&gt;
&lt;h4&gt;Additional Information&lt;/h4&gt;
&lt;p&gt; Cyril Labbé and Gilles Hubert (gilles.hubert@irit.fr) are co–corresponding authors.&lt;/p&gt;


The 10th Peer Review Congress

Evidence of Use of Template-Based Peer-Review Reports and Concern About Review Mills

misconduct

peer review

research integrity

The Tenth Peer Review Congress featured 3 days for presentations of new research into peer review and all aspects of scientific publication, bias, quality of reporting, and information access and dissemination.

This year’s meeting was hybrid, and all plenary sessions were livestreamed with opportunities to view presentations and engage with participants in person and virtually during and after the meeting. All recordings are now available on Underline for viewing.

<div style="position:relative;padding-top:max(60%,324px);width:100%;height:0;"><iframe style="position:absolute;border:none;width:100%;height:100%;left:0;top:0;" src="https://online.fliphtml5.com/ebtyf/mget/"  seamless="seamless" scrolling="no" frameborder="0" allowtransparency="true" allowfullscreen="true" ></iframe></div>


#### Why? 
Our aim is to encourage research into the quality and credibility of peer review and scientific publication, to establish the evidence base on which scientists can improve the conduct, reporting, and dissemination of scientific research.

#### Who? 
Participants include editors and publishers of scientific peer-reviewed journals, researchers, funders, bibliometric and informatics experts, information innovators, librarians, journalists, policymakers, ethicists, scientific information disseminators, and anyone interested in the progress of the scientific information enterprise and the quality of scientific evidence.

The Congress embraces a wide range of disciplines, including biomedicine, health and life sciences, applied sciences, basic sciences, physical and chemical sciences, mathematics, computer sciences, engineering, economics, and social sciences as well as new and emerging disciplines.

#### What? 
The Congress provides a forum for the presentation and discussion of new research into the quality and credibility of editorial and funding peer review, scientific and scholarly publication, research methods and reporting practices, identification and management of biases, transparency, reproducibility, access, publication and dissemination metrics, identification and management of threats to the integrity of science, and advances in information exchange.

**Follow us**

[![](https://assets.underline.io/markdown_image/1/image/662a22766b9788ed3766cb23cd156e32.png)](https://x.com/peerrevcongress)  [![](https://assets.underline.io/markdown_image/1/image/bb105afb1773a01fa169aa97ca1af49a.png)](https://www.youtube.com/channel/UCsG_0nKi3_Zfyv0yRPdxjWQ)  [![](https://assets.underline.io/markdown_image/1/image/034e1e70a194a9a4178fdf470ef796db.png)](https://bsky.app/profile/peerreviewcongress.bsky.social)



You need to log in with the email address you registered with. 



Please log in to explore this event.

Registration is required to attend this event, please [here](https://peerreviewcongress.org/registration-information/) to secure your ticket. Please check your spam/promotional inbox too.



Please register!

The Tenth Peer Review Congress features 3 days for presentations of new research into peer review and all aspects of scientific publication, bias, quality of reporting, and information access and dissemination.

<p>Cyril Labbé,<sup>1</sup> <span lang="fr-FR">Gilles Hubert,</span><sup>2</sup><span lang="fr-FR"> </span>Wendeline Swart,<sup>2</sup> <span lang="fr-FR">Guillaume Cabanac</span><sup>2,3</sup></p>
<h4>Objective </h4>
<p>Paper mills are well documented, and the existence of review mills has been suggested.<sup>1</sup> Our aim was to test tools and methods that would report evidence of such mills. We studied 4 datasets of peer review reports and found evidence for template-based peer review report practices.</p>
<h4>Design </h4>
<p>Attempting to maximize diversity, we collected 4 datasets gathering 148,159 peer review reports. We used webscraped peer review reports from MDPI journals (11 journals from 2018-2025: 47,593 articles and 122,831 reports), <i>BMJ</i> (97 articles from 2021-2024: 308 reports), <i>PeerJ</i> journals listed in the Multidisciplinary Open Peer Review Dataset (MOPRD)<sup>2</sup> (7 journals from 2015-2022: 6292 articles and 12,959 reports), as well as the available dataset NLPeer<sup>3</sup> (12,061 reports from 5 conferences in NLP: years 2016, 2017, 2020, and 2022, and F1000Research platform: year 2022). We investigated only plain text reports (excluding attached files) considering “round 1” only. We computed statistics on report length, common sequences of terms (CST), and similarity measures between reports.</p>
<h4>Results</h4>
<p> Depending on the dataset, the mean report length ranged from 251 to 530 words, and the median from 197 to 433 words. Reports with fewer than 20 words (from 1 to 2825, depending on the dataset) were mostly coming from resubmissions inaccurately reported as a round 1 submission. Regarding CST, excluding stop words (respectively including stop words), 8% to 12% (respectively 11.16% to 24.72%) of reports shared common sequences of at least 10 terms. This represented from 15% to 28% (CST excluding stop words) of articles having at least 1 such report. Regarding similarity, approximately 0.5% to 1.3% of reports were highly similar to another report (ie, cosine similarity excluding stop words greater than 0.75). This represented 0.8% to 3% of articles having at least 1 such report. After having automatically highlighted common CST in reports sharing more than 100 words, we analyzed them qualitatively in order to identify potential templates used by reviewers to write their reports. We were able to identify very generic chunks that are reused from report to report, sometimes by different identifiable reviewers.</p>
<h4>Conclusions</h4>
<p> Our results showed evidence of template-based peer review reports practices. We observed that the extent of these practices varies from one dataset to another. This might be due to reviewers’ practices, discrepancies in dataset sizes, differences in scientific fields, or to journals’, editors’, or publishers’ habits. More openly available review report datasets would help further characterize and understand this phenomenon.</p>
<h4>References</h4>
<p>1. Oviedo-García MÁ. The review mills, not just (self-)plagiarism in review reports, but a step further. <i>Scientometrics</i>. 2024;129:5805-5813. doi:10.1007/s11192-024-05125-w</p>
<p>2. Lin J, Song J, Zhou Z, Chen Y, Shi X. MOPRD: a multidisciplinary open peer review dataset. Neural Comput Applications. 2023;35:24191-24206. https://doi.org/10.1007/s00521-023-08891-5</p>
<p>3. Dycke N, Kuznetsov I, Gurevych I. NLPeer: a unified resource for the computational study of peer review. <i>arXiv</i>. Preprint posted online November 12, 2022.doi:10.18653/v1/2023.acl-long.277</p>
<sup>1</sup>Univ. Grenoble Alpes, CNRS, Grenoble INP, LIG (Laboratoire d’Informatique de Grenoble LIG - équipe SIGMA), Domaine Universitaire de Saint-Martin-d’Hères, Grenoble, France, cyril.labbe@imag.fr; <sup>2</sup>IRIT, Université de Toulouse, CNRS, Toulouse INP, UT, Laboratoire IRIT - équipe IRIS, Toulouse, France; <sup>3</sup>Institut Universitaire de France (IUF), Paris, France.</p>
<h4>Conflict of Interest Disclosures</h4>
<p> None reported.</p>
<h4>Funding/Support </h4>
<p>Cyril Labbé and Guillaume Cabanac acknowledge the NanoBubbles project (<a href="https://nanobubbles.hypotheses.org/"><span class="Hyperlink CharOverride-6">https://nanobubbles.hypotheses.org/</span></a>) that has received Synergy grant funding from the European Research Council (ERC) within the European Union’s Horizon 2020 program, grant agreement number 951393. Guillaume Cabanac received funding from the Institut Universitaire de France (IUF).</p>
<h4>Additional Information</h4>
<p> Cyril Labbé and Gilles Hubert (gilles.hubert@irit.fr) are co–corresponding authors.</p>


poster

<p>Markus K Heinemann,<sup>1</sup> Andreas Boening,<sup>2</sup> Kazunori Okabe,<sup>3</sup> Jessica Bogensberger,<sup>4</sup> Zulfugar Timur Taghiyev<sup>2</sup></p>
<h4>Objective</h4>
<p> Double-anonymized peer review is thought to enhance objectivity but causes additional work for editorial staff.<sup>1-3</sup> The aim of this study was to evaluate its efficiency in a midsized cardiothoracic surgical journal.</p>
<h4>Design</h4>
<p> In July 2017, the journal introduced anonymization, performed by a secretary during the first editorial office check, and introduced custom questions to the review form asking reviewers if they had an idea about the origin of a manuscript despite its anonymization and, if so, why. Descriptive statistics were used for analysis, and a 2-proportion <i>z</i> test was utilized as applicable.</p>
<h4>Results</h4>
<p> Between July 2017 and November 2024, a total of 1735 manuscripts were amenable for anonymization and sent for peer review to a mean of 2 reviewers. For 525 manuscripts (30.3%), at least 1 reviewer had an idea about its origin for at least 1 of 4 potential reasons (multiple entries possible; n = 597): faulty anonymization (232), references (172), familiar with work (146), and characteristic language (47). In 82 manuscripts (15.6%), more than 1 reason was given by at least 1 reviewer. Forty guesses (7.6%) were incorrect for the following reasons: misled by impressions of being familiar with the work (20), references (9), faulty anonymization (6), and characteristic language (5). Multiple reviewers guessed for the same reason in 8 cases and for different reasons in 32 cases. One of the reviewers was incorrect in 5 manuscripts; all reviewers guessed correctly in 35. Among 649 accepted manuscripts (37.4% of the total manuscripts), 223 (34.3%) were deanonymized by reviewers, whereas 426 stayed anonymized (65.6%). The 2-proportion <i>z</i> test demonstrated higher acceptance rates for anonymized compared with deanonymized manuscripts (426 vs 223; <i>z</i> = −11.27).</p>
<h4>Conclusions</h4>
<p> Perfect anonymization remains challenging, if not impossible, because hints identifying origins can be hidden throughout a manuscript. Roughly 30% of manuscripts were correctly deanonymized by the reviewers of an admittedly limited specialty surgical community. Even though the results seem to support different editorial dispositions of anonymized vs deanonymized manuscripts, editors must be aware of the confined reliability of anonymization and weigh it against the additional administrative workload.</p>
<h4>References</h4>
<p>1. Parmanne P, Laajava J, Järvinen N, Harju T, Marttunen M, Saloheimo P. Peer reviewers’ willingness to review, their recommendations and quality of reviews after the Finnish Medical Journal switched from single-blind to double-blind peer review. Res Integr Peer Rev. 2023;8:14. doi:10.1186/s41073-023-00140-6</p>
<p>2. Ucci MA, D’Antonio F, Berghella V. Double- vs single-blind peer review effect on acceptance rates: a systematic review and meta-analysis of randomized trials. <i>Am J Obstet Gynecol MFM</i>. 2022;4(4):100645. doi:10.1016/j.ajogmf.2022.100645</p>
<p>3. Kmietowicz, Z. Double blind peer reviews are fairer and more objective, say academics. <i>BMJ</i>. 2008;336(7638):241. doi:10.1136/bmj.39476.357280.DB</p>
<sup>1</sup>German Society for Thoracic and Cardiovascular Surgery (DGTHG), Germany, heinemann@uni-mainz.de; <sup>2</sup>Universitaetsklinik Giessen, Germany; <sup>3</sup>Bell Land General Hospital, Osaka, Japan; <sup>4</sup>Thieme Publishers, Stuttgart, Germany.</p>
<h4>Conflict of Interest Disclosures </h4>
<p>Jessica Bogensberger is an employee of Thieme Publishers, the publishing house of the journal investigated.</p>


Efficiency of Author Anonymization in Peer Review

<p>Yurong Fei-Bloom,<sup>1</sup> Matthew Welch<sup>2</sup></p>
<h4>Objective </h4>
<p><i>Molecular Biology of the Cell</i> (<i>MBoC</i>) serves as our community cell science journal for members of the American Society for Cell Biology (ASCB) and cell scientists worldwide. ASCB members contribute high-quality research manuscripts to MBoC, enhancing its reputation as a platform for enduring work. Nearly half of the articles published in <i>MBoC</i> come from ASCB members, and most submissions from members are accepted. In our annual surveys, ASCB members expressed great value in having a community-led society journal where they can publish their research, which ensures that funds used to publish remain within our community to support meetings, awards, mentorship, and professional training. The objective of this study was to examine the implications of guaranteed peer review for submissions from ASCB members. </p>
<h4>Design</h4>
<p> Starting in January 2024, <i>MBoC </i>guaranteed that all newly submitted research manuscripts with current ASCB members as corresponding authors would undergo the peer review process.<sup>1</sup> It is important to note that guaranteed peer review did not imply guaranteed acceptance; editorial standards<sup>2</sup> remained the same for all submissions, whether from ASCB members or non-members. Editors were aware of which manuscripts were submitted by members, but reviewers were not aware of the member status. The Editor in Chief carefully assessed each nonmember submission to determine whether it should be desk rejected or moved forward to the peer-review stage. The peer-review process was overseen by dedicated monitoring editors with subject matter expertise. This structure enabled these editors to manage the peer review effectively, fostering a fair and unbiased system that upholds the integrity of the review for all submissions, including those from members.</p>
<h4>Results</h4>
<p> <i>MBoC</i> received 120 submissions (new or revisions) from ASCB members in 2023 and 204 in 2024, which is a 70% increase from the previous year. The total submissions rose by 13%, from 373 in 2023 to 421 in 2024. In 2023, the journal accepted 180 manuscripts, including 102 (57%) from members. In 2024, it accepted 165 manuscripts, with 140 (85%) from members. In 2023, the journal rejected 180 manuscripts, including 12 from members. In 2024, it rejected 233 manuscripts, including 12 from members. The rejection rate of the member manuscripts was 9% in 2023 and 6% in 2024, suggesting that our members consistently submit high-quality work.<i> </i>The average time from submission to the first decision for peer-reviewed articles was 34 days, 2 days fewer than in 2023 (36 days). </p>
<h4>Conclusions</h4>
<p><i> </i>Findings of<i> </i>this study suggest that guaranteed peer review increased submissions from ASCB members while maintaining the quality and timing of the peer review process. The data also suggest that the community values straightforward and guaranteed access to constructive peer feedback.<sup>3</sup> </p>
<h4>References</h4>
<p>1. Fei-Bloom Y, Welch M. New ASCB Member Benefit—Guaranteed Peer Review in MBoC. ASCB Member News, Publishing, Society News. December 22, 2023. Accessed July 15, 2025. <a href="https://www.ascb.org/society-news/new-member-benefit-guaranteed-peer-review-in-mboc/">https://www.ascb.org/society-news/new-member-benefit-guaranteed-peer-review-in-mboc/</a></p>
<p>2. Baum B. Reviewing papers as you would like your papers to be reviewed. <i>Mol Biol Cell</i>. 2019;30:3013-3014. doi.org/10.1091/mbc.E19-05-0273</p>
<p>3. Drubin DG. Any jackass can trash a manuscript, but it takes good scholarship to create one (how <i>MBoC</i> promotes civil and constructive peer review). <i>Mol Biol Cell</i>. 2011;22:525-527. doi.org/10.1091/mbc.e11-01-0002</p>
<sup>1</sup>American Society for Cell Biology, Rockville, MD, US, yfeibloom@ascb.org; <sup>2</sup>Department of Molecular and Cell Biology, University of California, Berkeley, Berkeley, CA, US.</p>
<h4>Conflict of Interest Disclosures </h4>
<p>None reported.</p>


Manuscript Submissions Following Implementation of Guaranteed Peer Review

<p>Douglas K. Novins,<sup>1</sup> Mary K. Billingsley,<sup>2</sup> Robert R. Althoff<sup>3</sup> </p>
<h4>Objective</h4>
<p> For many journals in science and medicine, a large number of submitted manuscripts may be rejected by editors before peer review. Given the high rates of such “desk rejections,” it is important that editorial reviews minimize bias in this decision-making process. One potential source of bias is the editor having access to information regarding the authors of manuscripts, which may result in editorial decision being influenced by authors’ academic reputation as well as other author characteristics.<sup>1,2</sup> Unfortunately, a recent review was unable to identify any empirical studies of such “triple-blind” peer review.<sup>3</sup> In this pilot study, we masked author information in the initial stage of editorial review to explore whether this changes patterns in the articles that our journal sends out for peer review.</p>
<h4>Design</h4>
<p> The <i>Journal of the American Academy of Child and Adolescent Psychiatry</i> utilizes a 2-round editor in chief (EIC) review process during which the EIC is aware of author identity in both rounds of review. In the standard round 1 review, the EIC completes a high-level reading of the manuscript and the cover letter, checks for test recycling, and reviews study preregistration. In the standard round 2 review, the EIC completes a detailed read of the manuscript. In this preregistered pretest/posttest study (<a href="https://doi.org/10.17605/OSF.IO/TNYFX">https://doi.org/ 10.17605/OSF.IO/TNYFX</a>), we followed the above standard process in the pretest. In the posttest, the identity of the authors was masked in the round 1 review and tasks that unmask author identity, such as reading the cover leader, were moved to the round 2 review. </p>
<h4>Results</h4>
<p> A total of 95 manuscripts were included in the pretest and 102 were included in the posttest. In the masked process, significantly more manuscript decisions were made in round 2 (49/102) than in the unmasked process (26/95) (χ² = 8.91; 1 degree of freedom; <i>P</i> &lt; .01) (<b>Figure 25-1112</b>). Overall, EIC decisions (reject, transfer, assignment to an action editor) did not differ between the pretest and posttest (χ² = 2.48; 2 degrees of freedom; <i>P</i> = .29). </p>

![](https://assets.underline.io/markdown_image/1/image/aafe3983a7ba8c694206726665c8c41c.png)

<h4>Conclusions </h4>
<p>The shift of decisions from round 1 to round 2 in the posttest review process is consistent with the shift of key EIC tasks to the round 2 review, resulting in more manuscripts also receiving a detailed read of their manuscript than in the pretest. </p>
<h4>References</h4>
<p>1. Fox CW, Meyer J, Aimé E. Double-blind peer review affects reviewer ratings and editor decisions at an ecology journal. <i>Funct Ecol</i>. 2023;37(5):1144-1157. doi:10.1111/1365-2435.14259</p>
<p>2. Srivastava DS, Bernardino J, Marques AT, et al. Editors are biased too: an extension of Fox et al. (2023)’s analysis makes the case for triple-blind review. Funct Ecol. 2024;38(2):278-283. doi:10.1111/1365-2435.14483</p>
<p>3. Polnaszek BE, Mei J, Cheng C, et al. Triple-blind peer review in scientific publishing: a systematic review. <i>Am J Obstet Gynecol MFM</i>. 2024;6(4):101320. doi:10.1016/j.ajogmf.2024.101320</p>
<sup>1</sup><i>Journal of the American Academy of Child and Adolescent Psychiatry</i> (University of Colorado Anschutz Medical Campus); <sup>2</sup><i>Journal of the American Academy of Child and Adolescent Psychiatry</i> (American Academy of Child and Adolescent Psychiatry); <sup>3</sup><i>Journal of the American Academy of Child and Adolescent Psychiatry</i> (University of Vermont).</p>
<h4>Conflict of Interest Disclosures</h4>
<p> Douglas K. Novins reported receiving honorarium for serving as editor in chief for the <i>Journal of the American Academy of Child and Adolescent Psychiatry</i>, American Academy of Child and Adolescent Psychiatry and grant support from the National Institutes of Health. Robert R. Althoff reported receiving honorarium for serving as associate editor for the <i>Journal of the American Academy of Child and Adolescent Psychiatry</i> from the American Academy of Child and Adolescent Psychiatry; grant support from National Institutes of Health and Klingenstein Third Generation Foundation; and stock or Equity in WISER Systems, LLC.</p>
<h4>Acknowledgment</h4>
<p> We acknowledge the contributions of our journal’s senior editorial team in designing this study.</p>


Editor Initial Manuscript Review: A Masked Pilot Study

<p>Andrea Corvillon,<sup>1</sup> John Carpenter,<sup>1</sup> Nihar B. Shah<sup>2</sup></p>
<h4>Objective </h4>
<p>As the Atacama Large Millimeter/Submillimeter (ALMA) telescope, the largest radio telescope in the world, transitioned from panel-based to distributed peer review to manage increasing proposal volumes, new challenges emerged in aligning reviewer expertise with proposal content. Building on prior work highlighting the importance of reviewer-proposal match quality, this study evaluates a machine learning and optimization framework to improve assignment fairness and accuracy.</p>
<h4>Design</h4>
<p> The new assignment process has 2 steps: (1) using machine learning to measure similarity between a proposal’s topic and a reviewer’s expertise and (2) optimizing assignments based on a fairness metric. Proposal topics were inferred using latent Dirichlet allocation (LDA), trained on ALMA proposals submitted between 2012 and 2023; the topics were known based on a set of keywords selected by the principal investigators. Reviewer expertise was estimated using the same model trained on each reviewer’s past proposals. Both were represented as topic vectors, and cosine similarity was used to assess alignment. Each proposal was assigned to 10 reviewers, and each reviewer received 10 proposals. Assignments were optimized using the PeerReview4All algorithm,<sup>1</sup> which prioritizes proposals with the lowest similarity scores and least reviewer availability, ensuring fairer matches by improving assignments for the most disadvantaged proposals. This method was compared with ALMA’s approach from 2021 to 2022, which relied on direct keyword overlap. In 2021 and 2022, 1016 and 1087 reviewers assessed 1497 and 1729 proposals, resulting in 14,970 and 17,290 assignments, respectively.</p>
<h4>Results</h4>
<p> The new method was implemented in 2023, involving 1098 reviewers and 1635 proposals. Similarity scores were validated using survey data from cycles 2021 and 2022. On average, 89% of the reviewers rated their expertise. The median similarity of assignments in which reviewers identified themselves as experts was 0.35 compared with 0.04 for nonexpert assignments, confirming the metric’s reliability. Retrospective application of the similarity metric to cycles 2021 and 2022 using the same LDA model showed that the median similarity increased from 0.20 in 2022, the final cycle of the old algorithm, to 0.71 in 2023, the first cycle of the new algorithm, demonstrating a better alignment between reviewers and proposal assignments. Every cycle, on average, 87% of reviewers rated their expertise level on their assigned proposals. With the new algorithm, the percentage of reviewers identifying themselves as experts increased from 45% to 65%, while self-identified nonexperts decreased from 10% to 5%. This tendency continued in cycle 2024, when the new algorithm was also used (<b>Figure 25-1127</b>). Additionally, the new algorithm eliminated manual reassignments, reducing manual effort time from 3 to 5 days to nearly zero. These findings will be updated with analysis of new data from 2024 and 2025.</p>

![](https://assets.underline.io/markdown_image/1/image/1bbe2eac526f172fa408e3aeb5da040a.png)

<h4>Conclusions </h4>
<p>ALMA’s machine learning–based assignment framework improved reviewer-proposal alignment, increased expertise match rates, and eliminated manual reassignments.</p>
<h4>Reference</h4>
<p>1. Stelmakh I, Shah N, Singh A. PeerReview4All: fair and accurate reviewer assignment in peer review. <i>J Machine Learning Res</i>. 2021;22(163):1-66. Accessed July 16, 2025. <a href="https://jmlr.csail.mit.edu/papers/volume22/20-190/20-190.pdf"><span class="Hyperlink CharOverride-6">https://jmlr.csail.mit.edu/papers/volume22/20-190/20-190.pdf</span></a></p>
<sup>1</sup>Joint ALMA Observatory, Santiago, Chile, andrea.corvillon@alma.cl; <sup>2</sup>Carnegie Mellon University, Pittsburgh, PA, US.</p>
<h4>Conflict of Interest Disclosures</h4>
<p> Andrea Corvillon and John Carpenter are employed by the Joint ALMA Observatory (JAO), which is jointly managed by Associated Universities, Inc/National Radio Astronomy Observatory, the European Organization for Astronomy Research in the Southern Hemisphere (ESO), and the National Astronomical Observatory of Japan (NAOJ) on behalf of the ALMA partnership. Nihar B. Shah is employed by the Carnegie Mellon University and is a member of the Peer Review Congress Advisory Board but was not involved in the editorial review or decision for this abstract.</p>
<h4>Funding</h4>
<p> The study was funded by grant 1942124 from the National Science Foundation (NSF) to the JAO.</p>
<h4>Role of Funder/Sponsor</h4>
<p> This work is carried out as part of the duties of the authors at the Joint ALMA Observatory, which receives funding from the NSF through NRAO. The NSF did not contribute to the analysis presented in this contribution.</p>
<h4>Acknowledgments</h4>
<p> ALMA is a partnership of ESO (representing its member states), NSF (US), and the National Institute of Natural Sciences (Japan), together with the National Research Council of Canada (Canada), the National Science and Technology Council and the Institute of Astronomy &amp; Astrophysics, Academia Sinica (Taiwan), and the Korea Astronomy and Space Science Institute (Republic of Korea), in cooperation with the Republic of Chile.</p>


Optimizing Proposal Assignments in the Distributed Peer Review System of the World's Largest Radio Telescope Observatory

<p>Pravin Bolshete,<sup>1</sup> Madhulika Bolshete,<sup>1</sup> Priyanka Mate<sup>1</sup></p>
<h4>Objective</h4>
<p> To assess the current status of journals and publishers formerly listed in Beall’s List, including their operational status, recent publication activity, and indexing in recognized databases.</p>
<h4>Design</h4>
<p> A systematic evaluation of journals and publishers listed in Beall’s List was conducted. The list was accessed from <a href="https://beallslist.net/"><span class="Hyperlink CharOverride-6">https://beallslist.net/</span></a>, considering Beall’s original list has been defunct since January 2017. The current version of the list is hosted anonymously and was used solely as a tool to access the archived entities listed by Beall, excluding updates made after 2017. Each website link was checked for functionality, and the operational status of the respective journal or publisher was recorded from November to December 2024. Among journals that remained active, we further assessed whether they had published articles in 2024 and their indexing status, listed on their website, in databases such as PubMed/Medline, Scopus, Embase, Directory of Open Access Journals, and Web of Science.</p>
<h4>Results</h4>
<p> Of the total number of journals (N = 1310) and publishers (N = 1163) assessed, 617 (47.1%) and 624 (53.7%), respectively, were found to be defunct, with their websites no longer accessible. Among the journals that remained active<span class="CharOverride-4"> </span>(n = 693 [52.9%]), 461 (66.5%) had published articles in 2024. Indexing analysis revealed that 154 (22.2%) of these journals mentioned that they are listed in at least 1 recognized academic database.</p>
<h4>Conclusions</h4>
<p> A significant proportion of predatory journals and publishers listed in Beall’s List have ceased operations, yet a substantial number remain active and continue to publish research. Some of these journals have managed to be indexed in reputed databases, raising concerns about the persistence and evolving strategies of predatory publishing. The operational status of a journal and its indexing do not confirm or refute its current quality. Some journals may have reformed and implemented peer review and ethical publishing practices. However, we cannot confirm whether active journals have improved, remained predatory, or now operate under misleadingly enhanced credibility. Additionally, although the use of Beall’s archived list offers a consistent reference point, the anonymous nature of its current hosting presents limitations. Overall, these findings underscore the need for ongoing scrutiny and awareness required to safeguard academic integrity and prevent researchers from inadvertently submitting their work to predatory journals.</p>
<sup>1</sup>Sqarona Medical Communications, Pune, India, pravinbolshete@gmail.com.</p>
<h4>Conflict of Interest Disclosures</h4>
<p> All authors are affiliated with Sqarona Medical Communications, which provides medical writing services.</p>


Persistence and Indexing of Predatory Journals and Publishers: A Follow-Up Evaluation of Beall's List

<p>Marija Purgar,<sup>1,2</sup> Edward R. Ivimey-Cook,<sup>3</sup> Antica Culina,<sup>1,4</sup> Joshua D. Wallach<sup>2,5</sup></p>
<h4>Objective</h4>
<p> Preprints—preliminary research reports that have not yet undergone peer review—are becoming increasingly common across scientific fields.<sup>1</sup> However, little is known about preprint policies in journals that publish ecology and evolutionary biology research.<sup>2</sup></p>
<h4>Design</h4>
<p> In this cross-sectional analysis, we identified all journals included under the Web of Science 2023 Journal Citation Reports categories of Ecology and Evolutionary Biology. We reviewed journal and publisher websites to explore specific preprint policies, including guidelines regarding the possible locations for posting preprints (eg, on a preprint server), disclosure of preprints at time of manuscript submission and the required location (eg, in the cover letter), timing of preprint posting relative to manuscript submission (eg, before submission to a journal), and any additional actions on publication of the final article (eg, linking the preprint to the published version). The journal-level and publisher-level policies were classified as preprints allowed without restrictions, preprints considered on a case-by-case determination (manuscripts with preprints evaluated by the journal on an individual basis), or preprints prohibited. Journals with no identified policy were categorized as having no preprint policy. We used a Wilcoxon rank sum test to compare 2023 Journal Impact Factors between journals with a preprint policy at either journal- or publisher-level and those without any preprint policy.</p>
<h4>Results</h4>
<p> We identified 230 eligible ecology and evolutionary biology journals published by 69 different publishers, of which 119 (51.7%) included preprint policies in their author guidelines—either through journal-specific policies (n = 109) or by directly referencing their publisher’s preprint policies (n = 10). There were 73 additional journals (31.7%) without their own preprint policies, which did not directly reference their publisher’s preprint policies but were associated with publishers that had favorable policies. Overall, there were 38 journals (16.5%) without any journal-level or publisher-level preprint policies. Of the 192 journals with either journal-level or publisher-level preprint policies, 191 (99.5%) explicitly allowed preprints and 1 (0.5%) considered preprints on a case-by-case basis. The median (IQR) Journal Impact Factor was higher among journals with a journal-level or publisher-level preprint policy (2.4 [1.7-3.7]) compared with those without any preprint policy (0.6 [0.5-0.9]) (<i>P</i> &lt; .001). Among the 109 journals with journal-level preprint policies (<b>Table 25-1002</b>), 62 (56.9%) provided information about specific platforms (eg, bioRxiv, SSRN, arXiv), 24 (22.0%) required disclosure of preprints during manuscript submission, and 35 (32.1%) provided specific guidance on the timing of preprint posting relative to manuscript submission. </p>

![](https://assets.underline.io/markdown_image/1/image/9a918b235beaea8ae656fcd347310f00.png)

<h4>Conclusions</h4>
<p> In this study, only one-half of identified ecology and evolutionary biology journals provided preprint policies in their journal-level author guidelines or offered clear instructions regarding the use of preprints. This lack of clarity may lead to confusion among authors about whether posting a preprint before submitting to a peer-reviewed journal conflicts with the journal’s policies on prior publication. </p>
<h4>References</h4>
<p>1. Abdill RJ, Blekhman R. Meta-research: tracking the popularity and outcomes of all bioRxiv preprints. <i>eLife</i>. 2019;8:e45133. doi:10.7554/eLife.45133</p>
<p>2. Noble DW, Xirocostas ZA, Wu NC, et al. The promise of community-driven preprints in ecology and evolution. <i>EcoEvoRxiv</i>. Preprint posted online June 13, 2024. doi:10.32942/X2SS46</p>
<sup>1</sup>Division for Marine and Environmental Research, Ruđer Bošković Institute, Zagreb, Croatia; <sup>2</sup>Department of Epidemiology, Rollins School of Public Health, Emory University, Atlanta, GA, US, joshua.wallach@emory.edu; <sup>3</sup>School of Biological Sciences, One Health &amp; Veterinary Medicine, University of Glasgow, Scotland, UK; <sup>4</sup>Netherlands Institute of Ecology, Royal Netherlands Academy of Arts and Sciences, Wageningen, the Netherlands; <sup>5</sup>Collaboration for Regulatory Rigor, Integrity, and Transparency, Yale School of Medicine, New Haven, CT, US.</p>
<h4>Conflict of Interest Disclosures</h4>
<p> Marija Purgar, Edward R. Ivimey-Cook, and Antica Culina are members of the <i>Society for Open, Reliable, and Transparent Ecology and Evolutionary Biology</i>; Marija Purgar and Edward R. Ivimey-Cook are on the current Board of Directors. Joshua D. Wallach reports funding from Arnold Ventures to the Yale Collaboration for Regulatory Rigor, Integrity, and Transparency, the National Institute on Alcohol Abuse and Alcoholism of the National Institutes of Health under award 1K01AA028258, Johnson &amp; Johnson (through the Yale Open Data Access Project), and the US Food and Drug Administration. No other disclosures were reported. </p>
<h4>Funding/Support </h4>
<p>Marija Purgar was funded by the Croatian Science Foundation project number DOK-2021-02-6688. Marija Purgar gratefully acknowledges the financial support for this research by the Fulbright US Student Program, which is sponsored by the US Department of State and the Croatian-American Fulbright Commission. Antica Culina was funded by the Croatian Science Foundation project number IP-2022-10-2872.</p>
<h4>Additional Information</h4>
<p> Marija Purgar is a co–corresponding author (mpurgar@irb.hr).</p>


Preprint Policies in Ecology and Evolutionary Biology Journals

<p>Sherrie L. Kelly,<sup>1</sup> Clare F. Stone,<sup>2</sup> Catherine Fiscus,<sup>2</sup> Ashlie Jackman-Juler,<sup>2</sup> Miriam Lewis Sabin<sup>1</sup></p>
<h4>Objective</h4>
<p> Previous research has highlighted the importance of preprints in increasing the transparency and rapid dissemination of scientific communication. This study examines publication rates of manuscripts submitted to Preprints With <i>The Lancet </i>(PPwTL), a collaboration between The Lancet Group and SSRN, Elsevier’s preprint server. </p>
<h4>Design</h4>
<p> This quality improvement study had 3 phases. First, we assessed the number of preprints submitted, posted, and removed from PPwTL from January 1, 2018, to December 1, 2024. Second, we focus on the number of preprints submitted in this period where a version of record (VOR) can be found. Finally, we surveyed corresponding authors of preprints without publications at Elsevier to explore potential barriers to publication.</p>
<h4>Results</h4>
<p> From 2018 to 2024, among The Lancet Group’s 24 journals, 35,703 preprints were submitted to PPwTL with annual submissions increasing by 183% from 2089 in 2018 to 5915 in 2024 (<b>Figure 25-1109</b>). The annual acceptance rate declined slightly, dropping to 67% in 2024 (3987 accepted of 5915 submitted) from 72% in 2018 (1509 of 2089), while the rate of removals at the authors’ request reached 17% in 2024 (686 removed of 3987 posted), compared with 9% in 2018 (130 of 1509). Throughout this period, the average time from submission to posting was 2 days, including our screening processes. We ran an internal report to identify how many of the 35,703 preprints that were submitted have a VOR in Science Direct (Elsevier publications). We identified 18,543 (51.9%) submissions with an Elsevier VOR and 17,160 (48.0%) with no VOR in Science Direct. To identify publications from PPwTL submissions that occur in non-Elsevier journals, we cross-checked the 17,160 papers with an internal Fate of Manuscript FORMS report, which lists VORs outside of Elsevier. This report revealed 2639 of the initial 35,703 submissions (7.4%) had non-Elsevier publications in the period 2020 to 2024. The combined numbers of papers with a VOR (18,543 in Elsevier journals and 2,639 outside of Elsevier) is 21,182 and represents an overall publication rate of 56.3% for PPwTL submissions. This number is likely to be higher, because the FORMS report covers publications from 2020 to 2024 and the initial submissions data cover 2018 to 2024. Of the 2639 publications in non-Elsevier journals, 507 (19.2%) were published in Springer Nature journals; 145 (5.5%) in PLOS journals; 146 (5.5%) in BMJ journals, and 25 (0.9%) in JAMA Network journals. Findings from the ongoing author survey are anticipated in the coming months.</p>

![](https://assets.underline.io/markdown_image/1/image/2a93036b341819b667d4246a28d1b5e3.png)

<h4>Conclusions</h4>
<p> It is important to understand the impact of preprints on the scholarly record, and it’s encouraging to note the increased acceptance of preprints. A significant number of PPwTL manuscripts that do not get published in Elsevier journals nevertheless are published elsewhere, benefitting the scholarly community and strengthening the preprint/journal landscape. </p>
<sup>1</sup>The Lancet Group, Elsevier, London, UK; <sup>2</sup>SSRN, Elsevier, Rochester, NY, US, c.stone@elsevier.com.</p>
<h4>Conflicts of Interest Disclosures</h4>
<p> None reported.</p>
<h4>Acknowledgments</h4>
<p> The authors express their gratitude to current and former members of the Preprints With<i>The Lancet</i>-SSRN Working Group, to Aschwin Wijnsma (Elsevier) who assisted with the FORM report used to examine the fate of manuscripts in this study, and to authors who contributed to Preprints With <i>The Lancet</i> and responded to the survey with consent.</p>


Authors' Journeys From Preprints With The Lancet on SSRN to Publication

<p>Julianne T. Nelson,<sup>1</sup> Tony Tse,<sup>1</sup> Swapna Mohan,<sup>1</sup> Yvonne Puplampu-Dove<sup>1</sup></p>
<h4>Objective</h4>
<p> The 2004 International Committee of Medical Journal Editors (ICMJE) policy required prospective registration of clinical trials as a condition for publication. Prior work reported widespread inconsistencies at earlier time points, including incomplete registration,<sup>1</sup> retrospective or unregistered trials,<sup>2</sup> and discrepancies between registered and published primary outcome measures.<sup>3</sup> This analysis aims to build on previous work, now 20 years post-ICMJE policy, and expands to include the global landscape of trial publication and registration.</p>
<h4>Design</h4>
<p> This cross-sectional analysis assessed a convenience sample of primary trial results publications first indexed in PubMed from July to September 2024 that was randomly selected by funding category to reflect a distribution of 40% industry and 60% nonindustry.<sup>2</sup> Each article was manually reviewed by 1 author. The journal title, first publication date, and any unique identifier from an ICMJE-recognized registry listed in the abstract or body text were extracted. Publications without trial registry information had their status confirmed via advanced search within the WHO International Clinical Trials Registry Platform (ICTRP). From a random subset of 100 sampled publications listing registry identifiers, again selected randomly for a 40% to 60% funding distribution, 1 author extracted first registration and study start dates from corresponding registration records to determine timing of registration (prospective or retrospective). The author also compared primary outcome measures (POMs) listed in publications and records. The first outcome measure mentioned was used for publications without explicitly identified POMs. Discrepancy codes were adapted from prior work.<sup>3</sup> Registered POMs with vague terms (eg, safety) were coded as “unclear.” All data were reviewed independently by a second author and differences resolved by discussion.</p>
<h4>Results</h4>
<p> Of 434 overall sampled articles from 322 journal titles on October 17, 2024, 82.7% (359 of 434) disclosed registration identifiers, of which 212 (59.1%) were listed in abstracts. Of 17 represented registries, ClinicalTrials.gov (69.6% [250 of 359]), ChiCTR (7.0% [25 of 359]), and ANZCTR (3.6% [13 of 359]) were cited most frequently. Among the subset of 100 trials, 23% were retrospectively registered (<b>Table 25-0939</b>). Among the extracted 135 POMs, 2.2% (3 of 135) were coded as unclear and 18.5% (25 of 135) as discordant: 52.0% (13 of 25) due to published POMs missing from registration records and 28.0% (7 of 25) due to published POMs registered as secondary outcome measures. No substantive differences between trials funded by industry and nonindustry sources were observed.</p>

![](https://assets.underline.io/markdown_image/1/image/7b295512bccfe7c6b1b8e3e25d99b948.png)

<h4>Conclusions</h4>
<p> Twenty years after the ICMJE policy, nearly one-fifth of sampled publications did not disclose registration identifiers. Of those that did, 40% of identifiers were not displayed in abstracts (inaccessible in PubMed). Among the subset, almost one-quarter were registered retrospectively. Of published POMs, approximately one-fifth were discordant with registered information. These findings do not differ widely from previous work of 10 years ago and suggest that the policy goals of full transparency and accountability through trial registration have yet to be reached.</p>
<h4>References</h4>
<p>1. Viergever RF, Karam G, Reis A, Ghersi D. The quality of registration of clinical trials: still a problem. <i>PLoS One</i>. 2014;9(1):e84727. doi:10.1371/journal.pone.0084727</p>
<p>2. Gopal AD, Wallach JD, Aminawung JA, et al. Adherence to the International Committee of Medical Journal Editors’ (ICMJE) prospective registration policy and implications for outcome integrity: a cross-sectional analysis of trials published in high-impact specialty society journals. <i>Trials</i>. 2018;19(1):448. doi:10.1186/s13063-018-2825-y</p>
<p>3. Turner EH, Mulder RT, Rucklidge JJ. Is mandatory prospective trial registration working? An update on the adherence to the International Committee of Medical Journal Editors guidelines across five psychiatry journals: 2015-2020. <i>Acta Psychiatr Scand</i>. 2021;144(5):510-517. doi:10.1111/acps.13353</p>
<sup>1</sup>The National Center for Biotechnology Information, National Library of Medicine, National Institutes of Health, Bethesda, MD, US, julianne.nelson@nih.gov. </p>
<h4>Conflict of Interest Disclosures</h4>
<p> None reported.</p>
<h4>Funding/Support</h4>
<p> This work was supported by the National Center for Biotechnology Information of the National Library of Medicine, National Institutes of Health.</p>
<h4>Role of the Funder/Sponsor</h4>
<p> The funder supported the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the abstract; and decision to submit the abstract for presentation.</p>
<h4>Disclaimer </h4>
<p>The views expressed in this article are those of the authors and do not necessarily reflect the views or policies of the National Institutes of Health.</p>
<h4>Additional Information </h4>
<p>Swapna Mohan and Yvonne Puplampu-Dove reported that they conducted this work under contract with ICF International Inc.</p>


Impact of ICMJE Trial Registration Policy at 20 Years

<p>Olmo R. van den Akker,<sup>1,2</sup> Marcel A. L. M. van Assen,<sup>1</sup> Marjan Bakker,<sup>1</sup> Jelte M. Wicherts<sup>1</sup></p>
<h4>Objective </h4>
<p>Preregistration,<sup>1</sup> the practice in which researchers publish their hypotheses, study design, and/or analysis plan before collecting or analyzing their data, has become more and more established in the scientific ecosystem even though the evidence that it prevents bias has been lacking. This set of studies was conducted to assess the effectiveness of preregistration to reduce bias in psychology.</p>
<h4>Design </h4>
<p>In the first study, conducted from 2021 to 2024, we assessed the consistency of preregistrations and their corresponding publications (published in 40 psychology journals). These preregistered publications were published and awarded a preregistration badge by a journal or a preregistration challenge prize by the Center for Open Science from 2017 to 2020. We also assessed whether any inconsistencies were disclosed by the authors. In the second study, we compared a subset of preregistration-publication pairs for which a preregistered statistical result could be extracted with a set of nonpreregistered studies (from 37 journals) that were selected for comparability via the Web of Science related records function. Using multilevel regressions, we examined whether preregistration was associated with <i>P</i> hacking, operationalized as a lower proportion of statistical results and smaller effect sizes. We also looked at associations with other research characteristics, such as the number of citations, sample sizes, and the frequency of statistical errors (assessed via Statcheck<sup>2</sup>).</p>
<h4>Results </h4>
<p>The sample included 300 preregistration-publication pairs as well as a subset of 193 preregistration-publications pairs matched with 193 nonpreregistered studies. We found inconsistencies for 6 important study elements (operationalizations of measured, manipulated, and dependent variables, data collection procedures, statistical model, and inference criteria [<b>Table 25-0947</b>]). The comparison between preregistered and nonpreregistered studies did not indicate an association between preregistration and statistical significance (β = 0.01; 99% CI, −0.56 to 0.59; <i>P</i> = .96), effect sizes (β = −0.04; 99% CI, −0.12 to 0.04; <i>P</i> = .18), and statistical errors (β = −1.19; 95% CI, −2.51 to 0.13; <i>P</i> = .08). However, preregistered studies typically had higher sample sizes (959.6 vs 536.6; β = 0.45; 99% CI, 0.14 to 0.76; <i>P</i> &lt; .001) and more citations (18.3 vs 15.1; β = 0.20; 95% CI, 0.01 to 0.40; <i>P</i> = .04).</p>

![](https://assets.underline.io/markdown_image/1/image/f985f58c82af47fcce202e540e6513e4.png)

<h4>Conclusions </h4>
<p>Ideally, publications would be concordant with their preregistrations and any deviations would be reported and explained. In these studies, this ideal was often not met. Preregistration was associated with higher statistical power and more subsequent citations, but it did not appear to prevent <i>P</i> hacking. The studies were limited by their focus on psychology, and generalizations to other fields should therefore be avoided (although economics and political science had comparable results<sup>3</sup>).</p>
<h4>References</h4>
<p>1. Rice DB, Moher D. Curtailing the use of preregistration: a misused term. <i>Perspect Psychol Sci</i>. 2019;14(6):1105-1108. doi:10.1177/1745691619858427</p>
<p>2. Nuijten MB, Hartgerink CH, Van Assen MA, Epskamp S, Wicherts JM. The prevalence of statistical reporting errors in psychology (1985-2013). <i>Behav Res Methods</i>. 2016;48:1205-1226. doi:10.3758/s13428-015-0664-2</p>
<p>3. Ofosu GK, Posner DN. Pre-analysis plans: an early stocktaking. <i>Perspect Polit</i>. 2023;21(1):174-190. doi:10.1017/S1537592721000931</p>
<sup>1</sup>Tilburg University, Tilburg, The Netherlands, ovdakker@gmail.com; <sup>2</sup>QUEST Center for Responsible Research, Berlin Institute of Health at Charité, Berlin, Germany.</p>
<h4>Conflict of Interest Disclosures</h4>
<p> None reported.</p>
<h4>Funding/Support</h4>
<p> The work in this presentation was supported by a Consolidator Grant (IMPROVE) from the European Research Council (grant number 726361).</p>
<h4>Additional Information</h4>
<p> Both studies presented in this abstract were preregistered: the assessment of preregistration effectiveness at <a href="https://osf.io/qbhsv)"><span class="Hyperlink CharOverride-6">https://osf.io/qbhsv</span></a>, and the comparison of preregistered with nonpreregistered studies at <a href="https://osf.io/7w5g2"><span class="Hyperlink CharOverride-6">https://osf.io/7w5g2</span></a>.</p>


Effectiveness of Preregistration in Psychology

<p>Cecilie Jespersen,<sup>1,2</sup> Zexing Song,<sup>3</sup> An-Wen Chan,<sup>3,4</sup> Asbjørn Hróbjartsson<sup>1,2</sup></p>
<h4>Objective </h4>
<p>Observational studies of interventions use causal inference to assess the impact of interventions on health-related outcomes.<sup>1</sup> Despite concerns about reporting bias, observational studies are not subject to the same registration requirements as clinical trials.<sup>2,3</sup> We aimed to determine the prevalence of registration among published observational studies of interventions, assess the association between registration and study characteristics, analyze journal registration policies, and explore authors’ and editors’ attitudes about registration.</p>
<h4>Design </h4>
<p>We conducted a meta-epidemiologic cross-sectional study triangulating data from 4 sources. First, we searched PubMed for observational studies published in 2023. Eligible studies were cohort or case-control studies with a control group that assessed causal effects of health interventions. Corresponding registration information was collected. Second, authors of included studies were surveyed to explore reasons for and barriers to registration. Third, editorial policies were sampled from 40 journals: 20 sample-representative journals and the journals ranked in the top 20 in Journal Citation Reports by 2023 Journal Impact Factor across 8 specialty categories. Fourth, 1 editor per journal was invited to share their perspectives on registration. Primary outcomes were the prevalence of registered observational studies of interventions published in 2023 and the estimated association between registration and study characteristics, assessed by multivariable logistic regression. Sample size was estimated based on an expected 15% registration rate.</p>
<h4>Results </h4>
<p>Among 1100 screened studies, 200 were included: 69 and 128 cohort studies with prospective and retrospective data collection, respectively, and 3 case-control studies. In total, 28 (14%) were registered, and 17 of these (61%) were prospectively registered (&lt;1 month of their start date) (<b>Table 25-1069</b>). Prospective design and protocol availability were positively associated with registration (retrospective vs prospective cohort: odds ratio [OR], 0.19 [95% CI, 0.07-0.54]; <i>P</i> = .002; no public protocol vs public protocol: OR, 0.04 [95% CI, 0.01-0.23]; <i>P</i> &lt; .001). The survey response rate was 23% (46 responses); 60% of authors supported registration, although many only when registration was deemed relevant. Identified barriers included lack of journal requirements for registration (56%) and limited resources (62%). None of the journal policies explicitly required registration of observational studies of interventions, while 12 (30%) encouraged it. Journals that encouraged registration had a higher 2023 Journal Impact Factor and more frequently encouraged public protocols. Editors had divergent opinions on registration. While some considered it to be worthwhile, just as many questioned the added value.</p>

![](https://assets.underline.io/markdown_image/1/image/99ddb5f59ea342d2117443673974b4db.png)

<h4>Conclusions </h4>
<p>Only 1 in 7 contemporary observational studies of interventions were registered, although more often in cohort studies with prospective data collection and studies with a publicly available protocol. Authors identified the lack of journal requirements to registration as a key registration barrier, and only one-third of journals had supportive policies. Clearer guidance and journal policies on registration relevance (discriminating hypothesis-testing and hypothesis-generating studies) may reduce the risk of reporting biases in observational studies of interventions.</p>
<h4>References</h4>
<p>1. Hernán MA, Wang W, Leaf DE. Target trial emulation: a framework for causal inference from observational data. <i>JAMA</i>. 2022;328(24):2446-2447. doi:10.1001/jama.2022.21383</p>
<p>2. Williams RJ, Tse T, Harlan WR, Zarin DA. Registration of observational studies: is it time? <i>CMAJ</i>. 2010;182(15):1638-1642. doi:10.1503/cmaj.09225</p>
<p>3. Leducq S, Zaki F, Hollestein LM, et al. The majority of observational studies in leading peer-reviewed medicine journals are not registered and do not have a publicly accessible protocol: a scoping review. <i>J Clin Epidemiol</i><span lang="da-DK">. 2024;170:111341. doi:10.1016/j.jclinepi.2024.111341</span></p>
<sup>1</sup>Cochrane Denmark &amp; Centre for Evidence-Based Medicine Odense (CEBMO), University of Southern Denmark, Odense, Denmark, ceciliejespersen@health.sdu.dk; <sup>2</sup>Open Patient data Explorative Network (OPEN), Odense University Hospital, Odense, Denmark; <sup>3</sup>Institute of Health Policy, Management and Evaluation, University of Toronto, Toronto, Ontario, Canada; <sup>4</sup>Women’s College Research Institute, Dept. of Medicine, University of Toronto, Toronto, Ontario, Canada.</p>
<h4>Conflict of Interest Disclosures</h4>
<p> An-Wen Chan is a member of the Peer Review Congress Advisory Board but was not involved in the review or decision for this abstract. No other disclosures were reported.</p>
<h4>Acknowledgments</h4>
<p> We thank all researchers who participated in the author survey for their valuable contribution to the findings of this study.</p>


Next from The 10th Peer Review Congress

Efficiency of Author Anonymization in Peer Review

Similar lecture

Comparison of Content in Published and Unpublished Peer Review Reports