United States

&lt;p&gt;Franki Y. H. Kung,&lt;sup&gt;1&lt;/sup&gt; Mariam Aly,&lt;sup&gt;2&lt;/sup&gt; Shahana Ansari,&lt;sup&gt;3&lt;/sup&gt; Eliana Colunga,&lt;sup&gt;4&lt;/sup&gt; M. J. Crockett,&lt;sup&gt;5,6&lt;/sup&gt; Amanda B. Diekman,&lt;sup&gt;3&lt;/sup&gt; Pablo Gomez,&lt;sup&gt;7&lt;/sup&gt; Paul C. McKee,&lt;sup&gt;8&lt;/sup&gt; Miriam Pérez,&lt;sup&gt;9&lt;/sup&gt; Sarah M. Stilwell,&lt;sup&gt;10&lt;/sup&gt; Matthew Goldrick&lt;sup&gt;11,12&lt;/sup&gt;&lt;/p&gt;

&lt;h4&gt;Objective&lt;/h4&gt; Accumulating data suggest peer reviews can exhibit bias&lt;i&gt; &lt;/i&gt;(ie, systematically devaluing certain topics, methods, or the study of nondominant populations&lt;sup&gt;1&lt;/sup&gt;) and can be unfair with respect to process (eg, decisions are made inconsistently) or people (eg, authors are disrespected).&lt;sup&gt;2&lt;/sup&gt; Recent studies have shown that biases and unfair behaviors in peer review tend to undermine the self-reported experiences and career progression of researchers, including those from underserved populations, such as women, nonbinary individuals, and people from racial and ethnic groups other than White.&lt;sup&gt;1-3&lt;/sup&gt; This study examined whether peer review exercises increased psychology and neuroscience trainees’ perceived readiness to respond to biased and unfair reviewers.&lt;/p&gt;

&lt;h4&gt;Design &lt;/h4&gt;Single-session synchronous online trainings (conducted during 2023-2024) allowed for trainer-trainee and trainee-trainee interactions. Trainees were recruited for this study through social media, targeted academic organizations, and graduate programs. Trainers (professors and a postdoctoral fellow with multiple years of peer review experience) reviewed processes occurring between journal submission and receipt of a decision letter. Trainees were then given a simplified, anonymized decision letter and reviews. Exercises and discussions illustrated how trainees could plan manuscript revisions and compose responses to critiques. Finally, trainees were given examples of biased and unfair reviews, along with strategies for responding to them while acknowledging the power dynamics inherent to review. Discussion included how to maintain motivation when encountering such reviews. Participants were asked to complete surveys before and after online training in which they rated their ability to perform the following targeted skills on a scale from 1 (very little skill) to 5 (great deal of skill): define and identify bias in peer reviews; define and identify unfair language in peer reviews; respond to reviews using fair language; respond to biased or unfair reviews; and promote an inclusive and equitable peer-review culture.&lt;/p&gt;

&lt;h4&gt;Results &lt;/h4&gt;A total of 64 participants (42 PhD students [66%], 10 predoctoral students [16%], 8 postdoctoral trainees [13%], and 4 other trainees [5%]) rated their level of skill for 5 targeted skills. Ordinal regressions showed participants had significantly higher ratings for all skills at posttest. For example, the mean (SD) self-ratings for the skill to respond to biased on unfair reviews increased from 2.0 (1.01) on the pretest to 4.09 (0.89) on the posttest (&lt;b&gt;Table 25-0893&lt;/b&gt;).&lt;/p&gt;

![](https://assets.underline.io/markdown_image/1/image/f801de28833ed0c4470dce4f2815a32b.png)

&lt;h4&gt;Conclusions &lt;/h4&gt;
&lt;p&gt;After these peer review exercises, psychology and neuroscience trainees had an increase in their perceived readiness to confront biased and unfair reviews. Work is ongoing to scale up this training using on-demand materials and further assess training impact, including measures beyond trainee self-reports.&lt;/p&gt;
&lt;h4&gt;References&lt;/h4&gt;
&lt;p&gt;1. Rogers LO, Moffitt U, McLean KC, Syed M. Research as resistance: naming and dismantling the master narrative of “good” science. &lt;i&gt;Am Psychol&lt;/i&gt;. 2024;79:484-496. doi:10.1037/amp0001246&lt;/p&gt;
&lt;p&gt;2. Silbiger NJ, Stubler AD. Unprofessional peer reviews disproportionately harm underrepresented groups in STEM. &lt;i&gt;PeerJ&lt;/i&gt;. 2019;7:e8247. doi:10.7717/peerj.8247&lt;/p&gt;
&lt;p&gt;3. Aly M, Colunga E, Crockett MJ, et al. Changing the culture of peer review for a more inclusive and equitable psychological science. &lt;i&gt;J Exp Psychol Gen&lt;/i&gt;. 2023;152:3546-3565. doi:10.1037/xge0001461&lt;/p&gt;
&lt;sup&gt;1&lt;/sup&gt;Department of Psychological Sciences, Purdue University, West Lafayette, IN, US; &lt;sup&gt;2&lt;/sup&gt;Department of Psychology, University of California Berkeley, Berkeley, CA, US; &lt;sup&gt;3&lt;/sup&gt;Department of Psychological and Brain Sciences, Indiana University Bloomington, Bloomington, IN, US; &lt;sup&gt;4&lt;/sup&gt;Department of Psychology and Neuroscience, University of Colorado Boulder, City, CO, US; &lt;sup&gt;5&lt;/sup&gt;Department of Psychology, Princeton University, Princeton, NJ, US; &lt;sup&gt;6&lt;/sup&gt;Department of Psychology, University Center for Human Values, Princeton University, Princeton, NJ, US; &lt;sup&gt;7&lt;/sup&gt;Department of Psychology, Skidmore College, Saratoga Springs, NY, US; &lt;sup&gt;8&lt;/sup&gt;Department of Psychology and Neuroscience, Duke University, Durham, NC, US; &lt;sup&gt;9&lt;/sup&gt;Department of Psychology, North Park University, Chicago, IL, US; &lt;sup&gt;10&lt;/sup&gt;Department of Health Behavior and Health Equity, University of Michigan, Ann Arbor, MI, US; &lt;sup&gt;11&lt;/sup&gt;Department of Linguistics, Northwestern University, Evanston, IL, US, matt-goldrick@northwestern.edu; &lt;sup&gt;12&lt;/sup&gt;Cognitive Science Program, Northwestern University, Evanston, IL, US.&lt;/p&gt;
&lt;h4&gt;Conflict of Interest Disclosures&lt;/h4&gt;
&lt;p&gt; None reported.&lt;/p&gt;
&lt;h4&gt;Funding/Support&lt;/h4&gt;
&lt;p&gt; This work was supported by grants from the National Science Foundation (DGE-2224777, DGE-2224779, DGE-2436430). &lt;/p&gt;
&lt;h4&gt;Role of the Funder/Sponsor&lt;/h4&gt;
&lt;p&gt; The funder of the study had no role in design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the abstract; and decision to submit the abstract for presentation.&lt;/p&gt;
&lt;h4&gt;Additional Information&lt;/h4&gt;
&lt;p&gt; Training materials are freely available (&lt;a href=&quot;https://osf.io/658ev&quot;&gt;&lt;span class=&quot;Hyperlink CharOverride-6&quot;&gt;https://osf.io/658ev&lt;/span&gt;&lt;/a&gt;).&lt;/p&gt;


The 10th Peer Review Congress

Peer Review Exercises to Enhance Trainees&#39; Readiness to Confront Unfair or Biased Reviews

instructions for authors

diversity and inclusion

bias

The Tenth Peer Review Congress featured 3 days for presentations of new research into peer review and all aspects of scientific publication, bias, quality of reporting, and information access and dissemination.

This year’s meeting was hybrid, and all plenary sessions were livestreamed with opportunities to view presentations and engage with participants in person and virtually during and after the meeting. All recordings are now available on Underline for viewing.

<div style="position:relative;padding-top:max(60%,324px);width:100%;height:0;"><iframe style="position:absolute;border:none;width:100%;height:100%;left:0;top:0;" src="https://online.fliphtml5.com/ebtyf/mget/"  seamless="seamless" scrolling="no" frameborder="0" allowtransparency="true" allowfullscreen="true" ></iframe></div>


#### Why? 
Our aim is to encourage research into the quality and credibility of peer review and scientific publication, to establish the evidence base on which scientists can improve the conduct, reporting, and dissemination of scientific research.

#### Who? 
Participants include editors and publishers of scientific peer-reviewed journals, researchers, funders, bibliometric and informatics experts, information innovators, librarians, journalists, policymakers, ethicists, scientific information disseminators, and anyone interested in the progress of the scientific information enterprise and the quality of scientific evidence.

The Congress embraces a wide range of disciplines, including biomedicine, health and life sciences, applied sciences, basic sciences, physical and chemical sciences, mathematics, computer sciences, engineering, economics, and social sciences as well as new and emerging disciplines.

#### What? 
The Congress provides a forum for the presentation and discussion of new research into the quality and credibility of editorial and funding peer review, scientific and scholarly publication, research methods and reporting practices, identification and management of biases, transparency, reproducibility, access, publication and dissemination metrics, identification and management of threats to the integrity of science, and advances in information exchange.

**Follow us**

[![](https://assets.underline.io/markdown_image/1/image/662a22766b9788ed3766cb23cd156e32.png)](https://x.com/peerrevcongress)  [![](https://assets.underline.io/markdown_image/1/image/bb105afb1773a01fa169aa97ca1af49a.png)](https://www.youtube.com/channel/UCsG_0nKi3_Zfyv0yRPdxjWQ)  [![](https://assets.underline.io/markdown_image/1/image/034e1e70a194a9a4178fdf470ef796db.png)](https://bsky.app/profile/peerreviewcongress.bsky.social)



You need to log in with the email address you registered with. 



Please log in to explore this event.

Registration is required to attend this event, please [here](https://peerreviewcongress.org/registration-information/) to secure your ticket. Please check your spam/promotional inbox too.



Please register!

The Tenth Peer Review Congress features 3 days for presentations of new research into peer review and all aspects of scientific publication, bias, quality of reporting, and information access and dissemination.

<p>Franki Y. H. Kung,<sup>1</sup> Mariam Aly,<sup>2</sup> Shahana Ansari,<sup>3</sup> Eliana Colunga,<sup>4</sup> M. J. Crockett,<sup>5,6</sup> Amanda B. Diekman,<sup>3</sup> Pablo Gomez,<sup>7</sup> Paul C. McKee,<sup>8</sup> Miriam Pérez,<sup>9</sup> Sarah M. Stilwell,<sup>10</sup> Matthew Goldrick<sup>11,12</sup></p>

<h4>Objective</h4> Accumulating data suggest peer reviews can exhibit bias<i> </i>(ie, systematically devaluing certain topics, methods, or the study of nondominant populations<sup>1</sup>) and can be unfair with respect to process (eg, decisions are made inconsistently) or people (eg, authors are disrespected).<sup>2</sup> Recent studies have shown that biases and unfair behaviors in peer review tend to undermine the self-reported experiences and career progression of researchers, including those from underserved populations, such as women, nonbinary individuals, and people from racial and ethnic groups other than White.<sup>1-3</sup> This study examined whether peer review exercises increased psychology and neuroscience trainees’ perceived readiness to respond to biased and unfair reviewers.</p>

<h4>Design </h4>Single-session synchronous online trainings (conducted during 2023-2024) allowed for trainer-trainee and trainee-trainee interactions. Trainees were recruited for this study through social media, targeted academic organizations, and graduate programs. Trainers (professors and a postdoctoral fellow with multiple years of peer review experience) reviewed processes occurring between journal submission and receipt of a decision letter. Trainees were then given a simplified, anonymized decision letter and reviews. Exercises and discussions illustrated how trainees could plan manuscript revisions and compose responses to critiques. Finally, trainees were given examples of biased and unfair reviews, along with strategies for responding to them while acknowledging the power dynamics inherent to review. Discussion included how to maintain motivation when encountering such reviews. Participants were asked to complete surveys before and after online training in which they rated their ability to perform the following targeted skills on a scale from 1 (very little skill) to 5 (great deal of skill): define and identify bias in peer reviews; define and identify unfair language in peer reviews; respond to reviews using fair language; respond to biased or unfair reviews; and promote an inclusive and equitable peer-review culture.</p>

<h4>Results </h4>A total of 64 participants (42 PhD students [66%], 10 predoctoral students [16%], 8 postdoctoral trainees [13%], and 4 other trainees [5%]) rated their level of skill for 5 targeted skills. Ordinal regressions showed participants had significantly higher ratings for all skills at posttest. For example, the mean (SD) self-ratings for the skill to respond to biased on unfair reviews increased from 2.0 (1.01) on the pretest to 4.09 (0.89) on the posttest (<b>Table 25-0893</b>).</p>

![](https://assets.underline.io/markdown_image/1/image/f801de28833ed0c4470dce4f2815a32b.png)

<h4>Conclusions </h4>
<p>After these peer review exercises, psychology and neuroscience trainees had an increase in their perceived readiness to confront biased and unfair reviews. Work is ongoing to scale up this training using on-demand materials and further assess training impact, including measures beyond trainee self-reports.</p>
<h4>References</h4>
<p>1. Rogers LO, Moffitt U, McLean KC, Syed M. Research as resistance: naming and dismantling the master narrative of “good” science. <i>Am Psychol</i>. 2024;79:484-496. doi:10.1037/amp0001246</p>
<p>2. Silbiger NJ, Stubler AD. Unprofessional peer reviews disproportionately harm underrepresented groups in STEM. <i>PeerJ</i>. 2019;7:e8247. doi:10.7717/peerj.8247</p>
<p>3. Aly M, Colunga E, Crockett MJ, et al. Changing the culture of peer review for a more inclusive and equitable psychological science. <i>J Exp Psychol Gen</i>. 2023;152:3546-3565. doi:10.1037/xge0001461</p>
<sup>1</sup>Department of Psychological Sciences, Purdue University, West Lafayette, IN, US; <sup>2</sup>Department of Psychology, University of California Berkeley, Berkeley, CA, US; <sup>3</sup>Department of Psychological and Brain Sciences, Indiana University Bloomington, Bloomington, IN, US; <sup>4</sup>Department of Psychology and Neuroscience, University of Colorado Boulder, City, CO, US; <sup>5</sup>Department of Psychology, Princeton University, Princeton, NJ, US; <sup>6</sup>Department of Psychology, University Center for Human Values, Princeton University, Princeton, NJ, US; <sup>7</sup>Department of Psychology, Skidmore College, Saratoga Springs, NY, US; <sup>8</sup>Department of Psychology and Neuroscience, Duke University, Durham, NC, US; <sup>9</sup>Department of Psychology, North Park University, Chicago, IL, US; <sup>10</sup>Department of Health Behavior and Health Equity, University of Michigan, Ann Arbor, MI, US; <sup>11</sup>Department of Linguistics, Northwestern University, Evanston, IL, US, matt-goldrick@northwestern.edu; <sup>12</sup>Cognitive Science Program, Northwestern University, Evanston, IL, US.</p>
<h4>Conflict of Interest Disclosures</h4>
<p> None reported.</p>
<h4>Funding/Support</h4>
<p> This work was supported by grants from the National Science Foundation (DGE-2224777, DGE-2224779, DGE-2436430). </p>
<h4>Role of the Funder/Sponsor</h4>
<p> The funder of the study had no role in design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the abstract; and decision to submit the abstract for presentation.</p>
<h4>Additional Information</h4>
<p> Training materials are freely available (<a href="https://osf.io/658ev"><span class="Hyperlink CharOverride-6">https://osf.io/658ev</span></a>).</p>


Peer Review Exercises to Enhance Trainees' Readiness to Confront Unfair or Biased Reviews

poster

<p>Christopher Baethge,<sup>1,2</sup>Hannah Jergas<sup>3</sup></p>
<h4>Objective</h4>
<p> Although quotations are central to scientific publications, they often misrepresent the cited reference.<sup>1</sup> Inaccurate quotations may mislead readers, undermine the line of argument of a paper, and threaten the integrity of the scientific record.<sup>2,3</sup> We updated an earlier meta-analysis<sup>1</sup> to assess improvement in recent years.</p>
<h4>Design</h4>
<p> This systematic review and meta-analysis, preregistered on Open Science Framework (OSF), was conducted following the Cochrane Handbook and PRISMA statement. It focuses on quotation inaccuracy—errors of content—as opposed to bibliographical errors, often referred to as citation errors. We searched MEDLINE, PubMed Central via PubMed, and Web of Science from January 1, 2014, through October 1, 2023, for studies on quotation errors in the medical literature, with no date, language, or source restrictions, and we hand-searched all reference lists of included studies. Literature screening, data extraction, and risk of bias estimation were carried out by 2 authors independently. We adopted the original studies’ definitions<sup>1</sup> of major inaccuracies (ie, not at all in accordance with the claim of the original authors), minor inaccuracies (ie, inconsistencies and factual errors not severe enough to contradict a statement by citing authors), secondary quotations (ie, quotations of sources referencing an original study), and total quotation errors. Summary proportions of quotation errors were estimated in random-effects meta-analyses (log-transformed proportions and DL τ<sup>2</sup> estimates), with heterogeneity measures, 95% confidence intervals, and 95% prediction intervals. Time trends were investigated in meta-regressions.</p>
<h4>Results</h4>
<p> Based on 46 studies included, with 32,074 quotations/references checked, 16.9% were incorrect (95% CI, 14.1%-20.0%). Of these, 8.0% (95% CI, 6.4%-10.0%) were major inaccuracies and 7.8% (95% CI, 5.7%-10.5%) were minor inaccuracies. Heterogeneity throughout all meta-analyses was high (<b>Table 24-0811</b>). Rates of total errors have not improved during the last decade (slope: −0.002; 95% CI, −0.03 to 0.02; <i>P</i> = .85) and neither have rates of major errors (slope: 0.002; 95% CI, −0.02 to 0.03; <i>P</i> = .89). There were no associations with risk of bias, publication bias, number of references, or medical specialty (surgical, nonsurgical, other), but the total error rate was statistically significantly (<i>P</i> = .04; binomial, n = 25) and negatively correlated with Journal Impact Factor (−0.25; Spearman, n = 23). Interrater reliability was sufficient (Cohen κ: 0.73), and summary estimates were supported by sensitivity analyses employing arcsine square root data transformation and Sidik and Jonkman τ<sup>2</sup> estimates.</p>

![](https://assets.underline.io/markdown_image/1/image/6088d5f3f296b8d165f7d47e3340260a.png)

<h4>Conclusions</h4>
<p> Quotation inaccuracy remains prevalent in scientific medical texts. Despite seemingly broad awareness among journal editors and probably among many authors, there is no evidence of improvement in recent years. It is uncertain how the utilization of artificial intelligence in writing articles and in double-checking quotation inaccuracy will impact the problem, but human effort will likely remain important in creating and vetting quotes—on the part of authors, editors, reviewers, and readers.</p>
<h4>References</h4>
<p>1. Jergas H, Baethge C. Quotation accuracy in medical journal articles—a systematic review and meta-analysis. <i>PeerJ</i>. Published online October 27, 2015. doi:10.7717/peerj.1364</p>
<p>2. Baethge C. Importance, errors, and patterns of quotations to psychiatric original articles. <i>Pharmacopsychiatry</i>. 2020;53(6):247-255. doi:10.1055/a-1167-35673</p>
<p>3. Peoples N, Østbye T, Yan LL. Burden of proof: combating inaccurate citation in biomedical literature. <i>BMJ</i>. Published online November 6, 2023. doi:10.1136/bmj-2023-076441</p>
<sup>1</sup><i>Deutsches Ärzteblatt </i>and<i> Deutsches Ärzteblatt International</i>, Editorial Offices, Cologne, Germany, <span lang="de-DE">baethge@aerzteblatt.de; </span><sup>2</sup>Department of Psychiatry and Psychotherapy, Faculty of Medicine, University of Cologne, Cologne, Germany; <sup>3</sup>Department of Neurology, Faculty of Medicine, University of Cologne, Cologne, Germany.</p>
<h4>Conflict of Interest Disclosures</h4>
<p> Both authors have published on this topic before. Christopher Baethge is employed by a general medical journal (<i>Deutsches Arzteblatt, Deutsches Arzteblatt International</i>).</p>
<h4>Acknowledgment</h4>
<p> The authors gratefully acknowledge Joan Albert Hammerstein’s help in screening the literature.</p>
<h4>Additional Information</h4>
<p> This study is registered on Open Science Framework (OSF) at <a href="https://osf.io/95rej/?view_only=8c5d2b51a8814278bff1fec1e40d0bb9"><span class="Hyperlink CharOverride-6">https://osf.io/95rej/?view_only=8c5d2b51a8814278bff1fec1e40d0bb9</span></a></p>


Quotation inaccuracy in medicine - A systematic review and meta-analysis

<p>Seba Qussini,<sup>1,2</sup> Farizah Mezer Anami,<sup>2</sup> Kris Dierickx<sup>1</sup></p>
<h4>Objective </h4>
<p>Many peer review attributes are widely criticized and remain poorly investigated, particularly in the context of proposals’ peer review.<sup>1</sup> This study aims to explore the diverse perspectives of stakeholders regarding the role of unmasking in funding proposals’ peer review and the implications of open peer review as part of the recent open science movement, specifically in biomedical research proposals’ peer review.</p>
<h4>Design </h4>
<p>To describe participants’ perspectives as constructed through their recent experiences, we have conducted a generic descriptive qualitative study<sup>2</sup> within a constructivist paradigm, using semistructured interviews to gather insights from reviewers, applicants, and peer review scholars. A total of 23 participants were selected through purposive and snowball sampling from funding agencies in Belgium and Qatar with whom no prior relationship had been established. Interviews were conducted between June 2024 and February 2025 by the first author (S.Q.). Transcribed interviews were analyzed according to the 6-step thematic framework analysis described by Braun and Clarke.<sup>3</sup> Initially, autogenerated transcripts were read and checked for in-depth familiarization with the data, which was followed by a line-by-line inductive coding, conducted iteratively after each set of 2 interviews. We followed the Consolidated Criteria for Reporting Qualitative Research (COREQ) guidelines.</p>
<h4>Results </h4>
<p>Codes with shared characteristics were grouped into categories, and ultimately 3 overarching themes were generated: (1) the importance of increased transparency in fund allocation procedures while maintaining anonymized reviewer identities; (2) open peer review as a feasible approach for enhancing transparency and accountability in funding proposals’ peer review; and (3) a growing critical stance toward traditional peer review systems, calling for alternative models like baseline or lottery funding procedures. Collectively, the results shed light on the perceived advantages and limitations of different peer review models and provide an understanding of how unmasking identities influences the fairness and objectivity of fund allocation decisions.</p>
<h4>Conclusions </h4>
<p>There is persistent preference for double-anonymous review among the scientific community; however, researchers are increasingly aware of the shortcoming of anonymized review, especially in light of current challenges within the funding landscape. Simultaneously, they recognize the importance of greater openness in peer review and increased transparency in fund allocation procedures.</p>
<h4>References</h4>
<p>1. Qussini S, MacDonald RS, Shahbal S, Dierickx K. Blinding models for scientific peer-review of biomedical research proposals: a systematic review. <i>J Empir Res Hum Res Ethics</i>. 2023;18(4):250-262. doi:10.1177/15562646231191424</p>
<p>2. Qussini S, Dierickx K. Multi-stakeholder perspectives on current attitudes toward (un)blinding reviewers’ identity in biomedical research proposals peer review: a qualitative study. OSF Registries. <a href="https://osf.io/dp65f"><span class="Hyperlink CharOverride-6">https://osf.io/dp65f</span></a></p>
<p>3. Clarke V, Braun V. Thematic analysis. <i>J Pos Psychol</i>. 2017;12(3):297-298. doi:10.1080/17439760.2016.1262613</p>
<sup>1</sup>Centre for Biomedical Ethics and Law, Faculty of Medicine, KU Leuven, Leuven, Belgium, seba.qussini@student.kuleuven.be; <sup>2</sup>Hamad Medical Corporation, The Medical Research Center, Doha, Qatar.</p>
<h4>Conflict of Interest Disclosures</h4>
<p> None reported.</p>
<h4>Funding/Support</h4>
<p> The publication and conference fees will be funded by the Medical Research Center at Hamad Medical Corporation through grant MRC-01-24-305.</p>
<h4>Role of the Funder/Sponsor</h4>
<p> The Medical Research Center had no role in the design, analysis, or interpretation of this study.</p>


Multi-Stakeholder Perspectives on Current Attitudes Toward Unmasking Reviewers' Identity in Biomedical Research Proposals Peer Review: A Qualitative Study

<p>Jong-Wook Ban,<sup>1</sup> Hans Lund,<sup>2</sup> Karen Robinson,<sup>3</sup> Ida Svege,<sup>4</sup> Jan-Ole Hesselberg<sup>5,6</sup></p>
<h4>Objective</h4>
<p> Many have argued that the need for new research should be determined through a systematic examination of existing evidence to maximize its value.<sup>1,2</sup> Our study evaluated the outcomes of following such arguments on funding applications. We tested a hypothesis that funding applications justifying new research with a systematic review would receive higher scores than those without such justification.</p>
<h4>Design</h4>
<p> We analyzed funding applications submitted to Stiftelsen Dam, one of Norway’s largest health foundations, during the 2024 cycle. We used multiple regression to test our hypothesis while controlling for the influence of the following 5 covariables: requested funding amount, principal investigator’s birth year, principal investigator’s gender, project’s health domain, and research methods. </p>
<h4>Results</h4>
<p> Of 420 applications received, 193 (46.0%) cited a systematic review, 181 (43.1%) referenced a published or unpublished study, and 46 (11.0%) referred to applicants’ own experience as justification for new research. The median requested funding amount was kr3,000,000 ($267,796), with an IQR of kr2,801,000 to kr3,000,000. There were 158 (37.6%) men and 247 (58.8%) women principal investigators, and 15 (3.6%) who did not disclose their gender. The median birth year of the principal investigators was 1976 (IQR, 1968-1982). Funding applications included research projects in mental health (122 [29.0%]), physical health (197 [46.9%]), well-being (42 [10.0%]), disability (18 [4.3%]), and other domains (41 [9.8%]). Qualitative, mixed, and quantitative methods were used in 53 (12.6%), 145 (34.5%), and 222 (52.9%) applications, respectively. The median funding application score was 4.8 (IQR, 4.4-5.1) on a 7-point scale, where 1 represents poor and 7 represents excellent. Applications that cited a systematic review as justification received scores 0.19 (95% CI, 0.03-0.35) points higher than those that relied on the applicants’ own experience (<b>Figure 25-1129</b>). This could have raised an application with an average score by approximately 100 places in a ranking of 1716 applications. Applications that cited a systematic review received scores 0.12 (95% CI, 0.02-0.22) points higher than those that cited other types of evidence. </p>

![](https://assets.underline.io/markdown_image/1/image/679eec507dc3d8912ae0b0169dacb029.png)

<h4>Conclusions</h4>
<p> Our exploratory analysis showed an association between citing a systematic review and higher funding application scores. Our study has the following limitations. First, we could only control for variables available in our data. So, it was impossible to exclude the influence of other potential confounders, such as study design. Second, we evaluated funding application scores rather than funding outcomes. We could not evaluate the funding outcomes due to the very small number of applications receiving funding each year. Because Stiftelsen Dam heavily relies on funding application scores to decide on funding, we used those scores as a surrogate for funding outcomes.</p>
<h4>References</h4>
<p>1. Chalmers I, Bracken MB, Djulbegovic B, et al. How to increase value and reduce waste when research priorities are set. <i>Lancet</i>. 2014;383(9912):156-165. doi:10.1016/S0140-6736(13)62229-1</p>
<p>2. Lund H, Brunnhuber K, Juhl C, et al. Towards evidence based research. <i>BMJ</i>. 2016;355:i5440. doi:10.1136/bmj.i5440</p>
<sup>1</sup>Section of Hospital Medicine, University of Chicago, Chicago, IL, US, jban@uchicago.edu; <sup>2</sup>Section of Evidence-Based Practice, Western Norway University of Applied Sciences, Bergen, Norway; <sup>3</sup>Johns Hopkins Evidence-Based Practice Center, Johns Hopkins University, Baltimore, MD, US; <sup>4</sup>Nordic Institute for Studies in Innovation, Research and Education, Oslo, Norway; <sup>5</sup>Stiftelsen Dam, Oslo, Norway; <sup>6</sup>Department of Psychology, University of Oslo, Oslo, Norway.</p>
<h4>Conflict of Interest Disclosures</h4>
<p> Ida Svege reported being the former Head of Program Development (Leder for programutvikling) of Stiftelsen Dam. Jan-Ole Hesselberg reported being the current Program Manager (Programsjef) of Stiftelsen Dam. </p>


The influence of using a systematic review to justify new research on funding application score

<p>Markus K. Heinemann,<sup>1</sup><span lang="de-DE"> </span>Andreas Boening,<sup>2</sup><span lang="de-DE"> Kazunori Okabe,</span><sup>3</sup><span lang="de-DE"> Jessica Bogensberger</span><sup>4</sup></p>
<h4>Objective</h4>
<p> In 2013, a cardiothoracic surgical journal introduced plagiarism detection (Crossref, powered by iThenticate). After a phase of 2 years during which all submissions were screened, since January 2015 only returned manuscripts after revision were checked by default. Primary submissions were then screened at the discretion of the Editor in Chief (EiC) only. The aim of this study was to report the 12-year experience.</p>
<h4>Design</h4>
<p> All manuscripts submitted between January 1, 2013, and December 31, 2024, and branded by the plagiarism search engine with a similarity index above 33% were analyzed regarding the originator of the screening (EiC or revision default), reasons (actual plagiarism or self-plagiarism, which is a form of duplication), similarity percentage, and final decision. Descriptive statistics were applied. All updates of the search tool were implemented.</p>
<h4>Results</h4>
<p> During the investigation period, 4083 manuscripts were submitted. A total of 99 (2.4%) showed a similarity index greater than 33%. The mean percentage was 53.3% (minimum, 34%; maximum, 99%; median, 50%; SD, ±14.8). Plagiarism and duplication search was prompted by the EiC in 48 papers and by automated default in 51 papers. Self-plagiarized or duplicated content was by the same authors in 71 of 99 papers (71.7%), by different authors in 18 (18.2%), and both in 10 (10.1%). Fifty-five manuscripts (55.6%) remained unpublished: 17 rejected without review, 34 rejected, and 4 withdrawn. Forty-four of 99 (44.4%) were accepted after revisions. There were 4 attempts at duplicate publication, all rejected. Preprints were discovered in 3 cases, with 1 accepted after revision: the preprint had been uploaded by a different journal for review and was rejected there, but then not deleted. The 44 manuscripts finally accepted contained previously published sequences by the same authors in 36, by different authors in 4, and both in 4. In 25 cases, the EiC rendered the similarity uncritical (eg, materials and methods section). The most frequent necessary correction was adequate citation. No adequate citation was given for self-plagiarized/duplicated content in 23 cases and for content by different authors in 4.</p>
<h4>Conclusions</h4>
<p> Although the overall rate of plagiarism and duplication can be considered low, scientific misconduct was detected and its publication prevented in a few cases. In many instances, duplicated passages contained similar material and methods previously published by the same group. Deficiencies in adequate citation are a problem and necessitate continued education of authors. Selective plagiarism and duplication screening remains an important tool, although chosen thresholds of similarity remain arbitrary. Further improvements of sensitivity are expected, especially with the implementation of machine learning advances. Editorial decisions, however, cannot be made on automated findings alone, but must be put into personal perspective.</p>
<sup>1</sup>German Society for Thoracic and Cardiovascular Surgery (DGTHG), Germany, heinemann@uni-mainz.de; <sup>2</sup><span lang="de-DE">Universitaetsklinik Giessen, Germany; </span><sup>3</sup><span lang="de-DE">Bell Land General Hospital, Osaka, Japan; </span><sup>4</sup>Thieme Publishers, Stuttgart, Germany.</p>
<h4>Conflict of Interest Disclosures</h4>
<p> Jessica Bogensberger is an employee of Thieme Publishers, the publishing house of the journal investigated. All other authors have no conflict of interest to declare.</p>


Experience With 12 Years of Plagiarism Screening

<p>Jhih-Yi Hsieh,<sup>1</sup> Aditi Raghunathan,<sup>1</sup> Nihar B. Shah<sup>1</sup></p>
<h4>Objective </h4>
<p>Collusion rings—researchers who manipulate reviewer assignments to review each other’s work—pose major challenges for machine learning and artificial intelligence conferences. As submissions grow substantially, automated reviewer assignment algorithms<sup>1</sup> have become common, often relying on the text similarity of reviewers’ past papers and the author’s submission.<sup>2</sup> While this text similarity is generally considered collusion safe,<sup>3</sup> this work investigated potential manipulations and proposes mitigations.</p>
<h4>Design </h4>
<p>For a colluding author and reviewer, we designed a realistic 2-step attack procedure: (1) reviewer profile curation, in which the colluding reviewer kept only 1 past paper that was the most similar to the author’s submission for text matching; and (2) submission abstract modification, in which the colluding author modified their submission’s abstract by adding background sentences related to the themes in the reviewer’s profile and inserting keywords that increased the text similarity to the reviewer. For scalability, we used a large language model (gpt-4-0125-preview [OpenAI]) to modify abstracts, although colluding authors in practice could manually refine them to reduce suspicion. We curated a dataset (3123 papers; 7900 reviewers) of the NeurIPS 2023 conference, which reviewed full papers (not abstracts). We then sampled reviewer-paper pairs where the reviewer initially ranked 20th, 101st, 501st, or 1001st among all reviewers by similarity, simulated collusion via our attack, and reported if the reviewer became highly ranked afterward. We proposed 2 hypotheses to improve robustness. (1) Requiring more past papers in reviewer profiles reduces attack success. (2) When a reviewer has multiple past papers in their profile, the similarities between each past paper and the submission are first calculated, then aggregated by taking the mean or maximum. We hypothesized that mean aggregation is more robust than maximum because averaging reduces manipulation effects. We tested these hypotheses via 2 ablations: (1) enforcing 10 papers in reviewer profiles and (2) comparing mean vs maximum under attack. Lastly, a randomized controlled trial tested for the identifiability of adversarial abstracts by unsuspecting human reviewers. Participants were randomly assigned either the benign or adversarial abstract to review.</p>
<h4>Results </h4>
<p>The attack was highly effective (<b>Table 25-0911</b>); 67% of reviewers initially ranked 1001st became top 5, despite not working on similar topics. Both hypotheses on robustness were confirmed. When reviewers selected 10 past papers (100 samples), only 19% of similarity rankings were in the top 5 after the attack. For mean vs maximum aggregation (100 samples each), 32% and 49% of rankings were in the top 5 after attack, respectively. Lastly, in the randomized controlled trial (116 samples), while participants had more complaints for manipulated papers, surprisingly, benign papers elicited similar complaints, suggesting plausible deniability.</p>

![](https://assets.underline.io/markdown_image/1/image/899f0f6d7dd7d88e89c0954872b543ac.png)

<h4>Conclusions </h4>
<p>This study found that conference reviewer assignments based on text matching are vulnerable to colluders and identified methods to improve robustness.</p>
<h4>References</h4>
<p>1. Shah NB. Challenges, experiments, and computational solutions in peer review. <i>Commun ACM</i>. June 2022. Accessed July 11, 2025. <a href="https://www.cs.cmu.edu/~nihars/preprints/SurveyPeerReview.pdf"><span class="Hyperlink CharOverride-6">https://www.cs.cmu.edu/~nihars/preprints/SurveyPeerReview.pdf</span></a></p>
<p>2. Cohan A, Feldman S, Beltagy I, Downey D, Weld D. SPECTER: document-level representation learning using citation-informed transformers. In: <i>Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics</i>. 2020, 2270-2282. doi:10.18653/v1/2020.acl-main.20</p>
<p>3. Wu R, Guo C, Wu F, Kidambi R, Van Der Maaten L, Weinberger K. Making paper reviewing robust to bid manipulation attacks. In: <i>International Conference on Machine Learning</i>. 2021: 11240-11250.</p>
<sup>1</sup>Carnegie Mellon University, Pittsburgh, PA, US, jhihyi.hsieh@gmail.com.</p>
<h4>Conflict of Interest Disclosures</h4>
<p> Jhih-Yi Hsieh, Aditi Raghunathan, and Nihar B. Shah are employees of Carnegie Mellon University. Nihar B. Shah is a member of the Peer Review Congress Advisory Board but was not involved in the review or decision for this abstract.</p>
<h4>Funding/Support </h4>
<p>This study was funded by the J.P. Morgan Research Scholar Program (ONR N000142212181, NSF 2200410, 1942124, 2310758), the AI2050 program at Schmidt Sciences (G2264481), the Google Research Scholar Program, Apple, Cisco, and Open Philanthropy.</p>
<h4>Role of the Funder/Sponsor </h4>
<p>The funders played no role in the design and execution of the experiment, data analysis, or preparation of the abstract.</p>
<h4>Additional Information </h4>
<p>gpt-4-0125-preview (OpenAI) was used between April and September 2024 to generate modified abstracts without human supervision. Jhih-Yi Hsieh takes responsibility for the integrity of the content generated.</p>


Vulnerability of Automated Text-Matching-Based Reviewer Assignments to Collusions

<p>Cyril Labbé,<sup>1</sup> Qinyue Liu,<sup>1</sup> Amira Barhoumi,<sup>1</sup> Olessya Miroshnichenko<sup>1</sup></p>
<h4>Objective</h4>
<p> For various reasons, duplicate publications are often considered problematic.<sup>1,2</sup> We tested a method to semiautomatically detect such publications from publicly available metadata registered with Crossref by publisher.<sup>3</sup></p>
<h4>Design</h4>
<p> We first queried Crossref using ISSNs or ISBNs to build a set of (random) digital object identifiers (DOIs) with abstracts for duplicates to be searched. For each DOI, Crossref was queried again to retrieve a new set of 200 DOIs with similar titles containing potential duplicates of publications in the first set. We then identified abstracts in the second set that were nearly identical to those in the first set, differing by only a few words. When DOIs of apparent duplicates were not registered by preprint platforms (eg, bioRxiv, medRxiv), we would then automatically analyze when DOIs were registered by a different publisher, had differences in authorship, or were published in journals or books. The procedure was applied to the following sets of DOIs chosen to represent a wide variety of publishers for which abstracts are registered with Crossref: <i>Science</i> publications (2023-2024), <i>PLOS One</i> (2023-2024), <i>International Journal of Molecular Sciences</i> (2024), <i>BioMed Research International</i> (2024), <i>Scientific Reports</i> (2024), and IGI Global (DOIs included in books published in 2024).</p>
<h4>Results</h4>
<p> The vast majority of duplicates found were not problematic, as they typically originated from preprint versions (<b>Table 25-0986</b>). For example, of 375 duplicated abstracts for articles published in <i>PLOS One</i>, 4 were not preprint publications. Moreover, close inspection revealed that 3 duplicates were republication of abstracts only. Only the 1 remaining abstract could be seen as problematic, as it was a duplication of the same exact content in 2 different journals, both open access.</p>

![](https://assets.underline.io/markdown_image/1/image/e018f0b6fdc9855b01f8f955fd195208.png)

<h4>Conclusions</h4>
<p> One limitation of the process is that despite having the exact same abstract, the body of the publications might be different. For many publications, abstracts are not registered with Crossref or sometimes are registered with errors. It is also difficult to identify when one of the DOIs represents a republication with an abstract alone. Abstracts publicly available at Crossref can be used to detect problematic duplicate publications as a cost-effective alternative to plagiarism detection tools. This experiment emphasizes the importance for publishers to answer positively to the Initiative for Open Abstracts (I4OA), which calls on all scholarly publishers to<span lang="fr-FR"> open the abstracts of their</span> published works and, where possible, to submit them to Crossref.</p>
<h4>References</h4>
<p>1. Tramèr MR, Reynolds DJ, Moore RA, McQuay HJ. Impact of covert duplicate publication on meta-analysis: a case study. <i>BMJ</i>. 1997;315(7109):635-640. doi:10.1136/bmj.315.7109.635</p>
<p>2. Errami M, Hicks JM, Fisher W, et al. Déjà vu—a study of duplicate citations in Medline. <i>Bioinformatics</i>. 2008;24(2):243-249. doi:10.1093/bioinformatics/btm574</p>
<p>3. van Eck NJ, Waltman L. Crossref as a source of open bibliographic metadata. <i>MetaArXiv</i>. Preprint posted online May 12, 2025. doi:10.31222/osf.io/smxe5</p>
<sup>1</sup>Université Grenoble Alpes, French National Centre for Scientific Research, Grenoble INP, Laboratoire d’Informatique de Grenoble, Grenoble, France, cyril.labbe@imag.fr.</p>
<h4>Conflict of Interest Disclosures</h4>
<p> None reported.</p>
<h4>Funding/Support</h4>
<p> We acknowledge the NanoBubbles project that has received Synergy grant funding from the European Research Council within the European Union’s Horizon 2020 program (grant agreement 951393).</p>
<h4>Role of the Funder/Sponsor</h4>
<p> The funders had no role in this research.</p>


Identifying Potential Duplicate Publications in the Scientific Literature Using Crossref

<p>Wendeline Swart,<sup>1</sup> Ophélie Fraisier-Vannier,<sup>1</sup> Guillaume Cabanac<sup>1,2</sup></p>
<h4>Objective </h4>
<p>Tortured phrases<sup>1</sup> in scientific articles, such as <i>amino corrosive</i> instead of <i>amino acid</i>, are potential signals of scientific misconduct. They alter established expressions using synonyms to avoid plagiarism detection. We highlighted the presence of tortured phrases in 1323 conference proceedings and identified the most impacted conference.</p>
<h4>Design </h4>
<p>We used the Problematic Paper Screener (PPS),<sup>2</sup> a tool we developed to detect research integrity issues, to identify conference proceedings published containing tortured articles, defined as articles with 5 or more tortured phrases or fewer than 5 tortured phrases but assessed as problematic by a human. We selected the conference with the largest number of tortured articles and probed with Dimensions (a bibliometric platform) and PubPeer (a postpublication peer review platform) the production of the experts listed in its proceedings. We counted the experts’ numbers of retractions, expressions of concern, and other red flags.</p>
<h4>Results</h4>
<p> The PPS tabulated 20,066 tortured articles published from 2005 to 2024 by 344 publishers. The conference proceedings subset comprised 7518 articles (37%) mainly published by the Institute of Electrical and Electronics Engineers (IEEE) (6156 [82%]; presented in 1348 conferences), AIP Publishing (648 [9%]), IOP Publishing (364 [5%]), and 12 other publishers.<sup>3</sup> The most problematic conference was the IEEE International Conference on Optimization, Computing, and Wireless Communication held in Ethiopia in January 2024, featuring 324 peer-reviewed articles affiliated with India and China only. We identified problematic content in the proceedings using 2 clues: (1) nonsensical text, found in 260 articles (80%) with a mean 11 tortured phrases (<b>Figure 25-1029</b>), and (2) questionable expertise in 2 listed groups. The organizing committee (34 individuals) included 6 persons whose work had been commented on on PubPeer for various concerns on 31 articles, 21 of which have been retracted. Concerns included tortured phrases, suspicion of purchase of an authorship position, plagiarized sections, nonsensical equations, and irrelevant references. The reviewers (134 individuals) included 31 persons (23%) who authored 67 articles that were commented on on PubPeer for integrity issues, 8 of which had an expression of concern and 9 of which have been retracted. Eight reviewers have had several retractions, including 1 reviewer with 3 retractions due to violations of IEEE policy on authorship.</p>

![](https://assets.underline.io/markdown_image/1/image/225f2b9bf6aa303b9b83081c41f1ee00.png)

<h4>Conclusions </h4>
<p>As of February 2025, IEEE tagged with an expression of concern 15 of 260 articles (6%) flagged by the PPS (with a mean of 4 tortured phrases per article) and reported on PubPeer. IEEE had not issued expressions of concern or other corrective labels for articles with more tortured phrases. Limitations of this analysis include the focus on a single conference and the possibility that the PPS may identify false positives or less serious problems that do not require an expression of concern or retraction. However, reassessment may be needed for many of the 20,066 PPS-flagged articles.</p>
<h4>References</h4>
<p>1. Cabanac G, Labbé C, Magazinov A. Tortured phrases: a dubious writing style emerging in science—evidence of critical issues affecting established journals. <i>arXiv</i>. Preprint posted online July 12, 2021. doi:10.48550/arXiv.2107.06751</p>
<p>2. Cabanac G, Labbé C, Magazinov A. The ‘Problematic Paper Screener’ automatically selects suspect publications for post-publication (re)assessment. <i>arXiv</i>. Preprint posted online October 7, 2022. doi:10.48550/arXiv.2210.04895</p>
<p>3. Swart W, Cabanac G. Year after year: Tortured conference series thriving in computer science. <i>arXiv</i>. Preprint posted online October 17, 2023. doi:10.48550/arXiv.2401.02422</p>
<sup>1</sup>Université de Toulouse, IRIT (UMR 5505 CNRS), Toulouse, France, guillaume.cabanac@univ-tlse3.fr; <sup>2</sup>Institut Universitaire de France, Paris, France.</p>
<h4>Conflict of Interest Disclosures </h4>
<p>Guillaume Cabanac is the administrator of the Problematic Paper Screener, a public platform that uses metadata from Digital Science and PubPeer via no-cost agreements, and has been in touch with most of the major publishers and their integrity officers, offering pro bono consulting regarding detection tools to various actors in the field, including Clear Skies, Morressier, River Valley, Signals, and STM. No other disclosures were reported.</p>
<h4>Funding/Support </h4>
<p>Guillaume Cabanac received funding from the Institut Universitaire de France and the NanoBubbles project, which received Synergy grant funding from the European Research Council within the European Union’s Horizon 2020 program (grant agreement No. 951393).</p>


Tortured Phrases as a Sign of Possible Misconduct in Proceedings from an Engineering Conference

<p>Giovanni Colavizza,<sup>1,2</sup> Lauren Cadwallader,<sup>3</sup> Iain Hrynaszkiewicz<sup>3</sup></p>
<h4>Objective </h4>
<p>This research explored whether articles that demonstrate 1 or more open science practices received more citations on average than similar articles that do not demonstrate open science practices by applying an existing linear regression model<sup>1</sup> to an openly available dataset.</p>
<h4>Design </h4>
<p>This study applied a previously published citation prediction model<sup>1</sup> with data from openly available sources including OpenAlex, DataCite, CrossRef, and the French Open Science Monitor dataset,<sup>2</sup> which includes all research publications from open data sources with at least 1 French author. This study evaluated whether open science practices in publications are associated with changes in citation impact. A total of 479,700 journal articles were analyzed, which were from across 10 broad categories of research domains, have at least 1 French author, were published in 2020-2022, and for which the metadata, including citation counts, could be extracted from the publicly available sources at the time of this study (July 2024). The citation prediction method controlled for factors known to influence citations such as article age, open access status, research discipline, length of reference lists, and numbers of authors. Associations were assessed with an ordinary least squares model.</p>
<h4>Results </h4>
<p>The analysis showed that a citation advantage was associated with all 3 open science practices (data, code, and preprint sharing), both individually and cumulatively. Posting a preprint had the largest association with citations and was correlated with a 19.0% higher number of citations, compared with articles that did not have preprints. The associations with sharing research data and code were smaller—and correlated with 14.3% and 13.5% higher number of citations, respectively. These advantages were cumulative, that is, articles with a preprint, shared research data, and shared code attracted a mean of 46.8% more citations than articles that do not share these outputs. Variation in citation advantage was seen across disciplines (<b>Table 25-0867</b>). For example, articles that had preprint postings had a 1.0% higher number of citations in the fields of earth, ecology, energy and 62.5% more citations in the humanities, compared with articles in the same disciplines that did not have preprints. Study limitations include the dataset bias toward French publications (we might observe different citation correlations in a more international sample), and the results for the data and code sharing citation advantage were not directly comparable with the results in previous studies due to differences in the definition and calculation of open science practices.</p>

![](https://assets.underline.io/markdown_image/1/image/acaf7e3c6dc29e934136a937db6aade0.png)

<h4>Conclusion </h4>
<p>This study explored the French Open Science Monitor dataset and investigated whether open science indicators were associated with a citation premium received by the publications that follow them. To our knowledge, this is the largest scale analysis of such a question to date using nearly 500,000 publications from many scientific domains, which found a positive association between the open science practices of data, code, and preprint sharing and postpublication citation counts.</p>
<h4>References</h4>
<p>1. Colavizza G, Cadwallader L, LaFlamme M, et al. An analysis of the effects of sharing research data, code, and preprints on citations. <i>PLOS One</i><span lang="fr-FR">. 2024;19(10): e0311493. doi:10.1371/journal.pone.0311493</span></p>
<p><span lang="fr-FR">2. Ministère de l’Enseignement supérieur, de la Recherche et de l’Innovation. Baromètre de la science ouverte (général). Accessed July 15, 2024. </span><a href="https://data.enseignementsup-recherche.gouv.fr/explore/dataset/open-access-monitor-france/information."><span class="Hyperlink CharOverride-6" lang="fr-FR">https://data.enseignementsup-recherche.gouv.fr/explore/dataset/open-access-monitor-france/information</span></a></p>
<sup>1</sup>University of Bologna, Bologna, Italy; <sup>2</sup>University of Copenhagen, Copenhagen, Denmark; <sup>3</sup>PLOS, Cambridge, UK, lcadwallader@plos.org.</p>
<h4>Conflict of Interest Disclosures</h4>
<p> PLOS works to promote adoption of open science practices and provided funding for this study for the collection, management, analysis, and interpretation of the data by Giovanni Colavizza. PLOS also provided support in the form of salaries for authors Lauren Cadwallader and Iain Hrynaszkiewicz. PLOS also had a role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, and approval of the abstract; and decision to submit the abstract for presentation.</p>
<h4>Additional Information</h4>
<p> Data and code necessary to reproduce the results are available: <a href="https://figshare.com/articles/dataset/A_study_on_the_citation_impact_of_Open_Science_Indicators_in_the_French_Open_Science_Monitor/27822663"><span class="Hyperlink CharOverride-6">https://doi.org/10.6084/m9.figshare.27822663</span></a></p>


Citations of Articles With Open Science Indicators in the French Open Science Monitor Dataset

<p>Stephan B. Bruns,<sup>1</sup> Helmut Herwartz,<sup>2</sup> John P. A. Ioannidis,<sup>3</sup> Chris-Gabriel Islam,<sup>2</sup> Fabian H. C. Raters<sup>2</sup></p>
<h4>Objective </h4>
<p>The economics literature predominantly reports statistical values in data tables. This study developed a tool for extraction of statistical values in tables to identify reporting errors and to analyze the association between journal data and code availability policies and article characteristics.</p>
<h4>Design </h4>
<p>DORIS (Diagnosis of Reporting Errors in Scraped Tables)<sup>1,2</sup> automatically extracts statistical values (eg, coefficients and SEs) from tables via web scraping in R to extract tables from HTML articles and text mining in Python to interpret data. This analysis included the top 50 economics journals that provided articles in HTML from 1998 to 2016. Tests were divided into 2 categories: main (first 3 table rows from main tables) and nonmain (all other rows from main tables and other tables). We used a staggered difference-in-differences design to assess the association between data and code availability policies and article outcomes<sup>3</sup>: reporting errors using a dummy for inconsistency altering significance level; statistical significance proxied by <i>z</i>-value, presence of visual indicators of statistical significance, and type of statistical value reported; logarithmized number of tests per article, tables with tests, and a dummy for appendix position; and citations measured by the logarithmized citation count for the first 7 years.</p>
<h4>Results </h4>
<p>The study included 578,132 statistical tests (median: 27 per table) from 15,725 tables (median: 4 per article) in 3746 articles (median: 76 per journal) that were published in 31 journals. Based on a sample of 3068 statistical tests, DORIS had a false discovery rate of 1.2%. Analysis revealed that 547 of 3677 articles (14.8%) had at least 1 reporting error in main tests altering the statistical significance of findings. The errors were prevalently oriented toward lowering <i>P</i> values and claiming statistical significance. Assessment of data and code availability policies included 21 journals and 535,838 tests. These policies were associated with slightly less emphasis on statistical significance (smaller z-values: −0.16 [90% CI, −0.33 to 0], <i>P</i> = .10; fewer statistical significance symbols: −4 [90% CI, −6 to −2] percentage points [pp]; <i>P</i> = .004; fewer tests reported with a focus on significance (<i>t</i>, <i>z</i>, or <i>P </i>values instead of SEs and CIs): −8 [90% CI, −13 to −3] pp; <i>P</i> = .004), more rigor in reporting (more tests: 11% [90% CI, −1% to 22%]; <i>P</i> = .12; more tables: 15% [90% CI, 7% to 23%]; <i>P</i> = .002; more appendix tables: 4 [90% CI, 2 to 7] pp; <i>P</i> = .009), and fewer reporting errors in nonmain tests (−0.2 [90% CI, −0.4 to 0] pp; <i>P</i> = .08), but there was no difference in citations (−9% [90% CI, −21% to 4%]; <i>P</i> = .25).</p>
<h4>Conclusions </h4>
<p>DORIS may be used in the peer review process in the economics literature to improve article quality and to generate large scale data for future meta-research. Data and code availability policies may help improve the reliability of published economics research.</p>
<h4>References</h4>
<p>1. Better Papers. Home page. Accessed June 12, 2025. <a href="https://betterpapers.org/%23sec-faqs"><span class="Hyperlink CharOverride-6">https://betterpapers.org/#sec-faqs</span></a></p>
<p>2. Bruns SB, Herwartz H, Ioannidis JPA, Islam C-G, Raters FHC. Statistical reporting errors in economics. <i>MetaArXiv</i>. Preprint posted online September 01, 2023. doi:10.31222/osf.io/mbx62</p>
<p>3. Borusyak, K, Jaravel, X, Spiess, J. Revisiting event-study designs: robust and efficient estimation. <i>Rev Econ Studies</i>. 2024;91(6):3253-3285. doi:10.1093/restud/rdae007</p>
<sup>1</sup>Hasselt University, Hasselt, Belgium, stephan.b.bruns@gmail.com; <sup>2</sup>Georg August University Göttingen, Göttingen, Germany; <sup>3</sup>Meta-Research Innovation Center at Stanford (METRICS), Stanford, CA, US.</p>
<h4>Conflict of Interest Disclosures</h4>
<p> John P. A. Ioannidis is a member of the Peer Review Congress Advisory Board but was not involved in the review or decision for this abstract. No other disclosures were reported.</p>
<h4>Funding/Support</h4>
<p> Funding was received from the German Research Foundation (DFG) under the project “Replications in Empirical Economics: Necessity, Incentives and Impact,” with follow up project “Selective Reporting and the Evolving Research Landscape in Economics” (project number: 405039391) and from Hasselt University within the framework of BOF BILA (BOF21BL08).</p>
<h4>Role of Funder/Sponsor</h4>
<p> DFG and BOF BILA funding helped in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the abstract; and decision to submit the abstract for presentation.</p>


Automated Interpretation of Statistical Tables to Assess Reporting Errors and Associations With Open Science Policies in Economic Journals

<p>Constant Vinatier,<sup>1</sup> Ayu Putu Madri Dewi,<sup>2</sup> Gwénaël Dumont,<sup>1</sup>Tracey Weissgerber,<sup>3</sup>Vladislav Nachev,<sup>3</sup>Gowri Gopalakrishna,<sup>2,3,4</sup> Maud Scheidecker,<sup>1</sup> François-Joseph Arnault,<sup>1</sup> Nicholas J. DeVito,<sup>5</sup> Guillaume Freyermuth,<sup>6</sup> Mathieu Acher,<sup>6,10</sup> Gauthier Le Bartz Lyan,<sup>6</sup> Inge Stegeman,<sup>7,8</sup>Mariska M. G. Leeflang,<sup>2</sup>F. Naudet<sup>9,10</sup></p>
<h4>Objective </h4>
<p>Despite open science policies in major biomedical journals, adherence remains uncertain. This study evaluated automated tools, from regular expressions to large language models (LLMs), for assessing core open science practices in leading biomedical journals.</p>
<h4>Design </h4>
<p>We retrospectively assessed research articles from a sample of 10 major generalist medical journals (<i>Annals of-Internal Medicine</i>, <i>BMJ</i>, <i>BMC Medicine</i>, <i>Canadian Medical Association Journal</i> [<i>CMAJ</i>], <i>JAMA</i>, <i>JAMA</i> <i>Network Open</i>, <i>Lancet</i>, <i>Nature Medicine</i>, <i>New England Journal of Medicine</i>, and <i>PLoS Medicine</i>) from 2020 to 2023. Articles were retrieved via PubMed using a Peer Review of Electronic Search Strategies (PRESS) search strategy. The database comprised random samples of 103 randomized controlled trials (RCTs), 98 meta-analyses (MAs), and 111 other research articles (RAs). We evaluated 13 open science practices, including study registration, data sharing, and protocol sharing (open access or upon request). Each article was evaluated by 2 independent raters, with any disagreements resolved by a third rater. Seven different automated tools—rtransparent, oddpub, ctRegistries, ContriBot, DataSeer, SciScore, and an LLM (Llama 3-70B)—were used. Diagnostic accuracies were estimated using sensitivities, specificities, F1 scores, and LR+ and LR-.</p>
<h4>Results </h4>
<p>Manual extraction in the 312 articles identified registration in 98% (101/103) of RCTs, 69% (68/98) of MAs, and 18% (20/111) of RAs. Open data were present in 6% (6/103) of RCTs, 36% (35/98) of MAs, and 13% (15/111) of RAs and accessible upon request in 78% (80/103), 41% (40/98), and 59% (66/111), respectively. Protocols were openly available in 84% (87/103) of RCTs, 64% (63/98) of MAs, and 20% (22/111) of RAs and accessible upon request in 5% (5/103), 3% (3/98), and 3% (3/111), respectively. The accuracy of automated tools varied depending on the practice evaluated, with F1 scores ranging from 1.00 (Conflict of Interest statement, rtransparent) to 0.16 (SciScore, registration). For study registration, a simple tool using regular expressions, such as rtransparent, demonstrated good sensitivity (77%; 95% CI, 70%-83%) and high specificity (93%; 95% CI, 88%-97%). Data sharing detection remained challenging; for instance, rtransparent detects data sharing with a sensitivity of 74% (95% CI, 68%-80%) and a specificity of 59% (95% CI, 46%-70%). Different diagnostic accuracies were observed depending on the type of research and the journal, likely due to different formatting standards. All results are shown in <b>Table 25-1025</b>. Limitations include the declarative nature of some practices (eg, data sharing).</p>

![](https://assets.underline.io/markdown_image/1/image/f6bf4287ea9763381f2096129dbea4e8.png)

<h4>Conclusions </h4>
<p>Our study provides a detailed description of core open science practices across leading biomedical journals. It also highlights current challenges regarding the accuracy of automated tools in detecting these practices. While these tools likely provide valuable insights into overall practices, it is crucial to remain aware of the potential ranking biases introduced by these tools, as well as their limitations in providing detailed feedback for individual studies.</p>
<sup>1</sup>Univ Rennes, Inserm, EHESP, Irset (Institut de recherche en santé, environnement et travail), UMR_S 1085, Rennes, France, constant.vinatier1@gmail.com; <sup>2</sup>Department of Epidemiology and Data Science, Amsterdam University Medical Centers, Amsterdam, the Netherlands; <sup>3</sup>QUEST Center for Responsible Research, Berlin Institute of Health at Charité–Universitätsmedizin Berlin, Berlin, Germany; <sup>4</sup>Department of Epidemiology, Faculty of Health, Medicine, and Life Sciences, Maastricht University, Maastricht, the Netherlands; <sup>5</sup>Bennett Institute for Applied Data Science, Nuffield Department of Primary Care Health Sciences, University of Oxford, Oxford, UK; <sup>6</sup>Univ Rennes, IRISA, Inria, CNRS, Rennes, France; <sup>7</sup>Department of Otorhinolaryngology and Head and Neck Surgery, University Medical Center Utrecht, Utrecht, the Netherlands; <sup>8</sup>Brain Center, University Medical Center Utrecht, Utrecht, the Netherlands; <sup>9</sup>Univ Rennes, CHU Rennes, Inserm, EHESP, Irset (Institut de recherche en santé, environnement et travail), UMR_S 1085, Rennes, France; <sup>10</sup>Institut Universitaire de France (IUF), France.</p>
<h4>Conflict of Interest Disclosures</h4>
<p> None reported.</p>
<h4>Funding/Support</h4>
<p> As part of the OSIRIS project, this work was supported by the European Union’s Horizon Europe Research and Innovation Program under grant agreement number 101094725. Constant Vinatier, Ayu Putu Madri Dewi, Gowri Gopalakrishna, Nicholas J. DeVito, Inge Stegeman, Mariska M. G. Leeflang, and <span lang="fi-FI">F. Naudet</span> are members of this project.</p>
<h4>Role of the Funder/Sponsor</h4>
<p> The funder had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.</p>


Next from The 10th Peer Review Congress

Quotation inaccuracy in medicine - A systematic review and meta-analysis

Similar lecture

[V] Motivations to Participate in the Peer Review Process at the Journal of Urology