United States

&lt;p&gt;Yagmur Ozturk,&lt;sup&gt;1&lt;/sup&gt; Frédérique Bordignon,&lt;sup&gt;2,3&lt;/sup&gt; Cyril Labbé,&lt;sup&gt;1&lt;/sup&gt; François Portet&lt;sup&gt;1&lt;/sup&gt;&lt;/p&gt;
&lt;h4&gt;Objective&lt;/h4&gt;
&lt;p&gt; To address the challenge of tracking the use of unreliable sources in scientific literature, we analyzed citations to retracted articles (CRAs) using retraction reasons (RRs) and citation context (CC) analysis. Our aim was to inform the development of automated tools for postpublication peer review (PPPR) by identifying instances in which a CRA is not only an unreliable source, but may also introduce unreliability into the citing work itself, depending on the way it is cited. We examined the degree to which the content of the citation context aligns with the RR, indicating potential risk for the integrity of the citing article.&lt;/p&gt;
&lt;h4&gt;Design&lt;/h4&gt;
&lt;p&gt; RRs were obtained from the Retraction Watch database&lt;sup&gt;1&lt;/sup&gt; for a set of 22,558 publications that cite retracted sources accessed from the Problematic Paper Screener.&lt;sup&gt;2&lt;/sup&gt; We selected 29 of 108 retraction reasons that directly relate to scientific content of the article (eg, unreliable or fabricated data or results). To ensure access to CCs, we limited our dataset to open access citing articles and excluded those that had themselves been retracted. This yielded 4979 citing articles and 7424 unique CRA instances; 88% of these were preretraction citations. We then limited the rest of the analysis to a subset of 3049 CCs from the PubMed Central Open Access corpus.&lt;sup&gt;3&lt;/sup&gt; We used regular expressions to search for terms referencing concepts related to the retraction reasons (eg, “data,” “results,” “findings”) and manually validated their use in the CCs.&lt;/p&gt;
&lt;h4&gt;Results&lt;/h4&gt;
&lt;p&gt; Of 3049 CCs, we identified 57 that explicitly mentioned the exact issue mentioned in the RR. Among these, 12 emphasized similarity or consistency with the retracted findings, which are instances we interpret as potential indicators of unreliability in the citing paper, meriting PPPR. These 57 CCs span 18 research fields and involve 60 distinct RRs (some overlapping with the selected 29). The cases are distributed across a range of publishers and fields, offering a basis to be used in the development of automatic systems.&lt;/p&gt;

![](https://assets.underline.io/markdown_image/1/image/c740d6d31d9977a9f35654827d4b160b.png)

&lt;h4&gt;Conclusions&lt;/h4&gt;
&lt;p&gt; While retracted publications are unreliable sources, we argue that how the retracted work is used and why it was retracted matter more than the fact that a retracted reference is found in a publication. If the retracted source is used to support the citing work, this calls for a secondary assessment when it is detected. By combining RRs with CC analysis, our approach informs a practical method to flag such cases for PPPR and supports the development of tools for large-scale CRA assessment.&lt;/p&gt;
&lt;h4&gt;References&lt;/h4&gt;
&lt;p&gt;1. The Center for Scientific Integrity. Retraction Watch Database.&lt;a href=&quot;https://retractiondatabase.org/&quot;&gt;&lt;span class=&quot;Hyperlink CharOverride-6&quot;&gt; &lt;/span&gt;&lt;/a&gt;&lt;a href=&quot;https://retractiondatabase.org/&quot;&gt;&lt;/a&gt;&lt;/p&gt;
&lt;p&gt;2. Cabanac G, Labbé C, Magazinov A. The ‘Problematic Paper Screener’ automatically selects suspect publications for post-publication (re)assessment. &lt;i&gt;arXiv&lt;/i&gt;. Preprint posted online October 7, 2022. doi:10.48550/arXiv.2210.04895&lt;/p&gt;
&lt;p&gt;3. National Library of Medicine. PMC Open Access Subset. Accessed January 30, 2025. &lt;a href=&quot;https://pmc.ncbi.nlm.nih.gov/tools/openftlist/&quot;&gt;&lt;span class=&quot;Hyperlink CharOverride-6&quot;&gt;https://pmc.ncbi.nlm.nih.gov/tools/openftlist/&lt;/span&gt;&lt;/a&gt;&lt;/p&gt;
&lt;sup&gt;1&lt;/sup&gt;Univ. Grenoble Alpes, CNRS, Grenoble INP, LIG, Grenoble, France, yagmur.ozturk@univ-grenoble-alpes.fr; &lt;sup&gt;2&lt;/sup&gt;Ecole Nationale des Ponts et Chaussées, Institut Polytechnique de Paris, Marne-la-Vallée, France; &lt;sup&gt;3&lt;/sup&gt;LISIS, INRAE, Univ Gustave Eiffel, CNRS, Marne-la-Vallée, France.&lt;/p&gt;
&lt;h4&gt;&lt;a id=&quot;_idTextAnchor124&quot;&gt;&lt;/a&gt;Conflict of Interest Disclosures&lt;/h4&gt;
&lt;p&gt; None reported.&lt;/p&gt;
&lt;h4&gt;Funding/Support&lt;/h4&gt;
&lt;p&gt; We acknowledge the NanoBubbles project that has received Synergy grant funding from the European Research Council within the European Union’s Horizon 2020 program (grant agreement number 951393) (&lt;a href=&quot;https://cordis.europa.eu/project/id/951393&quot;&gt;&lt;span class=&quot;Hyperlink CharOverride-6&quot;&gt;https://cordis.europa.eu/project/id/951393&lt;/span&gt;&lt;/a&gt;).&lt;/p&gt;


The 10th Peer Review Congress

Citation Context Analysis of Retracted Articles: Leveraging Retraction Reasons to Track Unreliability in Citing Literature

postpublication peer review

retractions

peer review

The Tenth Peer Review Congress featured 3 days for presentations of new research into peer review and all aspects of scientific publication, bias, quality of reporting, and information access and dissemination.

This year’s meeting was hybrid, and all plenary sessions were livestreamed with opportunities to view presentations and engage with participants in person and virtually during and after the meeting. All recordings are now available on Underline for viewing.

<div style="position:relative;padding-top:max(60%,324px);width:100%;height:0;"><iframe style="position:absolute;border:none;width:100%;height:100%;left:0;top:0;" src="https://online.fliphtml5.com/ebtyf/mget/"  seamless="seamless" scrolling="no" frameborder="0" allowtransparency="true" allowfullscreen="true" ></iframe></div>


#### Why? 
Our aim is to encourage research into the quality and credibility of peer review and scientific publication, to establish the evidence base on which scientists can improve the conduct, reporting, and dissemination of scientific research.

#### Who? 
Participants include editors and publishers of scientific peer-reviewed journals, researchers, funders, bibliometric and informatics experts, information innovators, librarians, journalists, policymakers, ethicists, scientific information disseminators, and anyone interested in the progress of the scientific information enterprise and the quality of scientific evidence.

The Congress embraces a wide range of disciplines, including biomedicine, health and life sciences, applied sciences, basic sciences, physical and chemical sciences, mathematics, computer sciences, engineering, economics, and social sciences as well as new and emerging disciplines.

#### What? 
The Congress provides a forum for the presentation and discussion of new research into the quality and credibility of editorial and funding peer review, scientific and scholarly publication, research methods and reporting practices, identification and management of biases, transparency, reproducibility, access, publication and dissemination metrics, identification and management of threats to the integrity of science, and advances in information exchange.

**Follow us**

[![](https://assets.underline.io/markdown_image/1/image/662a22766b9788ed3766cb23cd156e32.png)](https://x.com/peerrevcongress)  [![](https://assets.underline.io/markdown_image/1/image/bb105afb1773a01fa169aa97ca1af49a.png)](https://www.youtube.com/channel/UCsG_0nKi3_Zfyv0yRPdxjWQ)  [![](https://assets.underline.io/markdown_image/1/image/034e1e70a194a9a4178fdf470ef796db.png)](https://bsky.app/profile/peerreviewcongress.bsky.social)



You need to log in with the email address you registered with. 



Please log in to explore this event.

Registration is required to attend this event, please [here](https://peerreviewcongress.org/registration-information/) to secure your ticket. Please check your spam/promotional inbox too.



Please register!

The Tenth Peer Review Congress features 3 days for presentations of new research into peer review and all aspects of scientific publication, bias, quality of reporting, and information access and dissemination.

<p>Yagmur Ozturk,<sup>1</sup> Frédérique Bordignon,<sup>2,3</sup> Cyril Labbé,<sup>1</sup> François Portet<sup>1</sup></p>
<h4>Objective</h4>
<p> To address the challenge of tracking the use of unreliable sources in scientific literature, we analyzed citations to retracted articles (CRAs) using retraction reasons (RRs) and citation context (CC) analysis. Our aim was to inform the development of automated tools for postpublication peer review (PPPR) by identifying instances in which a CRA is not only an unreliable source, but may also introduce unreliability into the citing work itself, depending on the way it is cited. We examined the degree to which the content of the citation context aligns with the RR, indicating potential risk for the integrity of the citing article.</p>
<h4>Design</h4>
<p> RRs were obtained from the Retraction Watch database<sup>1</sup> for a set of 22,558 publications that cite retracted sources accessed from the Problematic Paper Screener.<sup>2</sup> We selected 29 of 108 retraction reasons that directly relate to scientific content of the article (eg, unreliable or fabricated data or results). To ensure access to CCs, we limited our dataset to open access citing articles and excluded those that had themselves been retracted. This yielded 4979 citing articles and 7424 unique CRA instances; 88% of these were preretraction citations. We then limited the rest of the analysis to a subset of 3049 CCs from the PubMed Central Open Access corpus.<sup>3</sup> We used regular expressions to search for terms referencing concepts related to the retraction reasons (eg, “data,” “results,” “findings”) and manually validated their use in the CCs.</p>
<h4>Results</h4>
<p> Of 3049 CCs, we identified 57 that explicitly mentioned the exact issue mentioned in the RR. Among these, 12 emphasized similarity or consistency with the retracted findings, which are instances we interpret as potential indicators of unreliability in the citing paper, meriting PPPR. These 57 CCs span 18 research fields and involve 60 distinct RRs (some overlapping with the selected 29). The cases are distributed across a range of publishers and fields, offering a basis to be used in the development of automatic systems.</p>

![](https://assets.underline.io/markdown_image/1/image/c740d6d31d9977a9f35654827d4b160b.png)

<h4>Conclusions</h4>
<p> While retracted publications are unreliable sources, we argue that how the retracted work is used and why it was retracted matter more than the fact that a retracted reference is found in a publication. If the retracted source is used to support the citing work, this calls for a secondary assessment when it is detected. By combining RRs with CC analysis, our approach informs a practical method to flag such cases for PPPR and supports the development of tools for large-scale CRA assessment.</p>
<h4>References</h4>
<p>1. The Center for Scientific Integrity. Retraction Watch Database.<a href="https://retractiondatabase.org/"><span class="Hyperlink CharOverride-6"> </span></a><a href="https://retractiondatabase.org/"></a></p>
<p>2. Cabanac G, Labbé C, Magazinov A. The ‘Problematic Paper Screener’ automatically selects suspect publications for post-publication (re)assessment. <i>arXiv</i>. Preprint posted online October 7, 2022. doi:10.48550/arXiv.2210.04895</p>
<p>3. National Library of Medicine. PMC Open Access Subset. Accessed January 30, 2025. <a href="https://pmc.ncbi.nlm.nih.gov/tools/openftlist/"><span class="Hyperlink CharOverride-6">https://pmc.ncbi.nlm.nih.gov/tools/openftlist/</span></a></p>
<sup>1</sup>Univ. Grenoble Alpes, CNRS, Grenoble INP, LIG, Grenoble, France, yagmur.ozturk@univ-grenoble-alpes.fr; <sup>2</sup>Ecole Nationale des Ponts et Chaussées, Institut Polytechnique de Paris, Marne-la-Vallée, France; <sup>3</sup>LISIS, INRAE, Univ Gustave Eiffel, CNRS, Marne-la-Vallée, France.</p>
<h4><a id="_idTextAnchor124"></a>Conflict of Interest Disclosures</h4>
<p> None reported.</p>
<h4>Funding/Support</h4>
<p> We acknowledge the NanoBubbles project that has received Synergy grant funding from the European Research Council within the European Union’s Horizon 2020 program (grant agreement number 951393) (<a href="https://cordis.europa.eu/project/id/951393"><span class="Hyperlink CharOverride-6">https://cordis.europa.eu/project/id/951393</span></a>).</p>


poster

<p>Caitlin J. Bakker,<sup>1,2</sup> Erin E. Reardon,<sup>3</sup> Nicole Theis-Mahon,<sup>4</sup> Sara Schroter,<sup>5,6</sup> Lex Bouter,<sup>7,8</sup> Maurice P. Zeegers<sup>2</sup></p>
<h4>Objective</h4>
<p> Systematic reviews gather, appraise, and synthesize studies to inform research, practice, and policy. However, the inclusion of retracted articles, which often contain flaws and falsified or fabricated data, undermines the credibility of systematic reviews. Identifying retracted articles is challenging, as they are inconsistently flagged.<sup>1,2</sup> Our study validated and compared approaches to identify retracted articles and their retraction notices.</p>
<h4>Design</h4>
<p> Our study, guided by an advisory panel of information specialists and researchers, evaluated approaches for identifying retracted publications from 8 health sciences databases (Cochrane Library, Embase.com, Ovid Embase, Ovid Medline, Ovid PsycINFO, PubMed, Scopus, and Web of Science). Using a reference set of 43,544 retracted publications and 27,755 associated retraction notices from Retraction Watch, we identified items found in each database. From August 10 to 14, 2024, we determined how many of the available items could be retrieved using each approach per database. Two search strategies, database indexing, and 2 citation managers were evaluated. The complete methodology, including search strategies, is available in our protocol.<sup>3</sup> Recall (sensitivity) was calculated to evaluate identification effectiveness.</p>
<h4>Results</h4>
<p> Recall of retracted publications and notices varied across databases and retrieval approaches. Across databases, search strategy 2 consistently achieved the highest recall, with values ranging from 74.6% to 96.9%. Search strategy 1 also performed strongly, particularly in PubMed and Web of Science (both &gt;93%). In contrast, indexing-based retrieval showed variable performance, with high recall in PubMed (94.6%) and Ovid Medline (94.5%) but much lower in Embase.com (40.6%) and PsycINFO (34%). Citation manager tools (EndNote and Zotero) yielded lower recall, with values rarely exceeding 64%. Recall was lowest for Ovid Embase and PsycINFO regardless of method, while PubMed and Web of Science showed the highest recall.</p>

![](https://assets.underline.io/markdown_image/1/image/b3957ae8a92f24f08736a44da92e7d56.png)

<h4>Conclusions</h4>
<p> There was substantial variability in the ability of databases and retrieval approaches to identify retracted publications and notices. No single approach captured all items, underscoring the need for multiple approaches in an iterative identification process. </p>
<h4>References</h4>
<p>1. Bakker CJ, Reardon EE, Brown SJ, et al. Identification of retracted publications and completeness of retraction notices in public health. <i>J Clin Epidemiol</i>. 2024;173:111427. doi:10.1016/j.jclinepi.2024.111427</p>
<p>2. Boudry C, Howard K, Mouriaux F. Poor visibility of retracted articles: a problem that should no longer be ignored. <i>BMJ</i>. 2023;381:e072929. doi:10.1136/bmj-2022-072929</p>
<p>3. Bakker C, Reardon EE, Theis-Mahon NR, Schroter S, Bouter L, Zeegers M. Validation and comparison of methods to identify retracted publications during information retrieval. Open Science Framework. Cited June 7, 2025. <a href="https://osf.io/rwzym/"><span class="Hyperlink CharOverride-6">https://osf.io/rwzym/</span></a></p>
<sup>1</sup>University of Regina, Regina, SK, Canada, caitlin.bakker@uregina.ca; <sup>2</sup>Maastricht University, Maastricht, the Netherlands; <sup>3</sup>Emory University, Atlanta, GA, US; <sup>4</sup>University of Minnesota, Minneapolis, US; <sup>5</sup>BMJ, London, UK; <sup>6</sup>London School of Hygiene and Tropical Medicine, London, UK; <sup>7</sup>Amsterdam University Medical Center, Amsterdam, the Netherlands; <sup>8</sup>Vrije Universiteit Amsterdam, Amsterdam, the Netherlands.</p>
<h4>Conflict of Interest Disclosures </h4>
<p>Caitlin J. Bakker is cochair of the National Information Standards Organization Communication of Retractions, Removals and Expressions of Concern Standing Committee. No other disclosures were reported.</p>
<h4>Funding/Support </h4>
<p>This research is part of an ongoing PhD collaboration between BMJ and the team Meta-Research at Maastricht University (UM) on the responsible conduct of publishing scientific research. BMJ is published by BMJ Group, a wholly owned subsidiary of the British Medical Association. UM is a public legal entity in the Netherlands. This study is part of Caitlin J. Bakker’s self-funded BMJ/UM PhD. No exchange of funds has taken place for this research project.</p>
<h4>Role of the Funder/Sponsor</h4>
<p> The authors are wholly responsible for the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, and approval of the abstract; and decision to submit the abstract for presentation.</p>
<h4>Additional Information</h4>
<p> All authors express their own opinions and not necessarily that of their employers.</p>


Evaluating Approaches for Identifying Retracted Articles and Retraction Notices in Systematic Review Searching

<p>Guillaume Cabanac,<sup>1,2</sup> Kathryn Weber-Boer<sup>3</sup></p>
<h4>Objective</h4>
<p> We analyzed postretraction citations of retracted publications in biomedical and clinical sciences (BCS). Previous studies have investigated the citation of retracted clinical trials<sup>1</sup> and the citation of retracted publications in policy documents,<sup>2</sup> but the share of references cited after their retraction in BCS literature is unknown. We identified the articles citing postretracted references and present the results aggregated at the journal and publisher levels.</p>
<h4>Design</h4>
<p> We used Dimensions data accessed in BigQuery to delineate the corpus, including publications of type ‘article’, classified as “review article” or “research article”, with the field of research category BCS (32, in the ANZSRC 2020 classification), published between 2005 and 2024. We used the snapshot of the Dimensions dataset from 1 June 2025. These 12,240,884 articles were published in 44,901 journals from 11,961 publishers, and contained 401,909,282 references of 28,058,673 unique publications. We extracted the year of publication, journal titles, publisher names, and list of references cited. We retained those references which were removed, retracted, or withdrawn (henceforth ‘retracted’), based on data derived from Crossref, PubMed, Retraction Watch, and retraction notices issued by publishers. We excluded publications which cited retractions which had a retraction year earlier than or equal to the year of publication, to consider only “post-retraction references.”</p>
<h4>Results</h4>
<p> We observed a growing number of publications in the BSC literature that cite retracted publications. In 2005, there were 674 publications referencing 638 retracted publications 688 times. This grew to 13,927 publications referencing 14,214 retracted publications 15,339 times in 2024: a 2066% increase in publications and retracted references and a 2228% increase in retracted publications cited. This compared to an increase over the same period of 267% for publications in BSC (from 368,338 to 983,837) and 390% in citations (8,495,152 to 33,143,231). This is an increase in the proportion of publications with retracted references from to 0.18% to 1.69% of all BSC publications. The number of retracted publications referenced doubled since 2019. The connection between journal and publisher reflected the ownership relationship as of approximately January 2025. The journals in which post-retraction referencing of retracted literature occurred are many (7225 distinct journals, 17%), but over half of the publications (57,292 of 110,492) are currently owned by four major publishers: Elsevier (23,725; 21%), Springer Nature (17,727; 16%), Wiley (9488; 9%), and Frontiers (6352; 6%).<b>(Table 25-1151)</b></p>
<a href="/wp-content/uploads/2025_conference/tables/prc25-1151t1.png"> <img loading="lazy" src="/wp-content/uploads/2025_conference/tables/prc25-1151t1.png"> </a>
<h4>Conclusions</h4>
<p> The publishers of articles responsible for the most citations of postretracted references are the largest publishers of scientific literature by any measure. We could have ranked or displayed the results by publisher or journal as a percentage of publications overall; however, in this case, quantity matters more than share. This practice is preventable, with system-wide corrective efforts by the publishing industry, such as prepublication checks for retracted references, as a complement to the postpublication checks of retracted references that are also needed.<sup>3</sup></p>
<h4>References</h4>
<p>1. Kataoka Y, Banno M, Tsujimoto Y, et al. Retracted randomized controlled trials were cited and not corrected in systematic reviews and clinical practice guidelines. <i>J Clin Epidemiol</i>. 2022;150:90-97. doi:10.1016/j.jclinepi.2022.06.015</p>
<p>2. Malkov D, Yaqub O, Siepel J. The spread of retracted research into policy literature. <i>Quant Sci Stud</i>. 2023;4(1):68-90. doi:10.1162/qss_a_00243</p>
<p>3. Cabanac G. Chain retraction: how to stop bad science propagating through the literature. <i>Nature</i>. 2024;632(8027):977-979. doi:10.1038/d41586-024-02747-1</p>
<sup>1</sup>Université de Toulouse, IRIT (UMR 5505 CNRS), Toulouse, France; <sup>2</sup>Institut Universitaire de France, Paris, France; <sup>3</sup>Digital Science, London, UK, k.weberboer@digital-science.com.</p>
<h4>Conflict of Interest Disclosures </h4>
<p>Kathryn Weber-Boer is employed by Digital Science, which provided access to the Dimensions datasets that this work analyzed. Guillaume Cabanac received funding from the Institut Universitaire de France.</p>
<h4>Additional Information </h4>
<p>Guillaume Cabanac and Kathryn Weber-Boer are co–corresponding authors.</p>


Post-retraction References in Biomedical and Clinical Sciences

<p>Lonni Besançon,<sup>1</sup> Guillaume Cabanac,<sup>2,3</sup> Kathryn Weber-Boer<sup>4</sup></p>
<h4>Objective </h4>
<p>We identified retracted references in clinical guidelines published between 1991 and 2025 that were cited before and after retraction to study the scale of this contamination. Clinical guidelines have a concrete impact on the practice of medicine and, while active, are meant to reflect the state of the art. Previous work<sup>1</sup> has explored the citation of retracted trials in systematic reviews and clinical guidelines, showing both pre- and postretraction citations. We explored a population of retracted publications cited in clinical guidelines.</p>
<h4>Design </h4>
<p>We drew on 2 new datasets: clinical guidelines from Altmetric (n = 20,553) and retractions identified by Dimensions (n = 63,047). The clinical guidelines dataset<sup>2</sup> (articles indexed in Dimensions classified as clinical guidelines and documents published online by learned societies and associations) should consist of the latest version of each guideline, although as guidelines are updated, the guidelines that we studied may become obsolete. This dataset included links to guidelines and a list of publications referenced in each guideline. The retractions dataset was derived from CrossRef, PubMed, and Retraction Watch, and by matching publisher notices to publications; it included an identifier for the retracted publication and relevant linked notices with their type and date. We identified publications found both in the list of guideline references and in the list of retracted publications and selected those cited after their retraction date (398 unique guideline-retracted publication pairs). We manually examined a randomly selected 10% of the accessible English-language publications to assess the accuracy of the dataset (42 guideline-reference pairs). Our evaluation of this sample of our dataset assessed both whether the retraction identification was accurate and whether the citation of these publications in the guidelines was critical (or mentioned the retraction).</p>
<h4>Results </h4>
<p>We considered publications cited in the year of retraction to be preretraction citations. We found 487 guidelines containing 596 references to 319 publications that had been retracted after citation. Of these, 179 guidelines (37%) referenced 134 retracted publications 204 times within 5 years after retraction (<b>Figure 25-1004</b>). There were 107 guidelines (22%) containing 126 references to 73 publications that had been retracted more than 5 years before citation. All retraction datasets, including ours, contained inaccuracies. We could not find retraction notices for 4 references marked as retracted. Only 2 retracted references were cited critically (presented with some doubt as to the strength of their methods or conclusions), and none acknowledged the retracted status of the reference. Most references were used either as background literature or to support claims and recommendations made in the guidelines.</p>

![](https://assets.underline.io/markdown_image/1/image/ecbf789bf3acbffc6717850847a3a40b.png)

<h4>Conclusions </h4>
<p>Postretraction citations are preventable. The longer after the retraction appears, the more concerning the citation. In guidelines in particular, the persistence of retracted references is concerning. Our results highlight the need to systematically screen both new and existing clinical guidelines for contamination.</p>
<h4>References</h4>
<p>1. Kataoka Y, Banno M, Tsujimoto Y, et al. Retracted randomized controlled trials were cited and not corrected in systematic reviews and clinical practice guidelines. <i>J Clin Epidemiol</i>. 2022;150:90-97. doi:10.1016/j.jclinepi.2022.06.015</p>
<p>2. Altmetric. Clinical guidelines. Accessed June 19, 2025. <a href="https://www.altmetric.com/about-us/our-data/clinical-guidelines/">https://www.altmetric.com/about-us/our-data/clinical-guidelines/</a></p>
<sup>1</sup>Linköping University, Linköping, Sweden; <sup>2</sup>Université de Toulouse, Institut de Recherche en Informatique de Toulouse (UMR 5505), Centre National de la Recherche Scientifique, Toulouse, France; <sup>3</sup>Institut Universitaire de France, Paris, France; <sup>4</sup>Digital Science, London, UK, k.weberboer@digital-science.com.</p>
<h4>Conflict of Interest Disclosures</h4>
<p> Kathryn Weber-Boer is employed by Digital Science, which provided access to the Altmetric and Dimensions datasets that this work analyzed. Lonni Besançon has received a catalyst grant from Digital Science. No other disclosures were reported.</p>


Retracted Publications Referenced in Clinical Guidelines

<p>Ivan D. Florez,<sup>1,2</sup> Alberto Henriquez,<sup>3,4</sup> Andrés F. Estupinan-Bohorquez<sup>3,5</sup></p>
<h4>Objective </h4>
<p>We described the influence of retracted systematic reviews and meta-analyses (SRMAs) on clinical practice guidelines (CPGs) and the characteristics of retractions in CPGs.</p>
<h4>Design </h4>
<p>This cross-sectional study was conducted in 2 stages based on searches focused on the Retraction Watch (RW) database and MEDLINE from inception to November 30, 2024. In the first stage, we included SRMAs. We described the reasons for retractions, and we recategorized them based on our assessment. We identified the CPGs that cited the SRMA in the Google Scholar database. In the second stage, we included the retracted CPGs, described the reasons for retractions, and recategorized them into ethical or nonethical reasons based on our assessment. Nonethical reasons were categorized as editorial or administrative or outdated guidelines, while ethical reasons were reported according to RW categories. We used descriptive statistics to summarize the findings.</p>
<h4>Results </h4>
<p>In the first stage, we included 377 SRMAs, of which 211 (56.0%) were retracted due to peer review or publication manipulation (eg, detected “fake” reviewers); 30 (8.0%), due to duplicate or redundant publication; and 136 (36.1%), due to intellectual or authorship disputes, plagiarism, outdated publication, retraction of included studies, methodologic or data errors, and conflicts of interests. For 49 (13.0%), specific reasons were not provided. Of the retracted SRMAs, 41 (10.9%) were cited in CPGs; 19 (46.3%) of these SRMAs were retracted due to research integrity issues and 12 (29.3%), due to data errors or being outdated. For 10 (24.4%), specific reasons were not provided. Most retractions were due to manipulation of the publication or peer review process. The median time between publication and retraction of the SRMA used in CPGs was 12.0 (IQR, 3.5-25.0) months, and the median number of SRMA citations was 40 (IQR, 22-191). In the second stage, we included 36 CPGs of the 138 potential CPGs identified. Nine CPGs (25.0%) were retracted because of ethical reasons and 22 (61.1%) for nonethical reasons; the rest had no available information. The most common ethical reasons were plagiarism, authorship or intellectual property disputes, lack of disclosure of conflicts of interest, and discrepancies between the content and the cited evidence. Among the 22 CPGs retracted for nonethical reasons, 11 were due to dual publication or incorrect citations and 9 were due to outdated recommendations. The median publication to retraction time was 10 (range, 3-96) months. All of these CPGs were cited after their retraction date, and in all cases, the citations were used to support the background of the research studies.</p>
<h4>Conclusions </h4>
<p>Retracted SRMAs have been informing CPGs, which provide recommendations in practice and policy. The retraction of CPGs has been neglected. The RW database should be revised according to the specificities of CPGs. The most concerning reasons are ethical. Retracted CPGs continue to be cited after their retraction, mainly to inform the background sections of articles.</p>
<sup>1</sup>Department of Pediatrics, University of Antioquia, Medellín, Colombia, ivan.florez@udea.edu.co; <sup>2</sup>School of Rehabilitation Science, McMaster University, Hamilton, Ontario, Canada; <sup>3</sup>Universidad del Norte, Barranquilla, Colombia; <sup>4</sup>Universidad Metropolitana, Barranquilla, Colombia; <sup>5</sup>EPICLINICA SAS, Barranquilla, Colombia.</p>
<h4>Conflict of Interest Disclosures</h4>
<p> None reported.</p>


Retraction of Systematic Reviews and Clinical Practice Guidelines

<p>Caden Seraphine,<sup>1</sup> Abigail Chambers,<sup>1</sup> Susan Galandiuk<sup>2</sup></p>
<h4>Objective</h4>
<p> In recent years, <i>Diseases of the Colon &amp; Rectum</i> (<i>DCR</i>), a single society-owned specialty journal, sought to grow its online presence. The journal launched a YouTube channel in 2018, featuring videos of monthly English-language journal clubs (JCs) discussing topical publications beginning in 2020, followed by article discussions in Spanish (2022-2023), Japanese (2024), and Korean (2024). The journal maintained its Facebook and X (formerly Twitter) presence and started LinkedIn and Instagram accounts in 2023. Articles were given a nonpaywalled status for 1 month, and upcoming discussions with article hyperlinks were announced 4 times per month via social media. Articles were accessible by hyperlinks, and article citations were displayed in JC discussant slides. We sought to explore how these activities impacted measures of journal interaction and global reach.<sup>1,2</sup></p>
<h4>Design</h4>
<p> We performed a 7-year (2018-2024) retrospective cohort study of JC video-based educational content provided by <i>DCR</i> with respect to journal website and social media access following STROBE guidelines.<sup>3</sup> We examined efforts toward global engagement, including addition of journal-specific non–English-language (2 Spanish, 1 Korean, and 1 Japanese) discussions. Analyses of 4 popular social media platforms’ native analytics reporting and website engagement data examined viewer demographics (age, language, viewing country, and traffic source) and journal web-based interaction (number of views, number of unique visitors, and country of access) by month and year, trends in individual JC videos, and overall page engagement. We compared the median number of citations of JC articles (cumulative Web of Science citations) and their Altmetric scores (indicating article attention received) with mean citations of <i>DCR</i> articles per year.</p>
<h4>Results</h4>
<p> Only 1 JC featured videos of surgical procedures; all others discussed original articles. The growth in YouTube engagement since the journal’s YouTube channel launched in 2018 grew exponentially with the addition of monthly JCs in 2020. LinkedIn and Instagram were implemented in 2023, and cumulative traffic to the journal’s website from social media platforms tripled within 1 year, from 15,414 clicks in 2023 to 56,609 in 2024 (Facebook, X, Instagram, LinkedIn, and YouTube). Throughout the period, individuals aged 25 to 34 years led YouTube viewership. International engagement increased with <i>DCR</i>’s implementation of non-English JC discussions in 2022. Publications discussed during JCs consistently had higher Web of Science citations compared with the mean citations in the journal that year (<span class="CharOverride-4"><b>Table 25-0959</b></span>).</p>

![](https://assets.underline.io/markdown_image/1/image/dd454d0ce367b165c77b22150974e865.png)

<h4>Conclusions</h4>
<p> Younger audiences engage beyond traditional platforms of scholarly publishing in ways that hold inherent educational value. Journal Impact Factors exclude this reach and overlook the impact of web-based peer-reviewed educational content. Inclusion of social media and video-based learning tools, as well as efforts toward global reach, contribute to journal impact.</p>
<h4>References</h4>
<p>1. Narayan RR, Fleming AM, Gunder M, et al. Reflections from the Annals of Surgical Oncology social media committee: the impact of promoting surgical science online. Ann Surg Oncol. 2025;32(2):656-664. doi:10.1245/s10434-024-16420-4</p>
<p>2. Özkent Y. Social media usage to share information in communication journals: an analysis of social media activity and article citations. PLoS One. 2022;17(2):e0263725. doi:10.1371/journal.pone.0263725</p>
<p>3. von Elm E, Altman DG, Egger M, Pocock SJ, Gøtzsche PC, Vandenbroucke JP; STROBE Initiative. The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies. J Clin Epidemiol. 2008;61(4):344-349. doi:10.1016/j.jclinepi.2007.11.008</p>
<sup>1</sup>Price Institute of Surgical Research, University of Louisville, Louisville, KY, US; <sup>2</sup>Division of Colorectal Surgery, Department of Surgery, University of Louisville, Louisville, KY, US, s0gala01@louisville.edu.</p>
<h4>Conflict of Interest Disclosures</h4>
<p> Susan Galandiuk receives a stipend for services as editor for the journal Diseases of the Colon &amp; Rectum from the American Society of Colon and Rectal Surgeons. No other disclosures were reported.</p>
<h4>Acknowledgments</h4>
<p> Margaret Abby (Managing Editor, Diseases of the Colon &amp; Rectum; Research Grants Coordinator, Department of Surgery, University of Louisville, Louisville, KY), helped with obtaining data on articles discussed during journal clubs and their Web of Science citations.</p>


Video and Social Media Performance at a Surgical Journal with Video Journal Clubs

<p>Jeremy Y. Ng,<sup>1,2,3</sup> Daivat Bhavsar,<sup>4</sup> Laura Duffy,<sup>4</sup> Hamin Jo,<sup>4</sup> Cynthia Lokker,<sup>4</sup> R. Brian Haynes,<sup>4,5</sup> Alfonso Iorio,<sup>4,5</sup> Ana Marušić<sup>6</sup></p>
<h4>Objective </h4>
<p>This study examined the policies implemented by academic publishers regarding authors’ use of generative artificial intelligence (GenAI) tools, focusing on their regulation, disclosure requirements, and role in ensuring the integrity of scientific publications. By analyzing the prevalence and content of these policies, this study aimed to provide insight into the current landscape and inform future policy development in the rapidly evolving field of artificial intelligence (AI)–assisted research and publication.</p>
<h4>Design </h4>
<p>A cross-sectional audit was conducted on the publicly available policies of 163 academic publishers listed as members of the International Association of Scientific, Technical, and Medical Publishers. Policies were collected and analyzed between September 1 and December 31, 2023. Publishers without publicly accessible policies specific to GenAI use by authors were excluded. Data extraction and analysis were conducted independently in duplicate, with a third reviewer resolving discrepancies. The key policy components analyzed included authorship accreditation, disclosure requirements, and permissions for tasks such as research methods, content generation, image creation, and proofreading. Descriptive statistics were used to summarize the findings. Our protocol was registered.<sup>1</sup></p>
<h4>Results </h4>
<p>Of 163 academic publishers, 56 (34.4%) had publicly available policies guiding GenAI use by authors. None permitted authorship accreditation for AI tools, citing accountability concerns and alignment with ethical guidelines. Nearly all publishers with policies (49 of 56 [87.5%]) mandated disclosure of GenAI use, primarily in the Methods or Acknowledgments section. However, disclosure practices varied, with some publishers providing standardized templates while others left requirements vague. Four publishers completely prohibited GenAI use in manuscript preparation, while others allowed their use for specific tasks. Most (33 of 56 [58.9%]) publishers permitted GenAI for drafting nonmethodological sections (eg, Introductions), while 18 (32.1%) permitted their use in research methods, such as data analysis and organization. Few publishers addressed GenAI use in image generation (14 of 163 [8.6%]) or proofreading (15 of 163 [9.2%]). Only 1 publisher (0.6%) allowed citation of AI as primary sources, while 19 (11.6%) explicitly prohibited such citations. Our study has been published.<sup>2</sup></p>
<h4>Conclusions </h4>
<p>This audit highlights the inconsistent development of GenAI policies among academic publishers, with large variability in scope and clarity. While the prohibition of AI authorship and the emphasis on mandatory disclosure are consistent themes, inconsistencies in regulating specific tasks suggest a need for standardized and comprehensive policies. As AI technology and its applications in research evolve, publishers must adapt to safeguard scientific integrity. Given that the AI landscape is fast moving, future research includes updating this audit and comparing and contrasting current policies with those found in this study. Future research should also assess how policies are implemented and enforced by examining samples of published articles, as well as explore how these policies affect editors and reviewers, taking into account potential risks, such as privacy breaches and bias.</p>
<h4>References</h4>
<p>1. Bhavsar D, Lokker C, Haynes RB, Iorio A, Marusic A, Ng JY. Academic publisher artificial intelligence chatbot policies for authors: a cross-sectional audit. OSF Registries. Accessed July 11, 2025. doi:10.17605/OSF.IO/937ES</p>
<p><span lang="fr-FR">2. Bhavsar D, Duffy L, Jo H, et al. Policies on artificial intelligence chatbots among academic publishers: a cross-sectional audit. </span><i>Res Integr Peer Rev</i><span lang="fr-FR">. 2025;10(1):1. </span>doi:10.1186/s41073-025-00158-y</p>
<sup>1</sup>Institute of General Practice and Interprofessional Care, University Hospital Tübingen, Tübingen, Germany, jeremyyng.phd@gmail.com; <sup>2</sup>Robert Bosch Center for Integrative Medicine and Health, Bosch Health Campus, Stuttgart, Germany; <sup>3</sup>Centre for Journalology, Ottawa Hospital Research Institute, Ottawa, Canada; <sup>4</sup>Department of Health Research Methods, Evidence, and Impact, Faculty of Health Sciences, McMaster University, Hamilton, Ontario, Canada; <sup>5</sup>Department of Medicine, McMaster University, Hamilton, Ontario, Canada; <sup>6</sup>Department of Research in Biomedicine and Health and Center for Evidence-Based Medicine, University of Split School of Medicine, Split, Croatia.</p>
<h4>Conflicts of Interest Disclosures </h4>
<p>The authors declare no conflicts of interest. Ana Marušić is a member of the Peer Review Congress Advisory Board but was not involved in the review or decision for this abstract.</p>


[V] Policies on Artificial Intelligence Among Academic Publishers

<p>Eva Sarachaga,<sup>1</sup> Ines Steffens<sup>1</sup></p>
<h4>Objective </h4>
<p>In September 2023, <i>Eurosurveillance</i> adopted an artificial intelligence (AI) policy encouraging authors and reviewers to use AI tools responsibly. The policy states that authors should disclose tools such as large language models (LLMs), chatbots, and image-generating algorithms used in the production and writing of the manuscript. It further outlines that reviewers should declare whether they have used a chatbot or LLM tool in the generation of their review or in their correspondence to authors or editors.<sup>1</sup> Fifteen months after launching the policy, we investigated how many authors and reviewers had declared using AI tools and how. Additionally, we examined whether possible AI use was flagged by the plagiarism detection system, iThenticate, which checks whether incoming submissions overlap with published material.</p>
<h4>Design </h4>
<p>Data were extracted from the article submission system, Editorial Manager. In a cross-sectional design, we included all articles and reviews in the system since the policy was published in September 2023 until November 2024. Results were analyzed by domain, AI tool, and tool version. To investigate whether results from iThenticate differed between articles where AI use was declared versus where it was not declared, we compared scores of a convenience sample of 43 articles in each group and checked manually thereafter.</p>
<h4>Results </h4>
<p>Only 7% of authors (70 of 961) and 2% of reviewers (11 of 707) declared using AI tools. Language was the most frequent domain in both groups: 61 of 70 authors and 10 of 11 reviewers. Authors declared other domains, such as coding, data analysis, prediction identification, correspondence, and figure creation. ChatGPT was the most used AI tool in both groups: 54 of 70 authors and 6 of 11 reviewers. The second most used AI tools were DeepL for authors (2 of 70) and Paperpal for reviewers (2 of 11). Our results were in line with results from other studies.<sup>2,3</sup> AI use was not declared according to the policy: only 20% of authors (13 of 70) and 20% of reviewers (2 of 11) declared the tool version as required. The plagiarism score was lower in articles where AI use was declared (35 of 43) versus where it was not declared (39 of 43).</p>
<h4>Conclusions </h4>
<p>In line with findings by others, only a limited proportion of authors and reviewers declared using AI tools following implementation of an AI policy. The proportion was similar for authors and reviewers, with language being the most frequent domain and ChatGPT the most used tool in both groups. Authors and reviewers need to be reminded to comply better with the AI policy and to declare tool version.</p>
<h4>References</h4>
<p>1. E<span class="text-italic">urosurveillance. </span>Editorial policy: Responsible use of artificial intelligence (AI) tools. Accessed July 10, 2025. <a id="_idTextAnchor012"></a><a href="https://www.eurosurveillance.org/editorial-policy"><span class="Hyperlink CharOverride-6">https://www.eurosurveillance.org/editorial-policy#AI%20policy</span></a></p>
<p>2. Salvagno M, De Cassai A, Zorzi S, et al. The state of artificial intelligence in medical research: a survey of corresponding authors from top medical journals. <i>PLOS One</i>. 2024;19(8):e0309208. doi:10.1371/journal.pone.0309208</p>
<p>3. Else H. Should researchers use AI to write papers? Group aims for community-driven standards. <em>Science</em>. 2024;384(6693). doi:10.1126/science.z9gp5zo</p>
<sup>1</sup>European Centre for Disease Prevention and Control (ECDC), Stockholm, Sweden, evasarachaga01@gmail.com.</p>
<h4>Conflict of Interest Disclosures </h4>
<p>None reported.</p>
<h4>Acknowledgment</h4>
<p> We would like to thank the Eurosurveillance editorial team (Alina Buzdugan, Anirban Dey, Kathrin Hagmaier, Megan Osler, Elina Tast-Lahti, and Karen Wilson), with special thanks to Kathrin Hagmaier, at the ECDC who provided their feedback on this work.</p>


[V] Use of Generative Artificial Intelligence Tools by Authors and Reviewers of of the Journal Eurosurveillance

<p>Pawin Taechoyotin,<sup>1</sup> Daniel E. Acuna<sup>1</sup></p>
<h4>Objective </h4>
<p>Nowadays, there is an excessive load on human reviewers to the point where the quality of peer reviews is often compromised. To alleviate this excessive load, we explored the possibility of a multiagent AI peer review panel that we have developed that considers text, figures, and citations to produce peer reviews. This study is based on the advancement of large language models<sup>1,2</sup> and the study of AI agents for peer review,<sup>3</sup> which so far only utilize the textual content within the manuscript.</p>
<h4>Design</h4>
<p> The AI panel consisted of a leader agent, an experiment agent, an impact agent, and a clarity agent. Each agent utilized zero-shot prompting to the LLM model Claude Sonnet 3.5. Prompt engineering was used to craft the system and task prompt. The system prompts guided the agents to assume their specific role, and the task prompts provided clear instructions on the tasks to perform. Novelty was assessed by comparing the manuscript with similar articles from an external database of published literature. The manuscript, novelty assessment, and figures were sent to each agent for analysis, and the leader agent compiled the information from all agents, requested clarifications if needed, and produced the final review. Our AI multiagent panel was evaluated and compared with human reviews as well as simple prompt-based review<sup>2</sup> and findings of 2 previous studies.<sup>1,3</sup> Evaluation was done by graduate students via an arena where pairs of systems were selected based on their Elo score. The Elo score is a measure of the relative capability of each system based on their win/loss in the arena against other systems and was calculated using the Bradley-Terry model. The system names were hidden from the participants to reduce bias. The participants evaluated the reviews based on technical quality, constructiveness, clarity, and overall assessment before making their choice.</p>
<h4>Results</h4>
<p> Eleven graduate students in the computer science department participated in this study, which resulted in 140 matches between pairs of systems. The systems with similar Elo scores were most likely to be paired together. Mean Elo scores of the reviews produced from our system were higher than those of human review scores (<b>Table 25-0971</b>). Further analysis showed that reviews produced by our system included more detailed assessment of the limitations and possible improvements of the manuscript than reviews produced by humans. However, our system tended to overly praise work, which humans rarely do.</p>

![](https://assets.underline.io/markdown_image/1/image/e9b3e219d84889a96133d3b20f6fe73d.png)

<h4>Conclusions</h4>
<p> Our AI peer review system demonstrated potential in aiding the peer review process of publications as an addition to human reviews. It might act as a quick response to authors while human reviewers are reviewing the manuscript. Although there might be bias within the review, the author can choose to verify and utilize comments from this quick feedback as needed. Harm or the potential for negative effects were not studied. We acknowledge that the number of participants in this study was small. Thus, a larger study might have different results from our findings.</p>
<h4>References</h4>
<p>1. Liang W, Zhang Y, Cao H, et al. Can large language models provide useful feedback on research papers—a large-scale empirical analysis. <i>NEJM AI</i>. 2024;1(8):AIoa2400196. doi:10.1056/AIoa2400196</p>
<p>2. Wang H, Fu T, Du Y, et al. Scientific discovery in the age of artificial intelligence. <i>Nature</i>. 2023;620(7972):47-60. doi:10.1038/s41586-023-06221-2</p>
<p>3. D’Arcy M, Hope T, Birnbaum L, Downey D. Marg: multiagent review generation for scientific papers. <i>arXiv</i>. Preprint posted online January 8, 2024. doi:10.48550/arXiv.2401.04259</p>
<sup>1</sup>Science of Science and Computational Discovery Lab, Department of Computer Science, University of Colorado Boulder, Boulder, CO, US, daniel.acuna@colorado.edu.</p>
<h4>Conflict of Interest Disclosures</h4>
<p> Daniel E. Acuna reported being the founder of ReviewerZero AI LLC (<a href="https://www.reviewerzero.ai"><span class="Hyperlink CharOverride-6">https://www.reviewerzero.ai</span></a>), a company focused on research integrity solutions, which may be relevant to the topic of this publication.</p>


[V] Use of an AI Peer Review Panel to Assess Clarity, Novelty, and Impact

<p>Yidan Sun,<sup>1</sup> Mayank Kejriwal<sup>2</sup></p>
<h4>Objective </h4>
<p>Platforms like OpenReview<sup>1</sup> have become a popular standard for open peer review in conference-oriented fields such as computing.<sup>2</sup> Using OpenReview data from a machine learning conference, we examined (1) whether higher reviewer confidence is associated with more polarized scoring and (2) whether greater score variability is associated with rejection among papers with neutral mean scores. We also explored the assistive potential of a language model to assign sentiment labels to reviews of neutrally rated papers based on text alone.</p>
<h4>Design </h4>
<p>We used the OpenReview application planning interface to collect data on all International Conference on Learning Representations 2019 Conference Track submissions submitted by the September 27, 2018, deadline. The dataset included 4332 reviews across 1419 papers. Each review contained an integer score (1-10), a confidence rating (1-5), and a final accept or reject decision from a meta-reviewer. Analyses were conducted in January 2025. We used all 4332 reviews to assess whether reviewer confidence was associated with score extremity, defined as the absolute difference between a review score and the rubric midpoint (5), using Pearson correlation. We focused on 775 neutral<i> </i>papers—those with all review scores between 4 and 7 (inclusive)—and measured score variability as the SD of scores within each paper. We compared variability between accepted and rejected papers using 2-sided <i>t</i> tests. We then trained a Bidirectional Encoder Representations From Transformers (BERT)–based sentiment classifier<sup>3</sup> using polarized reviews scored less than 4 (negative) and greater than 7 (positive). The model achieved 90.9% accuracy on held-out review-level validation data. We then applied the model to all 2356 reviews from the 775 neutral papers and took the mean of predicted sentiment scores across each paper to produce a paper-level sentiment score. We labeled the top 246 papers—matching the number of actually accepted neutral papers—as “Accept” based on model sentiment. As a baseline, we ranked papers by mean score and labeled the top 246 as “Accept” by score.</p>

<h4>Results</h4> Reviewer confidence was positively correlated with score extremity (<i>r</i> = 0.15 [95% CI, 0.10-0.20]; <i>P</i> &lt; .001). Among the 775 neutral papers, 246 (31.7%) were accepted and 529 (68.3%) were rejected (<b>Table 25-1167</b>). Rejected neutral papers showed greater score variability than accepted papers (mean SD of 0.80 vs 0.59; difference, 0.21 [95% CI, 0.15-0.28]; <i>P</i> &lt; .001), supporting both hypotheses. Of the 246 papers labeled “Accept” by BERT, 137 were accepted. Of the 246 labeled “Accept” by the baseline, 203 were accepted. Among rejected papers, both methods labeled 20 as “Accept.”</p>

![](https://assets.underline.io/markdown_image/1/image/e390106030daa9fb11f54986bee7520b.png)

<h4>Conclusions</h4> Higher reviewer confidence was associated with more decisive scoring. Rejected neutral<i> </i>papers showed greater disagreement among reviewers. Despite moderate scores, review text often conveyed clear positive or negative sentiment, as inferred by the BERT model.</p>
<h4>References</h4>
<p class="abstract_abstract-text ParaOverride-4">1. OpenReview: an open platform for peer review. Accessed July 15, 2025. <a href="https://openreview.net"><span class="Hyperlink CharOverride-6">https://openreview.net</span></a></p>
<p class="abstract_abstract-text ParaOverride-4">2. Ford E. Defining and characterizing open peer review: a review of the literature<i>. J Scholarly Pub</i>. 2013;44(4):311-326. doi:10.3138/jsp.44-4-00</p>
<p class="abstract_abstract-text ParaOverride-3">3. Wolf T, Debut L, Sanh V, et al. Transformers: state-of-the-art natural language processing. In <i>Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations</i>. Association for Computational Linguistics; 2020:38-45.</p>
<sup>1</sup>Department of Industrial and Systems Engineering, University of Southern California, Los Angeles, CA, US, yidans@usc.edu; <sup>2</sup>Research Associate Professor, Principal Scientist, Information Sciences Institute, University of Southern California, Los Angeles, CA, US.
<h4>Conflict of Interest Disclosures</h4>
<p> None reported.</p>


[V] Reviewer Rating Variability and Confidence and Language Model Sentiment Prediction of Machine Learning Conference Papers

<p>Maud Bernisson<sup>1,2</sup></p>
<h4>Objective</h4>
<p> Researchers have raised concerns about scientific articles written by medical education and communication companies (MECCs) after litigations revealed that MECCs have been using unethical practices like ghostwriting.<sup>1</sup> Little is known about the literature produced by medical writing. While previous methods to assess this literature include surveying authors about ghostwriting,<sup>2,3</sup> this study offers a new method to collect this literature and makes it available to further explore medical writing.</p>
<h4>Design</h4>
<p> The method of collection of this literature used 3 different sources: (1) Web of Science, from which I collected 26,858 DOIs of articles, abstracts, and documents mentioning “medical writ*” in the acknowledgments; (2) the websites of 15 MECCs, found during the mapping of MECCs for another study, that publicize the literature they produce, and from which I collected 4555 additional DOIs; and (3) the Industry Documents Library of University of California, San Francisco (<a href="https://www.industrydocuments.ucsf.edu"><span class="Hyperlink CharOverride-6">https://www.industrydocuments.ucsf.edu</span></a>), in which I found the publication trackers of 5 companies and collected 229 DOIs of publications. The final database includes the metadata of 31,642 documents published between 1972 and 2025, collected from Web of Science, cleaned in R, and analyzed with the library dplyr (version 1.1.4).</p>
<h4>Results</h4>
<p> Transparency practices of disclosure of medical writing differed according to data sources. Only 252 of 4784 publications (5.3%) acknowledged medical writing in the Industry Documents Library and the MECCs websites. In contrast, a timeline of the literature collected from Web of Science identified an increase in transparency practices, with 138 publications acknowledging medical writing in 2008, 1475 in 2016, and 4265 in 2024. Topics also differed according to data sources. Four of the top 10 publishing journals of the MECC literature focused on health economics and outcomes research (<i>Value in Health</i>, <i>Journal of Medical Economics</i>, <i>Current Medical Research and Opinion</i>, <i>Pharmacoeconomics</i>). These 4 journals published 348 documents written by MECCs, which represents 7.6% of the data collected from MECCs websites. In this study, journals that published the MECC-written literature covered a range of topics. Among the journals assessed, a rheumatology journal published 605 MECC-written publications from 2008 to 2024; of these, 524 (86.6%) were meeting abstracts, which represented 43.3% of the 1211 meeting abstracts in the database. Limitations of this study include the difficulty in accessing sources for collecting this literature and that medical writing is not always acknowledged in published articles.</p>
<h4>Conclusions </h4>
<p>This study highlights the need to further explore use and disclosure of medical writing in the health economics and outcomes research literature, as it targets decision-makers such as payers and policymakers. The database used in this study offers different routes to study medical writing in more detail.</p>
<h4>References</h4>
<p>1. Sismondo S. <i>Ghost-Managed Medicine: Big Pharma’s Invisible Hands</i>. Mattering Press; 2018.</p>
<p>2. Flanagin A, Carey LA, Fontanarosa PB, et al. Prevalence of articles with honorary authors and ghost authors in peer-reviewed medical journals. <i>JAMA</i>. 1998;280(3):222-224. doi:10.1001/jama.280.3.222</p>
<p>3. Moffatt B, Elliott C. Ghost marketing: pharmaceutical companies and ghostwritten journal articles. <i>Perspect Biology Med</i>. 2007;50(1):18-31. doi:10.1353/pbm.2007.0009</p>
<sup>1</sup>LISIS, CNRS, INRAE, Université Gustave Eiffel, France, maud.bernisson@cnrs.fr; <sup>2</sup>Institute for Science in Society, Radboud University, the Netherlands.</p>
<h4>Conflict of Interest Disclosures</h4>
<p> None reported.</p>
<h4>Funding/Support</h4>
<p> The author was part of the NanoBubbles project, which has received Synergy grant funding from the European Research Council, within the European Union’s H2020 programme (grant agreement 951393).</p>
<h4>Additional Information</h4>
<p> The database for this study, including the Web of Science identifiers, DOIs (when they exist), and sources (MECCs websites, Industry Documents Library, and Web of Science) of 31,642 documents involving medical writing, collected on January 9-13, 2025, is available at <a href="https://zenodo.org/records/15537678"><span class="Hyperlink CharOverride-6">https://zenodo.org/records/15537678</span></a>.</p>


Next from The 10th Peer Review Congress

Evaluating Approaches for Identifying Retracted Articles and Retraction Notices in Systematic Review Searching

Similar lecture

Evaluating Approaches for Identifying Retracted Articles and Retraction Notices in Systematic Review Searching