United States

&lt;p&gt;Rachel Claus&lt;sup&gt;1&lt;/sup&gt;&lt;/p&gt;
&lt;h4&gt;Objective&lt;/h4&gt;
&lt;p&gt; Research quality criteria guide grant applications and reviewer evaluations. When research crosses disciplinary boundaries, it requires more expansive quality criteria. Research evaluation is challenging in this context because concepts of research quality are rooted in disciplinary tradition.&lt;sup&gt;1&lt;/sup&gt; Reviews are characterized by inconsistency and low agreement&lt;sup&gt;2&lt;/sup&gt; and exacerbated by poorly defined criteria.&lt;sup&gt;3&lt;/sup&gt; This presentation focuses on how consistently quality criteria are applied and interpreted by reviewers and explores reasons for inconsistency. &lt;/p&gt;
&lt;h4&gt;Design &lt;/h4&gt;
&lt;p&gt;All scoring data were collected from 2 noncompetitive review processes of multimillion-dollar research program proposals with aims to improve food security. In 2021, 32 research proposals were evaluated using 17 criteria on a standard 4-point Likert scale (0-3), with limited reviewer overlap. In 2024, 9 proposals were evaluated using 12 criteria, and 4 using 11 criteria, with no reviewer overlap. Each proposal was assessed by a panel of 3 reviewers who scored independently before discussing scores to reach consensus. In total, 45 proposals were evaluated by 80 individual subject matter experts across 45 panels. Score consistency was measured by discrepancy per criterion as the difference between the highest and lowest score awarded in a panel, and how many reviewers agreed on their individual assessments. The standard of consistency was met when at least 2 of 3 reviewers agreed, and the discrepancy between individual reviewer scores was less than or equal to 1 Likert scale point. A total of 696 panel-level measures of consistency were computed. Cross tabulations were used to identify frequencies of inconsistency by criterion. Interviews with reviewers were conducted to understand perceived reasons for individual review discrepancies and disagreement, and individual and panel-level score justifications were analyzed to explore criteria interpretations.&lt;/p&gt;
&lt;h4&gt;Results&lt;/h4&gt;
&lt;p&gt; There was a statistically significant relationship between consistency and the evaluation criteria (χ² = 61.013; &lt;i&gt;P&lt;/i&gt; &amp;lt; .001). Some criteria were more consistently applied than others. The frequency that each criterion met the standard of consistency is presented in &lt;b&gt;Table 25-1061&lt;/b&gt;. Reviewers more frequently applied the following criteria inconsistently when evaluating proposals: comparative advantage (26.7%), monitoring, evaluation, and learning (24.4%), and overall theory of change (24.4%). Justified and transparent costing had the highest rate (71.9%) of inconsistency in 2021 but was not evaluated in the 2024 review cycle. Individual score discrepancies and disagreement were perceived by reviewers to result from diverse disciplinary expertise and a constructive way to achieve comprehensive quality assessments. More problematic reasons for discrepancies included misalignment of criteria to the application and different interpretations resulting from individual values and perceived abilities to make judgments.&lt;/p&gt;

![](https://assets.underline.io/markdown_image/1/image/55ce83d15f0173abd5d1a75ab77f6b39.png)

&lt;h4&gt;Conclusions&lt;/h4&gt;
&lt;p&gt; The preliminary results indicate scope for criteria clarification to improve consistency in interpretation and reliability of individual reviewer’s quality assessments of proposals. The approach can be adapted to test and inform improvements to quality criteria.&lt;/p&gt;
&lt;h4&gt;References&lt;/h4&gt;
&lt;p&gt;1. Defila R, Di Giulio A. Transdisciplinary development of quality criteria for transdisciplinary research. In: Regeer BJ, Klaassen P, Broerse JEW, eds. &lt;i&gt;Transdisciplinarity for Transformation&lt;/i&gt;. Palgrave Macmillan, Cham; 2024. https://doi.org/10.1007/978-3-031-60974-9_5&lt;/p&gt;
&lt;p&gt;2. Pier EL, Brauer M, Filut A, et al. Low agreement among reviewers evaluating the same NIH grant applications. &lt;i&gt;Proc Natl Acad Sci U S A&lt;/i&gt;. 2018;115(12):2952-2957. doi:10.1073/pnas.1714379115&lt;/p&gt;
&lt;p&gt;3. Abdoul H, Perrey C, Amiel P, et al. Peer review of grant applications: criteria used and qualitative study of reviewer practices. &lt;i&gt;PLoS One&lt;/i&gt;. 2012;7(9):e46054. doi:10.1371/journal.pone.0046054&lt;/p&gt;
&lt;sup&gt;1&lt;/sup&gt;Royal Roads University, Victoria, BC, Canada, rachel.claus@royalroads.ca.&lt;/p&gt;
&lt;h4&gt;Conflict of Interest Disclosures&lt;/h4&gt;
&lt;p&gt; None reported.&lt;/p&gt;
&lt;h4&gt;Funding/Support&lt;/h4&gt;
&lt;p&gt; This research was supported by the Sustainability Research Effectiveness Program, the BC Graduate Scholarship, the Royal Roads Doctoral Scholarship, and the David Harris Flaherty Scholarship.&lt;/p&gt;
&lt;h4&gt;Role of the Funder/Sponsor&lt;/h4&gt;
&lt;p&gt; The research was done as part of the Sustainability Research Effectiveness Program, with input provided to the design of the study, review and approval of the abstract and input to the decision to submit the abstract for presentation. Scholarships provided general support without any role in design and conduct of the study; collection, management and interpretation of the data; preparation, review or approval of the abstract, and decision to submit the abstract for presentation.&lt;/p&gt;


The 10th Peer Review Congress

[V] Reviewers&#39; Interpretation and Application of Research Quality Criteria in Grant Peer Review

funding/grant peer review

peer review

quality assurance

The Tenth Peer Review Congress featured 3 days for presentations of new research into peer review and all aspects of scientific publication, bias, quality of reporting, and information access and dissemination.

This year’s meeting was hybrid, and all plenary sessions were livestreamed with opportunities to view presentations and engage with participants in person and virtually during and after the meeting. All recordings are now available on Underline for viewing.

<div style="position:relative;padding-top:max(60%,324px);width:100%;height:0;"><iframe style="position:absolute;border:none;width:100%;height:100%;left:0;top:0;" src="https://online.fliphtml5.com/ebtyf/mget/"  seamless="seamless" scrolling="no" frameborder="0" allowtransparency="true" allowfullscreen="true" ></iframe></div>


#### Why? 
Our aim is to encourage research into the quality and credibility of peer review and scientific publication, to establish the evidence base on which scientists can improve the conduct, reporting, and dissemination of scientific research.

#### Who? 
Participants include editors and publishers of scientific peer-reviewed journals, researchers, funders, bibliometric and informatics experts, information innovators, librarians, journalists, policymakers, ethicists, scientific information disseminators, and anyone interested in the progress of the scientific information enterprise and the quality of scientific evidence.

The Congress embraces a wide range of disciplines, including biomedicine, health and life sciences, applied sciences, basic sciences, physical and chemical sciences, mathematics, computer sciences, engineering, economics, and social sciences as well as new and emerging disciplines.

#### What? 
The Congress provides a forum for the presentation and discussion of new research into the quality and credibility of editorial and funding peer review, scientific and scholarly publication, research methods and reporting practices, identification and management of biases, transparency, reproducibility, access, publication and dissemination metrics, identification and management of threats to the integrity of science, and advances in information exchange.

**Follow us**

[![](https://assets.underline.io/markdown_image/1/image/662a22766b9788ed3766cb23cd156e32.png)](https://x.com/peerrevcongress)  [![](https://assets.underline.io/markdown_image/1/image/bb105afb1773a01fa169aa97ca1af49a.png)](https://www.youtube.com/channel/UCsG_0nKi3_Zfyv0yRPdxjWQ)  [![](https://assets.underline.io/markdown_image/1/image/034e1e70a194a9a4178fdf470ef796db.png)](https://bsky.app/profile/peerreviewcongress.bsky.social)



You need to log in with the email address you registered with. 



Please log in to explore this event.

Registration is required to attend this event, please [here](https://peerreviewcongress.org/registration-information/) to secure your ticket. Please check your spam/promotional inbox too.



Please register!

The Tenth Peer Review Congress features 3 days for presentations of new research into peer review and all aspects of scientific publication, bias, quality of reporting, and information access and dissemination.

<p>Rachel Claus<sup>1</sup></p>
<h4>Objective</h4>
<p> Research quality criteria guide grant applications and reviewer evaluations. When research crosses disciplinary boundaries, it requires more expansive quality criteria. Research evaluation is challenging in this context because concepts of research quality are rooted in disciplinary tradition.<sup>1</sup> Reviews are characterized by inconsistency and low agreement<sup>2</sup> and exacerbated by poorly defined criteria.<sup>3</sup> This presentation focuses on how consistently quality criteria are applied and interpreted by reviewers and explores reasons for inconsistency. </p>
<h4>Design </h4>
<p>All scoring data were collected from 2 noncompetitive review processes of multimillion-dollar research program proposals with aims to improve food security. In 2021, 32 research proposals were evaluated using 17 criteria on a standard 4-point Likert scale (0-3), with limited reviewer overlap. In 2024, 9 proposals were evaluated using 12 criteria, and 4 using 11 criteria, with no reviewer overlap. Each proposal was assessed by a panel of 3 reviewers who scored independently before discussing scores to reach consensus. In total, 45 proposals were evaluated by 80 individual subject matter experts across 45 panels. Score consistency was measured by discrepancy per criterion as the difference between the highest and lowest score awarded in a panel, and how many reviewers agreed on their individual assessments. The standard of consistency was met when at least 2 of 3 reviewers agreed, and the discrepancy between individual reviewer scores was less than or equal to 1 Likert scale point. A total of 696 panel-level measures of consistency were computed. Cross tabulations were used to identify frequencies of inconsistency by criterion. Interviews with reviewers were conducted to understand perceived reasons for individual review discrepancies and disagreement, and individual and panel-level score justifications were analyzed to explore criteria interpretations.</p>
<h4>Results</h4>
<p> There was a statistically significant relationship between consistency and the evaluation criteria (χ² = 61.013; <i>P</i> &lt; .001). Some criteria were more consistently applied than others. The frequency that each criterion met the standard of consistency is presented in <b>Table 25-1061</b>. Reviewers more frequently applied the following criteria inconsistently when evaluating proposals: comparative advantage (26.7%), monitoring, evaluation, and learning (24.4%), and overall theory of change (24.4%). Justified and transparent costing had the highest rate (71.9%) of inconsistency in 2021 but was not evaluated in the 2024 review cycle. Individual score discrepancies and disagreement were perceived by reviewers to result from diverse disciplinary expertise and a constructive way to achieve comprehensive quality assessments. More problematic reasons for discrepancies included misalignment of criteria to the application and different interpretations resulting from individual values and perceived abilities to make judgments.</p>

![](https://assets.underline.io/markdown_image/1/image/55ce83d15f0173abd5d1a75ab77f6b39.png)

<h4>Conclusions</h4>
<p> The preliminary results indicate scope for criteria clarification to improve consistency in interpretation and reliability of individual reviewer’s quality assessments of proposals. The approach can be adapted to test and inform improvements to quality criteria.</p>
<h4>References</h4>
<p>1. Defila R, Di Giulio A. Transdisciplinary development of quality criteria for transdisciplinary research. In: Regeer BJ, Klaassen P, Broerse JEW, eds. <i>Transdisciplinarity for Transformation</i>. Palgrave Macmillan, Cham; 2024. https://doi.org/10.1007/978-3-031-60974-9_5</p>
<p>2. Pier EL, Brauer M, Filut A, et al. Low agreement among reviewers evaluating the same NIH grant applications. <i>Proc Natl Acad Sci U S A</i>. 2018;115(12):2952-2957. doi:10.1073/pnas.1714379115</p>
<p>3. Abdoul H, Perrey C, Amiel P, et al. Peer review of grant applications: criteria used and qualitative study of reviewer practices. <i>PLoS One</i>. 2012;7(9):e46054. doi:10.1371/journal.pone.0046054</p>
<sup>1</sup>Royal Roads University, Victoria, BC, Canada, rachel.claus@royalroads.ca.</p>
<h4>Conflict of Interest Disclosures</h4>
<p> None reported.</p>
<h4>Funding/Support</h4>
<p> This research was supported by the Sustainability Research Effectiveness Program, the BC Graduate Scholarship, the Royal Roads Doctoral Scholarship, and the David Harris Flaherty Scholarship.</p>
<h4>Role of the Funder/Sponsor</h4>
<p> The research was done as part of the Sustainability Research Effectiveness Program, with input provided to the design of the study, review and approval of the abstract and input to the decision to submit the abstract for presentation. Scholarships provided general support without any role in design and conduct of the study; collection, management and interpretation of the data; preparation, review or approval of the abstract, and decision to submit the abstract for presentation.</p>


[V] Reviewers' Interpretation and Application of Research Quality Criteria in Grant Peer Review

poster

<p>Joanie Sims Gould,<sup>1</sup> Anne Lasinsky,<sup>2</sup> Adrian Mota,<sup>3</sup> Karim M. Khan,<sup>1,2,4</sup> Clare L. Ardern<sup>5,6</sup></p>
<h4>Objective</h4>
<p> There is robust debate about the perceived strengths and weakness of grant peer review. Much of the research on issues in grant peer review is based on quantitative analysis of funding or scoring outcomes, which does not illuminate the experiences of peer review committee members. The objective of this study was to explore and understand the experiences and challenges faced by Canadian health research grant peer reviewers.</p>
<h4>Design</h4>
<p> This qualitative study received ethics approval from the University Behavioural Research Ethics Board. Study conduct and reporting followed the Consolidated Criteria for Reporting Qualitative Research (COREQ) guideline. Chairs, peer reviewers, and Scientific Officers of the Canadian Institutes of Health Research (CIHR) project grant competition peer review panels were interviewed. CIHR staff prepared a list of 50 potential participants who represented the 4 primary branches of CIHR research (biomedical, clinical, health systems and services, population health) from a public website. The inclusion criterion was having participated in a CIHR Project Grant competition peer review panel at least once as a peer reviewer, chair, or scientific officer. The researchers randomly selected names from the list and sent a recruitment email inviting participants to an online semistructured interview. The response rate was 36%. Two experienced qualitative researchers recruited and interviewed participants on a rolling basis from February to August 2022. The study team met biweekly to review the interview transcripts. All participants provided verbal consent for audio recording and reporting of quotes at the beginning of the interview. The interview guide was developed based on a priori concepts of peer review and the study team’s research experience in the field of grant peer review. There were questions about participants’ background, training in peer review, strengths and challenges of the review process, and managing conflict and bias. The analysis used a framework analysis approach.<sup>1</sup> Data were sifted, charted, and sorted based on key issues and themes to identify a thematic framework, compare and contrast themes, and explore similarities and differences.</p>
<h4>Results</h4>
<p> Eighteen participants were interviewed, all of whom were mid- or senior-career researchers (age 42-77 years); 11 participants (61.1%) were women and 7 (38.9%) were men. Twelve participants (66.7%) identified as White, 3 (16.7%) as South Asian, and 3 (16.7%) as other race or ethnicity. Participants identified 3 threats to grant peer review: (1) lack of training and limited opportunities to learn, (2) challenges in differentiating and rating applications of similar strength, and (3) relying on academic reputations and personal relationships in the review process to differentiate grant applications of a similar rating.</p>
<h4>Conclusions</h4>
<p> Experienced grant peer reviewers in the Canadian health funding context identified an absence of training and learning opportunities for peer review, difficulty differentiating between applications of similar strength, and an emphasis on academic reputations and personal relationships in rating applications for the Project Grant Competition.</p>
<h4>Reference</h4>
<p>1. Srivastava A, Thomson SB. Framework analysis: a qualitative methodology for applied policy research. <i>Journal of Administration and Governance</i>. 2009;72. Accessed May 31, 2025. <a href="https://ssrn.com/abstract=2760705"><span class="Hyperlink CharOverride-6">https://ssrn.com/abstract=2760705</span></a></p>
<sup>1</sup>Department of Family Practice, The University of British Columbia, Vancouver, British Columbia, Canada; <sup>2</sup>School of Kinesiology, The University of British Columbia, Vancouver, British Columbia, Canada; <sup>3</sup>Canadian Institutes of Health, Ottawa, Ontario, Canada; <sup>4</sup>Canadian Institutes of Health–Institute of Musculoskeletal Health and Arthritis, Vancouver, British Columbia, Canada; <sup>5</sup>Department of Physical Therapy, The University of British Columbia, Vancouver, British Columbia, Canada, clare.ardern@ubc.ca; <sup>6</sup>Sport and Exercise Medicine Research Centre, La Trobe University, Melbourne, Victoria, Australia.</p>
<h4>Conflict of Interest Disclosures</h4>
<p> Adrian Mota is acting vice president, Research–Programs for the Canadian Institute of Health Research (CIHR). Karim M. Khan is scientific director for the CIHR Institute of Musculoskeletal Health and Arthritis (2017-2025). No other disclosures were reported.</p>
<h4>Funding/Support </h4>
<p>This work was supported by a CIHR research operating grant (scientific directors) held by Karim M. Khan. </p>
<h4>Role of the Funder/Sponsor</h4>
<p> The CIHR had no role in design and conduct of the study. The CIHR did not participate in interpretation of the data or the preparation of the abstract and did not participate in the decision to submit the abstract for presentation.</p>


[V] Experiences and Challenges Faced by Canadian Health Research Grant Peer Reviewers

<p>Larisa Melikhova,<sup>1</sup> Andrei Rostovtsev,<sup>2</sup> Vasiliy Vlassov<sup>3</sup></p>
<h4>Objective</h4>
<p> Dissernet is a volunteer network combating plagiarism since 2013. Since 2020, we expanded our assessment to translation plagiarism (ie, using another author’s text translated into a different language) from and into Russian and to purchasing of authorship. This report covers progress since 2016.</p>
<h4>Design</h4>
<p> In 2020, by tracing authors known for earlier research misconduct, we located potential international predatory journals and checked their publications for plagiarism. All known predatory journals and potential predatory journals were screened for compliance with 2 indicators: explosive growth of publications by Russian authors and the fake collaboration index. We selected predatory journals that were indexed in Scopus and Web of Science (WoS) and examined whether they contained plagiarism translated from Russian-language sources. These data were assessed and presented in a report by the Commission for Counteracting the Falsification of Scientific Research, which was appointed by the Russian Academy of Sciences.<sup>1</sup> These journals have been expelled from Scopus and WoS. This work was continued by studying translation plagiarism from and into Ukrainian language (in collaboration with Ukrainian scientists). In 2021 to 2022, Dissernet detected another mass fraud: purchased coauthorship.<sup>2</sup> We downloaded a database of the offers at the International Publisher website and identified publications in journals indexed in WoS and/or Scopus with abstracts that coincided with offers in the database.</p>
<h4>Results</h4>
<p> In the initial search, we selected 94 predatory journals, 83 of which were indexed in Scopus and 18 in WoS, and found in them 259 publications containing plagiarism translated from Russian-language sources. In examining purchased coauthorship, we identified 418 publications in journals indexed in Scopus and/or WoS that coincided with offers from International Publisher, at least 33 of which were later retracted by the journals. By May 2025, 5962 journal articles containing evidence of various types of research misconduct had been identified on the Dissernet website.<sup>3</sup> <br />A total of 837 authors had published articles with translation plagiarism; the most common disciplines included economics, law, and education. We detected 842 authors who bought authorship through International Publisher<sup>3</sup>; the most common disciplines included medical sciences, technical sciences, economics, and law. Authors of publications with different types of violations identified by Dissernet (N = 2880) were predominantly from Russia: 2418 people (84%). Among the remaining 462 authors, 184 (40%) were from Eastern countries (Iran, Iraq, India, China, etc) and 157 (34%) were from countries of post-Soviet space (Kazakhstan, Tajikistan, Ukraine, etc). All results of the work described herein are presented on the Dissernet site (in Russian).<sup>3</sup> The publications with academic misconduct can be seen in the sections “Journals” (referring to a particular journal) and “Persons” (referring to authors of the articles).</p>
<h4>Conclusions</h4>
<p> The practice of translation plagiarism is much easier today due to the use of artificial intelligence. Purchasing coauthorship became popular at the international level. We consider these findings as a further step in exploration of translation plagiarism in the countries of the post-Soviet space and beyond.</p>
<h4>References</h4>
<p>1. Chawla DS. Top officials at Russian universities embroiled in plagiarism scandal. August 14, 2020. Nature Index. Accessed July 16, 2025. <a href="https://www.nature.com/nature-index/news/top-officials-rectors-russian-universities-embroiled-plagiarism-scandal"><span class="Hyperlink CharOverride-6">https://www.nature.com/nature-index/news/top-officials-rectors-russian-universities-embroiled-plagiarism-scandal</span></a></p>
<p>2. Rostovtsev A, Melikhova L. A disaster that has become normal. <i>TRV Science</i>. 2023;370:1, 3. Accessed July 16, 2025. <a href="https://www.trv-science.ru/2023/01/katastrofa-kotoraya-stala-normoj/"><span class="Hyperlink CharOverride-6">https://www.trv-science.ru/2023/01/katastrofa-kotoraya-stala-normoj/</span></a></p>
<p>3. <span lang="ru-RU">Dissernet. Accessed </span>July 16, 2025. <a href="https://dissernet.org/"><span class="Hyperlink CharOverride-6" lang="ru-RU">https://dissernet.org/</span></a></p>
<sup>1</sup>Dissernet, Netanya, Israel, larisamelikhova@gmail.com; <sup>2</sup>Dissernet, Budva, Montenegro; <sup>3</sup>International Health Equity Agency, Tel Aviv, Israel.</p>
<h4>Conflict of Interest Disclosures</h4>
<p> None reported.</p>


[V] Plagiarism and publication frauds revealed by Dissernet

<p>Hans Lund,<sup>1</sup> Karen Lie,<sup>2</sup> Karen Robinson,<sup>3</sup> Jong-Wook Ban,<sup>4</sup> Birgitte Nørgaard<sup>5</sup></p>
<h4>Objective </h4>
<p>Researchers’ citation practices are influenced by a wide range of factors, many of which are unrelated to scientific quality or relevance, raising concerns about citation bias. Given the conceptual ambiguity, methodological heterogeneity, and ethical implications surrounding this topic, a scoping review was conducted to map the literature, classify study intentions, and identify research gaps. The objective was to systematically map the predictors of citation rates and categorize the types of studies evaluating these factors.</p>
<h4>Design </h4>
<p>This was a scoping review to identify all studies assessing predictors of citation rates within the health sciences, with a particular focus on citation bias—defined as the selective citation of literature based on specific characteristics rather than scientific merit.</p>
<h4>Results </h4>
<p>A total of 165 studies published between 1982 and 2023 were included. Fifty-four distinct factors were identified, encompassing 4,471,352 original studies, and were grouped into 4 categories: author-related (n = 17), study-related (n = 19), reporting-related (n = 10), and journal-related (n = 8). Among the 786 analyses evaluating these factors, 429 (54.6%) identified a significant increase in citation rates, 49 (6.2%) found a decrease, and 251 (31.9%) reported no effect. Most factors associated with increased citation rates were unrelated to the scientific content of the studies and instead reflected extrinsic characteristics. As the intention, interpretation of results, and conclusions varied considerably across the included studies, we categorized them based on how they analyzed factors associated with citation rates (<b>Figure 25-1083</b>). Of the 165 studies, 77 (46.7%) were classified under category A, which examined citation bias. In contrast, 81 studies (49.1%) (categories B1 and B2) recommended leveraging specific characteristics to enhance authors’ own citation rates. Category B was further subdivided due to ambiguity in distinguishing ethically acceptable recommendations from potentially conflicting (ie, unacceptable) ones. Seven studies (4.2%) examined factors that could improve the accessibility of published studies. <b>Figure 25-1083</b> presents these categories along a continuum ranging from questionable research practices to optimal scientific behavior.</p>

![](https://assets.underline.io/markdown_image/1/image/9c199478935d01067c36a0c7323ab8ed.png)

<h4>Conclusions </h4>
<p>Many factors associated with increased citation rates were unrelated to the scientific content of the studies. Furthermore, authors of nearly half of the studies explicitly recommended modifying paper characteristics to boost citations rather than prioritizing scientific contribution. Such recommendations may conflict with principles of scientific integrity, which emphasize relevance and methodological rigor over strategic citation practices. Furthermore, the results provide a robust foundation for conducting meta-analyses to quantify the extent of the problem, as they offer a comprehensive and systematically derived list of factors influencing citation rates. The high proportion of significant results suggests possible publication bias.</p>
<sup>1</sup>Section of Evidence-Based Practice, Western Norway University of Applied Sciences, Bergen, Norway, hans.lund@hvl.no; <sup>2</sup>VID Specialized University, Bergen, Norway; <sup>3</sup>Johns Hopkins Evidence-Based Practice Center, Johns Hopkins University, Baltimore, MD, US; <sup>4</sup>Section of Hospital Medicine, University of Chicago, Chicago, IL, US; <sup>5</sup>Department of Public Health, University of Southern Denmark, Odense, Denmark.</p>
<h4>Conflict of Interest Disclosures</h4><p>None reported.</p>


[V] Citation Biases and Citation-Boosting Strategies: A Scoping Review of Predictors

<p>Dante Dahabreh,<sup>1</sup> Kenny T. L. Ta,<sup>2</sup> Angela Rose Loczi-Storm,<sup>3</sup> Olivia V. Lim,<sup>4</sup> Dana Chen,<sup>5</sup> Tasneem M. Y. Issa,<sup>6</sup> Alexandria I. Kristensen-Cabrera,<sup>6</sup> Rahib K. Islam,<sup>7</sup> Robert P. Dellavalle,<sup>6,8</sup> Eamonn Maher<sup>6,8</sup></p>
<h4>Objective</h4>
<p> Open access (OA) publishing has revolutionized academic dissemination by providing free research access. However, the financial implications for authors and the impact on journal prestige, particularly in dermatology, are often unclear. This lack of clarity complicates researchers’ publication decisions. This study aimed to evaluate the cost-effectiveness of OA publication models in dermatology journals.</p>
<h4>Design</h4>
<p> We conducted a cross-sectional study of 144 dermatology journals from the Scimago database between May 13 and June 16, 2024. Of these, 106 journals met inclusion criteria (indexed in PubMed and having an OA option); 38 were excluded for not meeting inclusion criteria, having nonfunctional websites, or missing citation metrics. Journals were categorized as hybrid or full OA. We extracted 2022 metrics (CiteScore, h-index, Scimago journal rank, and 2-year Impact Factor), averaging them for a composite impact score. Article processing charges (APCs) were collected from journal websites. Cost-effectiveness was calculated as impact score divided by total OA publishing cost (cost for free OA journals was set to 1). Statistical analyses, including Wilcoxon rank sum test and Spearman correlation, were performed using R software (R Foundation for Statistical Computing).</p>
<h4>Results</h4>
<p> Median OA publishing costs were significantly higher for hybrid journals than full OA journals ($3675 vs $350, respectively; <i>P</i> &lt; .05). Hybrid journals also showed significantly higher median impact scores than full OA journals (23.95 vs 9.23; <i>P</i> &lt; .05) (<span class="CharOverride-4"><b>Figure 24-0826</b></span>). A moderate positive correlation (<i>R</i> = 0.66) was observed between OA publishing costs and impact scores. The <i>Journal of Drugs in Dermatology</i>, <i>Indian Journal of Dermatology</i>, and <i>Dermatology Online Journal</i> were the top 3 cost-effective paid OA options. The <i>Journal of Clinical and Aesthetic Dermatology</i> and the <i>Indian Journal of Dermatology, Venereology, and Leprology</i> were identified as top free OA journals.</p>

![](https://assets.underline.io/markdown_image/1/image/6d8e3e92b0051b51887e7602787ad475.png)

<h4>Conclusions</h4>
<p> Our findings indicate a trade-off in dermatology publishing: hybrid journals, despite higher APCs, generally achieved greater impact scores, suggesting a balance between cost and prestige. While our impact score used established metrics, its broader validation and comparability across other medical subspecialties require further investigation. This study’s cost-effectiveness calculations are based on listed APCs and do not account for fee-assistance programs (offered by 25 of 106 journals) or membership discounts (available for 22 of 106 journals), which can significantly alter the actual cost for individual authors. Future research, incorporating sensitivity analyses on varying APCs (including potential waivers or discounts) and exploring broader measures of influence, is essential to enhance the generalizability and impact of these findings and to better inform authors’ complex OA publication decisions.</p>
<sup>1</sup>Department of Internal Medicine, Kaiser Permanente, Oakland, CA, US; <sup>2</sup>University of Minnesota Medical School, Minneapolis, MN, US, ta000030@umn.edu; <sup>3</sup>College of Osteopathic Medicine of the Pacific Northwest, Western University of Health Sciences, Lebanon, OR, US; <sup>4</sup>Joan C. Edwards School of Medicine, Marshall University, Huntington, WV, US; <sup>5</sup>School of Osteopathic Medicine in Arizona, A.T. Still University, Mesa, AZ, US; <sup>6</sup>University of Minnesota Medical School, Minneapolis, MN, US; <sup>7</sup>Louisiana State University Health Sciences Center, New Orleans, LA, US; <sup>8</sup>Department of Dermatology, University of Minnesota, Minneapolis, MN, US.</p>
<h4>Conflict of Interest Disclosures</h4>
<p> None reported.</p>
<h4>Additional Information</h4>
<p> ChatGPT Omega 3 was used on June 18, 2024, to help with statistical analysis and figure generation.</p>


[V] Assessing the Cost Effectiveness of Publishing Open Access Dermatology Journals

<p>Mengyuan Fu,<sup>1,2</sup> Kexin Ling,<sup>3</sup> Xinyi Zhou,<sup>3</sup> Sneha Dave,<sup>4</sup> Can Li,<sup>3</sup> Luwen Shi,<sup>2,3</sup> Xiaodong Guan,<sup>2,3</sup> Joseph S. Ross<sup>1,5,6</sup></p>
<h4>Objective </h4>
<p>This study aims to examine whether the information sources cited within National Organization for Rare Disorders (NORD) rare disease reports were publicly available, as well as any associated costs to purchase access, given that patients, caregivers, and clinicians may rely on these sources for critical information to inform their clinical decision-making.</p>
<h4>Design </h4>
<p>We randomly selected 20% of all NORD rare disease reports available as of February 2024.<sup>1</sup> Citations within these reports were classified into 5 source types, including publications in scholarly journals (research articles, case reports, reviews), textbooks, and other internet resources. Each citation was searched using Google and categorized as open access (freely available online), accessible at cost, or unavailable online. For citations with associated costs to purchase access, we recorded the lowest listed price before taxes. A 10% cross-sample validation was conducted between each pair of researchers to check consistency. All data were collected between April and May, 2024. We used descriptive statistics to calculate the proportion of citations categorized by availability, overall and by year and source type, as well as median costs, overall and by source type. We used χ² tests to evaluate differences in open access rates by year and source type. A 2-sided <i>P</i> &lt; .05 was considered statistically significant. Analyses were performed using Stata MP version 16.0.</p>
<h4>Results</h4>
<p> Among 1333 NORD rare disease reports available as of February 2024, 267 were randomly selected for study. These 267 reports included 4445 citations to 3506 unique information sources; the median (IQR) number of citations per report was 15 (11-20), including 9 (5-13) research articles, 0 (0-1) case reports, 0 (0-0) review articles, 1 (0-3) textbook, and 2 (1-4) other internet resources. Among 3506 unique information sources cited, 1750 (49.9%) were open access, 1381 (39.4%) were accessible at cost, and 375 (10.7%) were unavailable online (<b>Table 25-0964</b>). Among citations accessible only at cost, the median (IQR) cost was $31.5 ($15.0-$40.0), whereas the total cost per report for all citations accessible only at cost was $167.7 ($82.0-$258.7). The overall proportion of citations published in scholarly journals available as open access increased from 30.0% (258/860) prior to 2001 to 79.8% (95/119) from 2021 onwards (<i>P</i> &lt; .001).</p>

![](https://assets.underline.io/markdown_image/1/image/9f667019d212a0e80f968da918f39cdb.png)

<h4>Conclusions </h4>
<p>While NORD should be applauded for making its rare disease reports publicly available, we found that half of the information sources cited by these reports were not open access, potentially limiting equitable access to critical information for rare disease communities.</p>
<h4>Reference</h4>
<p>1. National Organization for Rare Disorders. Rare disease database. Accessed March 1, 2024. <a href="https://rarediseases.org/rare-diseases/"><span class="Hyperlink CharOverride-6">https://rarediseases.org/rare-diseases/</span></a></p>
<sup>1</sup>Department of Internal Medicine, Yale School of Medicine, New Haven, CT, US, joseph.ross@yale.edu; <sup>2</sup>International Research Center for Medicinal Administration, Peking University, Beijing, China; <sup>3</sup>Department of Pharmacy Administration and Clinical Pharmacy, School of Pharmaceutical Sciences, Peking University, Beijing, China; <sup>4</sup>Generation Patient, Indianapolis, IN, US; <sup>5</sup>Center for Outcomes Research and Evaluation, Yale–New Haven Health System, New Haven, CT, US; <sup>6</sup>Department of Health Policy and Management, Yale School of Public Health, New Haven, CT, US.</p>
<h4>Conflict of Interest Disclosures </h4>
<p>Mengyuan Fu currently receives research support from the National Natural Science Foundation of China (72304011). Sneha Dave is the founder and executive director of Generation Patient, an organization that receives grant funding from Arnold Ventures, Commonwealth Fund, Disability Inclusion Fund, Third Wave Fund, Responsible Technology Youth Power Fund, the Helmsley Charitable Trust, Patient-Centered Outcomes Research Institute, and Lucile Packard Foundation for Children’s Health. Joseph S. Ross currently receives research support through Yale University from Johnson and Johnson to develop methods of clinical trial data sharing, from the Food and Drug Administration for the Yale-Mayo Clinic Center for Excellence in Regulatory Science and Innovation (CERSI) program (U01FD005938), from the Agency for Healthcare Research and Quality (R01HS022882), and from Arnold Ventures; formerly received research support from the Medical Device Innovation Consortium as part of the National Evaluation System for Health Technology (NEST) and from the National Heart, Lung and Blood Institute of the National Institutes of Health (NIH) (R01HS025164, R01HL144644); was an expert witness at the request of Relator’s attorneys, the Greene Law Firm, in a qui tam suit alleging violations of the False Claims Act and Anti-Kickback Statute against Biogen Inc. that was settled in September 2022; and is a deputy editor for JAMA. No other disclosures were reported.</p>


[V] Public Access to Information Cited in Rare Disease Reports

<p>Lesley Uttley,<sup>1</sup> Louise Falzon<sup>1</sup></p>
<h4>Objective </h4>
<p>This study aimed to identify current challenges in peer review relating to research culture using a targeted thematic analysis of research culture in health and biomedical research from a wider scoping review.<sup>1</sup></p>
<h4>Design </h4>
<p>To understand challenges of peer review influenced by research culture, the literature searches of a wider scoping review<sup>1</sup> were subject to a focused qualitative thematic analysis relating specifically to peer review, following PRISMA-ScR (scoping review) guidelines<sup>2</sup> and adhering to Nowell et al’s<sup>3</sup> trustworthiness criteria for thematic analysis. PubMed and Web of Science were searched for empirical studies published between January 2012 and April 2024 in health and biomedical research culture. Eligible articles for this study were those that reported relevance to peer review (regarding manuscript publication or funding applications) in the title or abstract. Editorials, discussion pieces without data, and articles relating to peer review for conferences were excluded. The main outcome was the qualitative themes from the included articles that were associated with peer review. Relevant papers were subjected to inductive content analyses by 2 authors (L.U. and L.F.), which involved noting key findings to identify recurring themes relating to the research culture topic or finding discussed. The themes were then deductively categorized according to a framework for research integrity across 4 domains: inclusivity, transparency, rigor, and objectivity.<sup>1</sup></p>
<h4>Results </h4>
<p>Fifty-three papers were included from a total of 1680 citations generated by literature searches. Relevant themes (n = 20) regarding the challenges of peer review, ordered by prevalence, were reviewed for thematic content across the 4 domains (<span class="CharOverride-4"><b>Figure 25-0850</b></span>). Inclusivity included gender biases (n = 8), status bias (n = 2), and diversity of peer review panels (n = 5); transparency included clarity of journal expectations (n = 11), open peer review advocacy (n = 4), and blinding of author characteristics (n = 6); rigor included predatory journals (n = 3), prevention of spin (n = 4), lack of statistical examination (n = 2), potential benefits of artificial intelligence (n = 3), need for training in peer review (n = 3), and potential harms from artificial intelligence (n = 1); and objectivity included willingness/refusal/lack of incentive (n = 4), burden (n = 5), influence of conflicts of interest of reviewers (n = 3), unprofessional comments (n = 1), self-citation bias (n = 2), novelty or positivism bias (n = 2), confirmation bias against innovative research (n = 1), and homophily (preference for shared characteristics of authors) or cronyism (n = 2).</p>

![](https://assets.underline.io/markdown_image/1/image/f39da803064b7591577d5d22f43931f3.png)

<h4>Conclusions </h4>
<p>A variety of research culture influences underpin peer review challenges in the traditional academic publishing model. Themes pertain to 4 cornerstones of research integrity<sup>1</sup> relating to inclusive workforces, transparency of processes, use of best practice methods, and the objectivity (and incentives) of peer review. Emerging threats from technological innovations in conjunction with incentives fuelled by research culture are likely shaping the research landscape. Sustained and joined-up conversations between authors, publishers, and reviewers about potential peer review innovations are required to safeguard genuine research and aid the dissemination of findings that promise authentic advancements in scientific research.</p>
<h4>References</h4>
<p>1. Uttley L, Falzon L, Byrne JA, et al. Research culture influences in health and biomedical research: rapid scoping review and content analysis. <i>J Clin Epidemiol</i>. 2024;178:111616. doi:10.1016/j.jclinepi.2024.111616</p>
<p>2. Tricco AC, Lillie E, Zarin W, et al. PRISMA extension for scoping reviews (PRISMA-ScR): checklist and explanation. <i>Ann Intern Med</i>. 2018;169(7):467-473. doi:10.7326/M18-0850</p>
<p>3. Nowell LS, Norris JM, White DE, Moules NJ. Thematic analysis: striving to meet the trustworthiness criteria. <i>Int J Qual Methods</i>. 201728;16(1). doi:10.1177/1609406917733847</p>
<sup>1</sup>School of Medicine and Population Health, University of Sheffield, UK, l.uttley@sheffield.ac.uk.</p>
<h4>Conflict of Interest Disclosures </h4>
<p>None reported.</p>
<h4>Funding/Support</h4>
<p> Lesley Uttley is supported by a grant from the UKRI Medical Research Council (MR/Z504063/1).</p>


[V] Research Culture Influences in Peer Review: A Targeted Thematic Analysis of Current Challenges in Peer Review

<p>Anne G. Dudley,<sup>1</sup> George Koch,<sup>2</sup> Kyle Rose,<sup>3</sup> Roei Golan,<sup>4</sup> Jennifer Regala,<sup>5</sup> Casey Seideman,<sup>6</sup> Amanda North,<sup>7</sup> Kevin Koo,<sup>8</sup> Kevan Sternberg,<sup>9</sup> Gina Badalato,<sup>10</sup> Benjamin Dropkin,<sup>11</sup> Nicholas Chakiryan,<sup>12</sup> Robert Siemens,<sup>13</sup> Peter Clark,<sup>14</sup> Andrew Harris<sup>11</sup> </p>
<h4>Objective</h4>
<p> Peer review is a critical aspect of academic publishing, yet the process takes significant time and energy for the reviewer and is a voluntary activity. Current surveys report high levels of urologist burnout, and recent events, including the COVID-19 pandemic, have led to a shift toward personal priorities outside of work potentially limiting reviewer pools. Within urology, editors report difficulty finding appropriate numbers of peer reviewers for submitted manuscripts. We sought to assess motivations to participate in the peer review process within a pool of recent <i>Journal of Urology</i> reviewers. </p>
<h4>Design</h4>
<p> <i>The Journal of Urology</i> partnered with members of the American Urologic Association publications team to develop and administer a web-based survey to a diverse group of reviewers from September 1 to December 31, 2023. All authors and reviewers over the preceding 3 years were invited to participate. The survey addressed various aspects including career stage, their experience as reviewers, and peer review process challenges, incentives, motivators, and feedback needs. </p>
<h4>Results</h4>
<p> Respondents (n = 275) completed an average of 9 reviews in the past 12 months and reported 16 years of experience as reviewers. Most reviewers were experienced urologists less than 11 years from training (64% [176]) with only 7% (18) currently in training (resident/fellow). Time emerged as a key variable with 86% (236) of respondents declining additional reviews due to time constraints. A total of 67% (184) of respondents reported reviewing time was worthwhile, yet only 35% (96) felt appropriately recognized for time and effort, and 55% (151) reported incentives would increase time spent on a peer review. Motivations to review included “to give back” (80% [220]), “to learn” (71% [195]), “to get involved” (61% [168]), and “to grow my career” (39% [107]). Most respondents (91% [250]) read other reviewers’ reviews to learn. When asked to select specific incentives to review more papers, American Urologic Association products such as waived meeting fees and membership were highly valued (64% [176]; 62% [170]), followed by recognition by local department leadership (43% [118]) and money (40% [109]). Only 14% (38) of respondents desired gear or swag, and only 23% (63) desired to be named in the journal alongside the manuscript.</p>
<h4>Conclusions</h4>
<p> Peer review motivations are diverse and suggest that urologists participate for professional development and an ongoing desire to learn and participate in the field as a whole. Study limitations include nonresponder bias, limited survey period, and lack of granular data on personal and professional motivators. Time remains an important constraint, but incentives may increase allocated time for academic pursuits. Professional meeting/membership fee waivers may be motivators to increase participation. Local efforts to recognize reviewers within departments may work synergistically to increase available reviewers and fulfill career development goals.</p>
<sup>1</sup>Connecticut Children’s, Hartford, CT, US, annedudleymd@gmail.com; <sup>2</sup>The Ohio State University Wexner Medical Center, Columbus, OH, US; <sup>3</sup>Ochsner Medical Center, New Orleans, LA, US; <sup>4</sup>Florida State University School of Medicine, Gainesville, FL, US; <sup>5</sup>Wolters Kluwer Health, Baltimore, MD, US; <sup>6</sup>Doernbecher Children’s Hospital at OHSU, Portland, OR, US; <sup>7</sup>The Children’s Hospital at Montefiore, Bronx, NY, US; <sup>8</sup>Mayo Clinic College of Medicine and Science, Rochester, MN, US; <sup>9</sup>Northwestern Medical Center, Chicago, IL, US; <sup>10</sup>Columbia University, New York, NY, US; <sup>11</sup>University of Kentucky, Lexington, KY, US; <sup>12</sup>H Lee Moffitt Cancer Center, Tampa, FL, US; <sup>13</sup>Queen’s University, Kingston, ON, Canada; <sup>14</sup>Levine Cancer Institute, Charlotte, NC, US.</p>
<h4>Conflict of Interest Disclosures</h4>
<p> None reported.</p>
<h4>Acknowledgment</h4>
<p> We thank Martha Keyes and the Journal of Urology publications staff for their assistance with this initiative. </p>


[V] Motivations to Participate in the Peer Review Process at the Journal of Urology

<p>Bahar Mehmani,<sup>1</sup> Silvia Dobre,<sup>1</sup> Ramadurai Petchiapppan<sup>1</sup> </p>
<h4>Objective</h4>
<p> To encourage evidence-based peer review studies, Elsevier launched the Peer Review Workbench (PRW) in 2022. Here we report on the studies that have since worked in PRW, as we believe peer review studies at large scale are needed for the progress of research on research and understanding how to improve the quality of the peer review process. </p>
<h4>Design</h4>
<p> PRW is a curated and anonymized dataset of all Elsevier journal manuscript and peer review metadata on more than 5 million authors and reviewers. In our white paper,<sup>1</sup> we describe all the attributions and results of some calculations such as variation of interrater agreement, number of peer reviewers per manuscript, peer review rounds, gender balance, and country representation in editorial, authorship, and reviewers’ composition across different disciplines and impact factor quartiles. Interested researchers are provided with a proposal protocol<sup>2</sup> to submit their research plan. All protocols go through rounds of peer review by the academic advisors of the PRW, who are listed in our page.<sup>2</sup> The accepted applications receive an onboarding training by our data scientist (R.P.) who further prepares the relevant data segment suitable for their primary and secondary research purposes. </p>
<h4>Results</h4>
<p> Since September 2022, PRW has received 12 proposals, of which 6 have been accepted. Of the 6 accepted proposals, all have been onboarded and are actively working with the data. Thirty-one researchers from 17 universities in 9 countries are involved in these research projects. Based on the self-reported data, 39% (n = 12) of these researchers are women and 61% (n = 19) are men; the options “Other” and “Prefer not to disclose” were both at 0%. We promote transparency in science and therefore publish all accepted proposals on a dedicated Peer Review Workbench Research Paper Series page on SSRN.<sup>2</sup> <b>Box 25-0963</b> shows the list of research proposals currently working with the data. So far, these proposals published on SSRN have attracted over 170 downloads and almost 800 views. </p>

![](https://assets.underline.io/markdown_image/1/image/808b0e6a6b910d2e6fd1130accd36e53.png)

<h4>Conclusions</h4>
<p> The breadth of the ongoing studies encourages us to continue adding Elsevier journal and manuscript metadata annually to PRW and organize our seasonal seminars starting from autumn 2025 to report on the progress and findings of each study. This may also help improve the visibility of PRW. </p>
<h4>References</h4>
<p>1. Petchiappan R, James K, Plume A, et al. Analysing Elsevier Journal Metadata within peer review Workbench. SSRN. Revised June 9, 2025. Accessed July 16, 2025. <a href="https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4211833"><span class="Hyperlink CharOverride-6">https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4211833</span></a></p>
<p>2. SSRN. PRW: Peer Review Workbench Research Paper Series. Accessed July 16, 2025. <a href="https://www.ssrn.com/index.cfm/en/peer-review-workbench/"><span class="Hyperlink CharOverride-6">https://www.ssrn.com/index.cfm/en/peer-review-workbench/</span></a></p>
<sup>1</sup>Elsevier, Amsterdam, the Netherlands, bahar.mehmani@elsevier.com. </p>
<h4>Conflict of Interest Disclosures</h4>
<p> All authors are employees of Elsevier, the owner of the Peer Review Workbench, the publisher of the journals whose metadata is hosted on PRW, and the owner of the submission system used for collecting the data. Elsevier is also the owner of the Scopus database that was used to enrich the journals’ metadata with their different impact factor quartiles and subject areas. Bahar Mehmani is a member of the Peer Review Congress Advisory Board but was not involved in the review or decision for this abstract. </p>


[V] Review of Proposals Submitted to Elsevier's Peer Review Workbench

<p>Sean Hays,<sup>1</sup> Tyler Carneal,<sup>1</sup> Christopher Kirman<sup>1</sup></p>
<h4>Objective</h4>
<p> A pilot study was conducted in which a community of SciPinion science experts (&gt;32,000 registered users) and the SciPinion web platform performed rapid peer review of a study.</p>
<h4>Design </h4>
<p>Review design elements were included to avoid sources of bias in key components of the peer review (reviewer participation, selection, engagement, and reporting). A review announcement, released in January 2025, included the general topic and expertise needs (eg, developmental neurotoxicity) but did not identify the specific manuscript. Within 24 hours of the announcement, 152 scientists applied and 5 reviewers were selected automatically by the web platform based on expertise metrics (years of experience, publications, counts of key words relevant to the manuscript). The review panel consisted of qualified scientists (mean [SD] years of experience, 37 [14] years; range, 21-52 years; mean [SD] publication count, 416 [124]; range, 216-550) across 4 countries. Reviewers were automatically verified via a community-driven reputation system based on past interactions with the web platform. To improve robustness of the process, the review was structured as a 3-round modified Delphi format. A $250 honorarium was offered to reviewers to complete their work under an accelerated schedule. Reviewers were masked to each other. In round 1, reviewers worked independently to review the manuscript and answer structured questions that included confidence ratings (0-10 scale, with higher number indicating higher confidence) for components of the manuscript with explanations and confidence in relying on the reported study for making policy decisions by health agencies. In round 2, reviewers worked deliberatively in reviewing and commenting on each other’s answers (13 comments submitted). In round 3, the reviewers returned to their round 1 responses, revised as needed after deliberation, and submitted final responses.</p>
<h4>Results </h4>
<p>All 3 rounds of the review were completed within 1 week of panel selection, and honoraria were initiated within 24 hours of completion. A comparison of the panel’s independent input from round 1 and their deliberative input in round 3 yielded a decrease in the confidence score of the manuscript (mean [SD] confidence score of 4.2 [2.5] after round 1 was 3.8 [2.2] after round 3) and an improvement in consensus across reviewers (based on Tastle and Wierman<sup>1</sup>), from 0.71 after round 1 to 0.77 after round 3) (<b>Figure 25-1065</b>). Confidence in using the study to support decision-making by health agencies was considered low by the review panel (mean [SD] score, 3.0 [1.2]).</p>

![](https://assets.underline.io/markdown_image/1/image/1045176dc2fe1ee30a141edb4aac3dc3.png)

<h4>Conclusions</h4>
<p> This pilot study found that rapid (ie, within 1 week) peer reviews can be performed with a process that includes multiple rounds of engagement and collaborative sharing of reviewer ratings and comments in a scalable manner via automation of multiple steps in the process, including ranking of candidate reviewers, contracting (agreement defining scope of work and compensation), scheduling of rounds, messaging, and payments. </p>
<h4>Reference</h4>
<p>1. Tastle WJ, Wierman MJ. Consensus and dissention: a measure of ordinal dispersion. <i>Int J Approx Reason</i>. 2007;45(3):531-545. doi:10.1016/j.ijar.2006.06.024</p>
<sup>1</sup>SciPinion, Bozeman, MT, ckirman@scipinion.com.</p>
<h4>Conflict of Interest Disclosures</h4>
<p> None reported.</p>
<h4>Funding/Support </h4>
<p>This work was self-funded by SciPinion.</p>


[V] Use of a 3-Round Modified Delphi Format to Support Rapid Peer Review

<p>Julián C. Velásquez Paz,<sup>1</sup> Andrés Zorrilla Vaca,<sup>2</sup> Markus Klimek,<sup>3</sup> Jose A. Calvache<sup>1,3</sup> </p>
<h4>Objective</h4>
<p> In recent years, systematic reviews (SRs) and meta-analyses (MAs)—traditionally regarded as reliable sources of causal evidence due to their methodological rigor—have increasingly been compromised by the inclusion of predatory literature, which is marked by misleading information and questionable editorial practices.<sup>1</sup> Despite ongoing efforts,<sup>2</sup> predatory literature continues to pose a major challenge in evidence synthesis. We aimed to assess the degree of inclusion of randomized controlled trials (RCTs) published in potentially predatory journals into SRs with MAs published in leading anesthesiology journals.</p>
<h4>Methods</h4>
<p> We conducted a cross-sectional study in which we included SRs with MAs published between 2021 and 2022 in the top 10 high-ranked anesthesia journals based on the Web of Science’s Journal Citation Report for 2021. Exclusion criteria included SRs without MAs or network MAs as well as those including observational studies and nonhuman studies. To determine whether an RCT was published in a potentially predatory journal, 2 authors independently followed a stepwise process: (1) an initial assessment of the publisher’s reputation, confirming if it was widely recognized and reputable in the academic publishing community; (2) verification of the journal’s indexing in the Committee on Publication Ethics or Directory of Open Access Journals; and (3) application of the Loyola Marymount University Journal Evaluation Tool (JET).<sup>3</sup> Discrepancies between the 2 authors were resolved by consensus. The JET evaluates journals based on key criteria such as transparency of editorial processes, rigor of peer review practices, adherence to ethical standards, presence in reputable indexing databases, archiving practices, and the quality of the journal’s website; each aspect is scored using defined indicators based on their credibility and risk of predatory practices (<b>Figure 25-1165</b>). We summarized our results using descriptive statistics.</p>

![](https://assets.underline.io/markdown_image/1/image/4fcb2f14a0f0a0a3896c444182448a51.png)

<h4>Results </h4>
<p>A total of 79 SRs with MAs including 1110 RCTs were analyzed, with a median of 10 RCTs per SR. Most studies originated from the US, China, and India (432 [38.9%]). After applying the stepwise approach, 29 (2.6%) of the analyzed RCTs were identified as published in journals classified by our process as untrustworthy; in addition, RCTs published in potentially predatory journals were included in 14 (17.7%) of the 79 studied SRs. Finally, after reviewing Beall’s list, <a href="https://www.predatoryjournals.org/"><span class="Hyperlink CharOverride-6">www.predatoryjournals.org</span></a>, and journals excluded from the Web of Science or Scopus since 2017, we found that only 5 (17.2%) of the RCTs published in journals classified as untrustworthy appeared on any of these lists.</p>
<h4>Conclusions</h4>
<p> This study highlights that RCTs published in potentially predatory journals are being cited in SRs with MAs published in leading anesthesiology journals. Our findings indicate that while lists of potentially predatory journals can be helpful, they are probably not sufficient to fully identify actual predatory journals. Further analysis is needed to determine the impact of RCTs published by potentially predatory journals on the main findings of these MAs.</p>
<h4>References</h4>
<p>1. Grudniewicz A, Moher D, Cobey KD, et al. Predatory journals: no definition, no defence. <i>Nature</i>. 2019;576(7786):210-212. doi:10.1038/d41586-019-03759-y</p>
<p>2. Cukier S, Helal L, Rice DB, et al. Checklists to detect potential predatory biomedical journals: a systematic review. <i>BMC Med</i>. 2020;18(1):104. doi:10.1186/s12916-020-01566-1</p>
<p>3. Rele S, Kennedy M, Blas N. Journal Evaluation Tool. Loyola Marymount University. 2017. Accessed July 10, 2024. <a href="https://digitalcommons.lmu.edu/librarian_pubs/40"><span class="Hyperlink CharOverride-6">https://digitalcommons.lmu.edu/librarian_pubs/40</span></a></p>
<sup>1</sup>Department of Anaesthesiology, Universidad del Cauca, Popayan, Colombia, jacalvache@unicauca.edu.co; <sup>2</sup>Department of Anesthesiology, Perioperative and Pain Medicine, Brigham and Women’s Hospital, Boston, MA, US; <sup>3</sup>Department of Anesthesiology, Erasmus University Medical Center Rotterdam, the Netherlands.</p>
<h4>Conflict of Interest Disclosures</h4>
<p> None reported.</p>


[V] Inclusion of Randomized Controlled Trials Published by Potentially Predatory Journals in Anesthesiology Systematic Reviews: A Cross-Sectional Study

Next from The 10th Peer Review Congress

[V] Experiences and Challenges Faced by Canadian Health Research Grant Peer Reviewers

Similar lecture

Diversity Among Reviewers Assigned to Evaluate a Paper as a Factor in Diversifying Perspective and Improving the Peer Review Process in Computer Science