United States

&lt;p&gt;Patrick Amboka,&lt;sup&gt;1&lt;/sup&gt; Tony Blair Aloo,&lt;sup&gt;1&lt;/sup&gt; Daniel Krugman,&lt;sup&gt;1,2&lt;/sup&gt; Abel Simiyu,&lt;sup&gt;1&lt;/sup&gt; Hiram Kariuki,&lt;sup&gt;1&lt;/sup&gt; Benard Ondiek,&lt;sup&gt;1&lt;/sup&gt; Nosa Orobaton,&lt;sup&gt;3&lt;/sup&gt; Emmy Igonya,&lt;sup&gt;1&lt;/sup&gt; Alphonsus Neba,&lt;sup&gt;1&lt;/sup&gt; Marta Vicente-Crespo,&lt;sup&gt;1,4&lt;/sup&gt; Julius Kirimi Sindi&lt;sup&gt;1&lt;/sup&gt;&lt;/p&gt;
&lt;h4&gt;Objective&lt;/h4&gt;
&lt;p&gt; This study explored editorial practices among African journals, examined the factors influencing these practices, and assessed authors’ perspectives and preferences in selecting journals for publication. A large proportion of African journals (63.2%) are neither discoverable via Google Scholar nor included in Scopus.&lt;sup&gt;1&lt;/sup&gt; However, their discoverability and inclusion in international platforms are nuanced and not always straightforward; hence, quality markers need to be better aligned.&lt;/p&gt;
&lt;h4&gt;Design&lt;/h4&gt;
&lt;p&gt; This study triangulated multiple sources of information and used qualitative data gathering techniques to capture nuances and provide deeper insights into the performance and visibility of African journals. Conducted between July and December 2024, the study utilized a cross-sectional design. In-depth interviews (n = 60), key informant interviews (n = 32), and focus group discussions (n = 7) were conducted in Kenya, Ethiopia, Nigeria, and Mozambique. A purposive sampling technique was used to identify participants. Ethical approvals were obtained from relevant institutions. We obtained consent from participants to record and publish the data in anonymized form. Qualitative data from the audio-recorded interviews were transcribed using Microsoft Word and exported to NVivo software for analysis. The data were analyzed using a reflexive thematic analysis.&lt;sup&gt;2&lt;/sup&gt; Emerging concepts from the qualitative data were also analyzed using Leximancer software.&lt;/p&gt;
&lt;h4&gt;Results&lt;/h4&gt;
&lt;p&gt; Four main themes emerged from participant interviews in this study. First, there are challenges in adhering to international editorial standards (“We often want to follow international best practices, but there’s limited guidance and mentorship available to help us meet those expectations consistently”). Second, financial constraints undermine editorial integrity (“Sometimes, we’re forced to accept submissions not based on quality, but because we need the processing fees to keep the journal running. It’s not ideal, but without funding, we have few options”). Third, technological and digital infrastructure gaps exist (“Our journal management system is outdated, and we lack IT support to upgrade it”). Fourth, authors have perceptions of inferiority and trust issues (“Authors often tell us they prefer to submit to Western journals because they assume ours won’t be taken seriously”).&lt;/p&gt;
&lt;h4&gt;Conclusions&lt;/h4&gt;
&lt;p&gt; Addressing financial constraints, peer review inefficiencies, and infrastructure gaps is critical to strengthening the editorial practices and credibility of African journals. Overcoming historical neglect through sustainable funding, capacity building, and technological advancement will be key to enhancing their global visibility, trust, and academic impact.&lt;/p&gt;
&lt;h4&gt;References&lt;/h4&gt;
&lt;p&gt;1. Amboka P, Sindi JK, Wamukoya M, et al. Discoverability of African journals by Google Scholar and inclusion in Scopus. VeriXiv. Preprint posted online November 1, 2024. &lt;a href=&quot;https://verixiv.org/articles/1-17&quot;&gt;&lt;span class=&quot;Hyperlink CharOverride-6&quot;&gt;https://verixiv.org/articles/1-17&lt;/span&gt;&lt;/a&gt;&lt;/p&gt;
&lt;p&gt;2. Byrne D. A worked example of Braun and Clarke’s approach to reflexive thematic analysis. Qual Quant. 2022;56:1391-1412. doi:10.1007/s11135-021-01182-y&lt;/p&gt;
&lt;sup&gt;1&lt;/sup&gt;African Population and Health Research Center, Nairobi, Nairobi County, Kenya, pamboka@aphrc.org; &lt;sup&gt;2&lt;/sup&gt;Department of Anthropology, Brown University, Providence, RI, US; &lt;sup&gt;3&lt;/sup&gt;Bill &amp;amp; Melinda Gates Foundation, Seattle, WA, US; &lt;sup&gt;4&lt;/sup&gt;School of Public Health, University of the Witwatersrand, Johannesburg, South Africa.&lt;/p&gt;
&lt;h4&gt;Conflict of Interest Disclosures&lt;/h4&gt;
&lt;p&gt; None reported.&lt;/p&gt;
&lt;h4&gt;Funding/Support&lt;/h4&gt;
&lt;p&gt; This study was funded by the Bill and Melinda Gates Foundation (grant INV-034355).&lt;/p&gt;
&lt;h4&gt;Role of the Funder/Sponsor&lt;/h4&gt;
&lt;p&gt; The funder had no role in the study design, data collection and analysis, decision to publish, or manuscript preparation.&lt;/p&gt;


The 10th Peer Review Congress

[V] Editorial Landscape of Journals in Kenya, Ethiopia, Nigeria, and Mozambique

editorial freedom and integrity

peer review process and models

editorial and peer review process

The Tenth Peer Review Congress featured 3 days for presentations of new research into peer review and all aspects of scientific publication, bias, quality of reporting, and information access and dissemination.

This year’s meeting was hybrid, and all plenary sessions were livestreamed with opportunities to view presentations and engage with participants in person and virtually during and after the meeting. All recordings are now available on Underline for viewing.

<div style="position:relative;padding-top:max(60%,324px);width:100%;height:0;"><iframe style="position:absolute;border:none;width:100%;height:100%;left:0;top:0;" src="https://online.fliphtml5.com/ebtyf/mget/"  seamless="seamless" scrolling="no" frameborder="0" allowtransparency="true" allowfullscreen="true" ></iframe></div>


#### Why? 
Our aim is to encourage research into the quality and credibility of peer review and scientific publication, to establish the evidence base on which scientists can improve the conduct, reporting, and dissemination of scientific research.

#### Who? 
Participants include editors and publishers of scientific peer-reviewed journals, researchers, funders, bibliometric and informatics experts, information innovators, librarians, journalists, policymakers, ethicists, scientific information disseminators, and anyone interested in the progress of the scientific information enterprise and the quality of scientific evidence.

The Congress embraces a wide range of disciplines, including biomedicine, health and life sciences, applied sciences, basic sciences, physical and chemical sciences, mathematics, computer sciences, engineering, economics, and social sciences as well as new and emerging disciplines.

#### What? 
The Congress provides a forum for the presentation and discussion of new research into the quality and credibility of editorial and funding peer review, scientific and scholarly publication, research methods and reporting practices, identification and management of biases, transparency, reproducibility, access, publication and dissemination metrics, identification and management of threats to the integrity of science, and advances in information exchange.

**Follow us**

[![](https://assets.underline.io/markdown_image/1/image/662a22766b9788ed3766cb23cd156e32.png)](https://x.com/peerrevcongress)  [![](https://assets.underline.io/markdown_image/1/image/bb105afb1773a01fa169aa97ca1af49a.png)](https://www.youtube.com/channel/UCsG_0nKi3_Zfyv0yRPdxjWQ)  [![](https://assets.underline.io/markdown_image/1/image/034e1e70a194a9a4178fdf470ef796db.png)](https://bsky.app/profile/peerreviewcongress.bsky.social)



You need to log in with the email address you registered with. 



Please log in to explore this event.

Registration is required to attend this event, please [here](https://peerreviewcongress.org/registration-information/) to secure your ticket. Please check your spam/promotional inbox too.



Please register!

The Tenth Peer Review Congress features 3 days for presentations of new research into peer review and all aspects of scientific publication, bias, quality of reporting, and information access and dissemination.

<p>Patrick Amboka,<sup>1</sup> Tony Blair Aloo,<sup>1</sup> Daniel Krugman,<sup>1,2</sup> Abel Simiyu,<sup>1</sup> Hiram Kariuki,<sup>1</sup> Benard Ondiek,<sup>1</sup> Nosa Orobaton,<sup>3</sup> Emmy Igonya,<sup>1</sup> Alphonsus Neba,<sup>1</sup> Marta Vicente-Crespo,<sup>1,4</sup> Julius Kirimi Sindi<sup>1</sup></p>
<h4>Objective</h4>
<p> This study explored editorial practices among African journals, examined the factors influencing these practices, and assessed authors’ perspectives and preferences in selecting journals for publication. A large proportion of African journals (63.2%) are neither discoverable via Google Scholar nor included in Scopus.<sup>1</sup> However, their discoverability and inclusion in international platforms are nuanced and not always straightforward; hence, quality markers need to be better aligned.</p>
<h4>Design</h4>
<p> This study triangulated multiple sources of information and used qualitative data gathering techniques to capture nuances and provide deeper insights into the performance and visibility of African journals. Conducted between July and December 2024, the study utilized a cross-sectional design. In-depth interviews (n = 60), key informant interviews (n = 32), and focus group discussions (n = 7) were conducted in Kenya, Ethiopia, Nigeria, and Mozambique. A purposive sampling technique was used to identify participants. Ethical approvals were obtained from relevant institutions. We obtained consent from participants to record and publish the data in anonymized form. Qualitative data from the audio-recorded interviews were transcribed using Microsoft Word and exported to NVivo software for analysis. The data were analyzed using a reflexive thematic analysis.<sup>2</sup> Emerging concepts from the qualitative data were also analyzed using Leximancer software.</p>
<h4>Results</h4>
<p> Four main themes emerged from participant interviews in this study. First, there are challenges in adhering to international editorial standards (“We often want to follow international best practices, but there’s limited guidance and mentorship available to help us meet those expectations consistently”). Second, financial constraints undermine editorial integrity (“Sometimes, we’re forced to accept submissions not based on quality, but because we need the processing fees to keep the journal running. It’s not ideal, but without funding, we have few options”). Third, technological and digital infrastructure gaps exist (“Our journal management system is outdated, and we lack IT support to upgrade it”). Fourth, authors have perceptions of inferiority and trust issues (“Authors often tell us they prefer to submit to Western journals because they assume ours won’t be taken seriously”).</p>
<h4>Conclusions</h4>
<p> Addressing financial constraints, peer review inefficiencies, and infrastructure gaps is critical to strengthening the editorial practices and credibility of African journals. Overcoming historical neglect through sustainable funding, capacity building, and technological advancement will be key to enhancing their global visibility, trust, and academic impact.</p>
<h4>References</h4>
<p>1. Amboka P, Sindi JK, Wamukoya M, et al. Discoverability of African journals by Google Scholar and inclusion in Scopus. VeriXiv. Preprint posted online November 1, 2024. <a href="https://verixiv.org/articles/1-17"><span class="Hyperlink CharOverride-6">https://verixiv.org/articles/1-17</span></a></p>
<p>2. Byrne D. A worked example of Braun and Clarke’s approach to reflexive thematic analysis. Qual Quant. 2022;56:1391-1412. doi:10.1007/s11135-021-01182-y</p>
<sup>1</sup>African Population and Health Research Center, Nairobi, Nairobi County, Kenya, pamboka@aphrc.org; <sup>2</sup>Department of Anthropology, Brown University, Providence, RI, US; <sup>3</sup>Bill &amp; Melinda Gates Foundation, Seattle, WA, US; <sup>4</sup>School of Public Health, University of the Witwatersrand, Johannesburg, South Africa.</p>
<h4>Conflict of Interest Disclosures</h4>
<p> None reported.</p>
<h4>Funding/Support</h4>
<p> This study was funded by the Bill and Melinda Gates Foundation (grant INV-034355).</p>
<h4>Role of the Funder/Sponsor</h4>
<p> The funder had no role in the study design, data collection and analysis, decision to publish, or manuscript preparation.</p>


poster

<p>James Kiilu,¹ Cecilia Munguti,¹ James Kigera,¹ Michael Mwachiro¹</p>
<h4>Objective </h4>
<p>The peer review process faces the challenges of recruiting reviewers and assessing the quality of the peer review process and manuscripts.<sup>1</sup> This study evaluated trends in reviewer performance (R-scores), manuscript quality (M-scores), and their association with reviewer recommendations and editorial decisions at the <i>Annals of African Surgery</i> over 7 years.</p>
<h4>Design </h4>
<p>This retrospective study analyzed peer review reports at the <i>Annals of African Surgery</i> from 2017 to 2023, using data extracted from the journal’s manuscript management system. Variables included manuscript ID, submission year, revision cycles, reviewer ID, recommendations, R-scores, M-scores, time to review, and editorial decisions. R-scores assessed review timeliness and relevance, while M-scores evaluated the manuscript across dimensions such as interest, quality, and originality. Descriptive statistics summarized reviewer invitation responses, and cross-tabulations were performed to examine trends in R-scores and M-scores against submission year, reviewer recommendations, revision cycles, and editorial decisions. Associations between reviewer recommendations and editorial outcomes were assessed using χ² tests. Continuous data were tested for skewness, with analysis of variance applied to normally distributed data and Kruskal-Wallis tests to skewed data.</p>
<h4>Results </h4>
<p>For the period under review, the journal had a total of 1193 reviewers, with 370 (31%) coming from the international community. The number of reviewers invited per manuscript varied significantly across years, as determined by an independent-samples Kruskal-Wallis test (H<i>2</i> = 18.46; <i>df </i>= 6; <i>P =</i> .005). The mean (SD) number of reviewers invited per manuscript was 6.35 (3.4) in 2017, peaked at 9.81 (9.6) in 2019, then declined to 5.12 (3.5) in 2021 before rising again to 7.02 (4.03) in 2023. There was a significant improvement in the reviewer acceptance rate, increasing from 29.9% in 2019 to 45.7% in 2023 (<i>P</i> &lt; .001) (<b>Table 25-0922</b>). R-scores decreased significantly over the study period, from mean (SD) 2.68 (0.53) in 2017 to 2.29 (0.65) in 2023 (<i>P</i> &lt; .001), while there was no significant change in M-scores. There was no statistically significant change in R-scores across different manuscript revision cycles, but M-scores showed significant improvement (from mean [SD] 2.60 [0.84] in the initial round of review to 2.17 [0.61] after 3 revisions, <i>F</i> = 2.413; <i>P</i> &lt; .05), accompanied by a marked reduction in review turnaround time (from mean [SD] 11.07 [9.9] to 1.29 [1.38] days; <i>P</i> = .01). R-scores and M-scores varied significantly by editorial decision, with the highest R-scores observed for manuscripts receiving a major revision decision (mean = 2.71; <i>P</i> &lt; .001) and the highest M-scores seen in rejected manuscripts (mean = 3.91; <i>P</i> &lt; .001). One-way analysis of variance demonstrated statistically significant differences in mean R-scores awarded by the 15 editors (<i>F</i><i>14, 3986</i> = 17.77; <i>P</i> &lt; .001). Reviewer recommendations were associated with editorial decisions (Pearson χ² = 288.80; <i>df </i>= 9; N = 1173; <i>P</i> &lt; .001).</p>

![](https://assets.underline.io/markdown_image/1/image/54831eb6ac1a19e13b97949096aac091.png)

<h4>Conclusions</h4>
<p> While reviewer acceptance rates improved significantly over time, there was no corresponding improvement in R-scores or M-scores. Future efforts should prioritize not only increasing reviewer participation but also strengthening the quality and consistency of reviewer feedback to more effectively support manuscript development.</p>
<h4>Reference</h4>
<p>1. Peterson CJ, Orticio C, Nugent K. The challenge of recruiting peer reviewers from one medical journal’s perspective. <i>Proc Bayl Univ Med Cent</i>. 2022;35(3):394-396. doi:10.1080/08998280.2022.2035189</p>
<sup>1</sup>Annals of African Surgery, Nairobi, Kenya; james.kiilu@annalsofafricansurgery.com.</p>
<h4>Conflict of Interest Disclosures</h4>
<p> James Kigera is a member of the Peer Review Congress Advisory Board but was not involved in the review or decision for this abstract.</p>


[V] Trends in Peer Review Metrics at the Annals of African Surgery

<p>Jorge Finke,<sup>1</sup> Sumi Sexton<sup>2</sup></p>
<h4>Objective </h4>
<p>The peer review process has been criticized for a lack of evidence that it improves the quality of scientific literature.<sup>1</sup> Some authors have reported peer reviews to be incoherent, lacking constructive feedback, and slow.<sup>2</sup> Importantly, little research has been done on peer review for narrative reviews. We have developed a checklist for peer reviewers to evaluate the effects on the quality of peer reviews and manuscripts.</p>
<h4>Methods</h4>
<p> This was a pseudorandomized nonblinded clinical trial. A 10-question checklist was developed for use by peer reviewers evaluating manuscripts for the medical journal <i>American Family Physician (AFP). </i>The checklist was created through the Delphi method, utilizing a panel of experts including <i>AFP</i> medical editors and editors from other journals who reviewed the proposed checklist. The checklist was internally validated by 2 medical editors independently comparing results of the new checklist to the journal’s existing manuscript quality scoring system. Manuscripts were assigned on alternating months to either the experimental group in which reviewers and editors applied the new checklist or the control group which utilized the traditional system. After peer review, the medical editor assigned to each manuscript rated the quality of each review, using a 0 to 100% scale to determine whether the checklist improved peer reviewer performance. Quality ratings of the initially submitted and finalized manuscripts were also determined by the medical editor using a 0 to100% scale. Changes in initial to finalized manuscript quality ratings were compared between the experimental and control groups.</p>
<h4>Results </h4>
<p>A total of 78 manuscripts were evaluated from December 2022 through March 2024. Forty of these manuscripts were assigned to peer reviewers utilizing the newly developed checklist, and 38 used the traditional review process. No statistically significant improvement was noted in review quality (68.5%; 95% CI, 63.5%-72.8% vs 65.7%; 95% CI, 61.0%-69.8%; <i>P</i> = .40), review word count length (683 words; 95% CI, 608.8-754.5 words vs 616.04 words; 95% CI, 545.8-684.1 words; <i>P</i> = .19) or speed of the peer review process (45.49 days; 95% CI, 37.2-52.8 days vs 37.3 days; 95% CI, 31.9-41.6 days; <i>P</i> = .08). Similarly, no statistically significant improvement was found in manuscript quality rating after revisions based on peer review with new checklist compared with the control group (difference, 15.7%; 95% CI, 9.1%-22.6% vs 12.6%; 95%CI, 6.7%-18.8%; <i>P</i> = .49).</p>
<h4>Conclusions</h4>
<p> Few interventions have had positive impacts in the peer-review process, including the appropriate use of checklists or guidelines.<sup>3</sup> While this study showed no significant improvements with the new checklist in the peer review quality rating, reviewer word counts, speed of reviewer process, and overall manuscript ratings, no parameters were negatively affected. The checklist was still formally adopted as a modified version of the previous format developed based on Delphi feedback from experts in the field.</p>
<h4>References</h4>
<p>1. Kelly J, Sadeghieh T, Adeli K. Peer review in scientific publications: benefits, critiques, &amp; a survival guide. <i>EJIFCC</i>. 2014;25(3):227-243.</p>
<p>2. Sciullo N, Duncan M. Professionalizing peer review suggestions for a more ethical and pedagogical review process. <i>J Schol Pub</i>. 2019;50:248-264. doi:10.3138/jsp.50.4.02</p>
<p>3. Gaudino M, Robinson NB, Di Franco A, et al. Effects of experimental interventions to improve the biomedical peer-review process: a systematic review and meta-analysis. <i>J Am Heart Assoc</i>. 2021;10(15):e019903. doi:10.1161/JAHA.120.019903</p>
<sup>1</sup>Contributing editor, <i>American Family Physician</i>, Leawood, KS, US, jfinke@bidmc.harvard.edu; <sup>2</sup>Editor in chief, <i>American Family Physician</i>, Leawood, KS, US.</p>
<h4>Conflict of Interest Disclosures</h4>
<p> None reported.</p>
<h4>Acknowledgment</h4>
<p> This study was conducted with support from American Family Physician as an internal quality improvement project and the intervention developed as adopted for use in the journal’s peer review process. This work was conducted with support from UL1TR002541 award through Harvard Catalyst (Biostatistics/Bioinformatics Consultation Program), The Harvard Clinical and Translational Science Center (National Center for Advancing Translational Sciences, National Institutes of Health), and financial contributions from Harvard University and its affiliated academic health care centers. The content is solely the responsibility of the authors and does not necessarily represent the official views of Harvard Catalyst, Harvard University and its affiliated academic health care centers, or the National Institutes of Health.</p>


[V] Impact of a Novel Checklist on the Peer Review Process

<p>Haley K. Holmer,<sup>1</sup> Edi E. Kuhn,<sup>1</sup> Camber Hansen-Karr,<sup>1</sup> Ed Reid,<sup>1</sup> Mark Helfand<sup>1</sup></p>
<h4>Objective</h4>
<p> To identify methodological concerns in Agency for Healthcare Research and Quality (AHRQ) Evidence-Based Practice Center (EPC) Program reports and assess the comprehensiveness of EPC Methods Guidance using the Systematic Reviewlution (SR) Framework.<sup>1</sup></p>
<h4>Design</h4>
<p> Cross-sectional analysis of reviewer comments from 25 EPC reports submitted between June 1, 2018, and March 1, 2023. The sample included systematic reviews (n = 16), systematic review updates (n = 4), and technical briefs (n = 5). For the internal, pre–peer-reviewed phase, 1 reviewer extracted comments received from associate editors, task order officers, and AHRQ leadership. In the external peer-reviewed phase, we extracted comments from peer reviewers, key informants, technical experts, partners, and public reviewers. A senior reviewer used the SR Framework,<sup>1</sup> a comprehensive typology of 67 systematic review problems, to classify concerns raised in the pre–peer-reviewed and/or peer-reviewed phase and to quantify frequencies. We identified patterns in 2 areas: (1) concerns raised in the pre–peer-review phase that persisted into the peer-reviewed phase and (2) concerns missed in the pre–peer-reviewed phase that were raised in the peer-reviewed phase. Further, we determined whether problems identified in the SR Framework are addressed in EPC Methods Guidance,<sup>2,3</sup> and we identified methodological concerns absent from the SR Framework.</p>
<h4>Results</h4>
<p> Of 5024 total reviewer comments, 15% (252 of 1717) of pre–peer-reviewed comments and 10% (348 of 3307) of peer-reviewed comments addressed methodological concerns. Nearly half of the methodological concerns identified during the pre–peer-review phase pertained to grading strength of evidence (SOE) (n = 116 [46%]), followed by concerns with assessing risk of bias (n = 23 [9%]), and spin (n = 21 [8%]). The most frequent concerns raised by peer reviewers also related to spin (n = 56 [16%]), grading SOE (n = 44 [13%]), and lack of clinical expert or stakeholder perspective (n = 41 [12%]). Spin-related concerns included conclusions that went beyond the evidence, policy implications disconnected from results, failure to incorporate SOE into conclusions, and biased tone in presentation. Comments raised in the peer-reviewed phase that were not raised in pre–peer-review phase included concerns about a priori protocol decisions or inflexible methods for answering review questions. The SR Framework failed to capture several key methodologic issues identified in EPC reports, including concerns about (1) SOE ratings, (2) inadequately defined outcomes, and (3) a priori protocol registration. Over 75% (n = 51) of the 67 problems in the SR Framework were addressed in EPC methods guidance.</p>
<h4>Conclusions</h4>
<p> Overall, EPC reports are of high methodological quality. Notable patterns in methodological concerns emerged, with specific issues pertaining to grading SOE, spin, need for clinical expert or stakeholder perspectives, and balancing adherence to an a priori protocol against potential bias introduced by an adaptive protocol. The persistence of spin-related concerns across both review phases suggests a challenge in maintaining objectivity in evidence synthesis. Notably, the SR Framework did not capture important methodological concerns in EPC reports, suggesting potential gaps in the SR Framework for evaluating the trustworthiness of evidence synthesis products.</p>
<h4><a id="_idTextAnchor075"></a>References</h4>
<p>1. Uttley L, Quintana DS, Montgomery P, et al. The problems with systematic reviews: a living systematic review. <i>J Clin Epidemiol</i>. 2023;156:30-41. doi:10.1016/j.jclinepi.2023.01.011</p>
<p>2. Methods Guide for Effectiveness and Comparative Effectiveness Reviews. Page last reviewed October 2022. Effective Health Care Program, Agency for Healthcare Research and Quality. <a href="https://effectivehealthcare.ahrq.gov/products/collections/cer-methods-guide"><span class="Hyperlink CharOverride-6">https://effectivehealthcare.ahrq.gov/products/collections/cer-methods-guide</span></a></p>
<p>3. Page MJ, McKenzie JE, Bossuyt PM, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. <i>BMJ</i>. 2021;372:n71. doi:10.1136/bmj.n71</p>
<sup>1</sup>Portland VA Research Foundation, Portland, OR, haley.holmer@va.gov.</p>
<h4>Conflict of Interest Disclosures</h4>
<p> None of the authors have any affiliations or financial involvement that conflicts with the material presented in this abstract.</p>
<h4>Funding/Support</h4>
<p> <a id="_idTextAnchor076"></a>This work is funded by the Agency for Healthcare Research and Quality (AHRQ) Effective Healthcare (EHC) Program through a contract to the Scientific Resource Center (Contract No. HHSA 75Q80122C00002). The authors of this abstract are responsible for its content. Statements in the abstract do not necessarily represent the official views of or imply endorsement by the AHRQ, US Department of Health and Human Services.</p>


[V] Identifying Methodological Concerns in Agency for Healthcare Research and Quality Evidence-based Practice Center Reports: Analysis of Editorial Review Comments

<p>Anjali Garg,<sup>1</sup> Preeti Panda,<sup>2</sup> Lydia Furman,<sup>3</sup> Kimberly Montez,<sup>4</sup> Alex R. Kemper,<sup>5</sup> Lewis First<sup>6</sup></p>
<h4>Objective</h4>
<p> To implement and evaluate a pilot mentorship program in which experienced peer reviewers mentor novice reviewers in writing peer reviews.</p>
<h4>Design</h4>
<p> In 2023, editorial board members of <i>Pediatrics</i>, the peer-reviewed journal of the American Academy of Pediatrics, volunteered for a pilot mentorship program to improve the peer review skills of early career physicians (medical students to junior faculty) to enhance the number of early career reviews within the pool present at <i>Pediatrics</i>. Mentors selected 1 or 2 mentees. The target participation for the pilot was 10 dyads. Literature review and expert consensus informed development of peer review mentorship materials. Curricula for mentees included didactics and checklists, and material for mentors provided feedback guidance. For objective assessment, the review quality instrument (RQI) was adapted into an 8-item, 3-point Likert scale questionnaire to score mentee reviews.<sup>1</sup> Mentees needed to complete at least 3 reviews with a score of 3 on at least 6 of 8 RQI items to graduate the program—a metric selected by the program leaders. Mentors and mentees completed a review of a regular article (as defined by the <i>Pediatrics </i>author guidelines) separately, then met together for structured feedback from the mentor to the mentee on the review. Each regular article was sent to the mentor based on their area of expertise; the turnaround time for completion was 3 weeks. The RQI was completed by the mentor, and the combined review was ultimately submitted to the editorial board by the mentor. Qualitative data regarding skills gained and areas for improvement were compiled and analyzed from intake and exit surveys completed by mentors and mentees.</p>
<h4>Results</h4>
<p> Eleven dyads enrolled, with 9 graduating from the program, an 81% graduation rate. Five dyads (55%) required 3 reviews to graduate, 3 (33%) required 4 reviews, and 1 (9%) required 5 reviews. The median number of RQI items scoring a 3 in each dyad’s first reviews increased from 4 (IQR, 2-7) to 7 (IQR, 6-8) by the last review. The range of total RQI scores increased from 16-24 (first reviews) to 23-24 (last reviews) for each dyad. Qualitative analysis indicated that the pilot program was feasible and effective. Notably, mentees reported increased comfort with peer review and improvement in their own manuscript writing skills. Mentors noted an improvement in their own reviewing skills through mentorship. Areas for improvement included targeted communication from the leadership team and allowing separate enrollment of mentees and mentors with matching by the program team to further enhance dissemination of this program.</p>
<h4>Conclusions</h4>
<p> The pilot program was successful in enrolling the target number of participants, graduating over 80% of the mentees with improvement of reviewing skills per increase in RQI scores, and demonstrated feasibility for future program building.</p>
<h4>Reference</h4>
<p>1. van Rooyen S, Black N, Godlee F. Development of the review quality instrument (RQI) for assessing peer reviews of manuscripts. <i>J Clin Epidemiol</i>. 1999;52(7):625-629. doi:10.1016/s0895-4356(99)00047-5</p>
<sup>1</sup>Division of Critical Care Medicine, Department of Pediatrics, Ann and Robert H. Lurie Children’s Hospital of Chicago, Chicago, IL, US, anjaliga07@gmail.com; <sup>2</sup>Division of Pediatrics, Department of Emergency Medicine, Stanford University, Stanford, CA, US; <sup>3</sup>University Hospitals Rainbow Babies &amp; Children’s, Cleveland, OH, US; <sup>4</sup>Department of Pediatrics, Wake Forest University School of Medicine, Winston-Salem, NC, US; <sup>5</sup>Division of Primary Care Pediatrics, Nationwide Children’s Hospital, Columbus, OH, US; <sup>6</sup>Department of Pediatrics, University of Vermont Children’s Hospital, Burlington, VT, US. </p>
<h4>Conflict of Interest Disclosures</h4>
<p> All authors are members of the Pediatrics editorial board.</p>
<h4>Acknowledgment</h4>
<p> We acknowledge Kate Larson and the Pediatrics editorial board administrative team for their help with the implementation of this program. </p>
<h4>Additional Information</h4>
<p> Preeti Panda is a co–corresponding author (preetipanda@stanford.edu).</p>


[V] A Pilot Program for Early Career Mentorship in Journal Peer-Review

<p>Marcus Tolentino Silva,<sup>1</sup>ís Freire Galvão<sup>2,3</sup> </p>
<h4>Objective</h4>
<p> Peer review is the core of scientific publication, and mentoring seems to enhance the quality of reviews.<sup>1</sup> We aimed to assess the effect of peer review training on reviewers’ preparedness to evaluate manuscripts.</p>
<h4>Design</h4>
<p> This study was designed to improve peer review quality for a diamond open access journal, <i>Epidemiologia e Serviços de Saúde: revista do SUS</i> (RESS). The intervention consisted of a 16-hour, in-person, 2-day workshop.<sup>2</sup> A checklist consisting of 31 critical, important, or desirable items for the preparation of peer review opinions was developed, and its use was encouraged to guide reports.<sup>3</sup> The courses took place in June (including associated editors and reviewers), September, and October 2024 in 3 Brazilian cities and consisted of practical sections of peer review, preceded by an explanation of the checklist and peer review process. Instructors were available for students’ questions, and discussions about bias in scientific communication were stimulated. Before starting, participants were invited to answer an anonymized questionnaire to assess their reviewing skills by means of a 5-point Likert scale (1: poor, 5: excellent), and another survey was sent electronically in January 2025 to assess checklist usefulness and the mean difference (MD) and 95% confidence interval (95% CI) of pre- and post-event confidence as a peer reviewer. We described participants’ characteristics and summarized means and standard deviations (SDs), and tested differences in distribution using Fisher exact, analysis of variance, Kruskal-Wallis, or <i>t</i> tests. All analyses were performed in Stata v.14.2. The project was approved as an extension activity of the University of Brasília, where all participants registered themselves.</p>
<h4>Results</h4>
<p> In total, 96 participants were trained; 51 (76.1%) were women and 38 (56.7%) had a doctoral degree, which was more frequent in first-edition participants (26 [70.3%]; <i>P</i> = .02). For 67 initial survey respondents, the mean (SD) familiarity with article submission was 4.1 (1.1) and with peer review, 3.5 (1.4). Mean (SD) confidence as a peer reviewer was 3.5 (1.3). The January 2025 survey had 43 respondents, who reported having issued a mean (SD) of 7.6 (25.9) peer reviews since the workshops. Confidence as a peer reviewer significantly increased (MD, 0.5 [95% CI, 0.1 to 1.0]; <i>P</i> = .01); 86.1% (n = 37) reported using the checklist to issue peer reviews and gave it a mean (SD) rating of 4.6 (0.9). Results are summarized in <b>Table 25-0888</b>.</p>

![](https://assets.underline.io/markdown_image/1/image/5f919a3984b0324269f4a1b78828f355.png)

<h4>Conclusions</h4>
<p> Peer review practical training sections and a checklist with suggested items improved peer reviewers’ confidence in evaluating manuscripts. Taking into account the costs and sustainability issues involved, this strategy may be viable to improve peer reviews.</p>
<h4>References</h4>
<p>1. Lyons-Warren AM, Aamodt WW, Pieper KM, Strowd RE. <br />A structured, journal-led peer-review mentoring program enhances peer review training. <i>Res Integr Peer Rev</i>. 2024;9(1):3. doi:10.1186/s41073-024-00143-x</p>
<p>2. Silva MT, Galv<span lang="ar-SA">ã</span>o TF. Systematization of peer review in Epidemiologia e Serviços de Saúde. <i>Epidemiol Serv Saude</i>. 2024;33:e20241001. doi:10.1590/S2237-96222024v33e20241001.en</p>
<p>3. Silva MT, Galv<span lang="ar-SA">ã</span>o TF. Revisão por pares: itens recomendados na elaboração de pareceres [Peer review: recommended items for preparing opinions]. June 19, 2024. Accessed May 24, 2025. <a href="https://osf.io/grn2a/"><span class="Hyperlink CharOverride-6">https://osf.io/grn2a</span></a></p>
<p class="abstract_conflict" lang="fr-FR"><span class="CharOverride-3" lang="pt-PT"><sup>1</sup></span><span lang="pt-PT">Faculdade d</span>e Ciências da Saúde, Universidade de Brasília, Brasília, Brazil; <span class="CharOverride-3" lang="pt-PT"><sup>2</sup></span>Faculdade de Ciências Farmacêuticas<span lang="pt-PT">, Universidade Estadual de Campinas, Campinas, Brazil, </span><a href="mailto:taisgalvao@gmail.com"><span class="Hyperlink CharOverride-6" lang="pt-PT">taisgalvao@gmail.com</span></a><span lang="pt-PT">; </span><span class="CharOverride-3" lang="pt-PT"><sup>3</sup></span><span lang="pt-PT">Secretaria de Vigilância em Saúde e Ambiente, Ministério da Saúde</span>, Brasília, Brazil.</p>
<h4>Conflict of Interest Disclosures </h4>
<p>Taís Freire Galv<span lang="ar-SA">ã</span>o reports being an executive editor of Epidemiologia e Serviços de Saúde: revista do SUS (RESS). Taís Freire Galvão is a member of the Peer Review Congress Advisory Board but was not involved in the review or decision for this abstract. No other disclosures were reported.</p>


[V] In-Person Peer Review Training to Improve Preparedness to Evaluate Manuscripts

<p>Susan Galandiuk,<sup>1</sup> Vaitheesh Jaganathan,<sup>2</sup> Hillary Simon<sup>1</sup></p>
<h4>Objective</h4>
<p> Despite advances in electronic manuscript submission systems, the editorial peer review process for medical journals has had few fundamental changes aside from promptness of review. In 2018, the editor of a society-owned journal created the <i>Reviewer’s Guild</i>, a mentored 1-year peer-review training program designed for surgical trainees, fellows in specialty training, and young academic faculty. The goal of the program was to attract the brightest individuals in the field, elevate the quality of peer review, and create a feeling of loyalty toward the journal. The program consists of a formal curriculum followed by 5 mentored reviews of full-length manuscripts. Reviews are graded by both mentors and assigned editors. Competitive program admission is by board member nomination and is limited to 45 individuals per year. From 2019 to 2025, 182 graduates completed the program. <i>Reviewer’s Guild</i> members are primarily (83%) from the United States, but 10 other countries are represented. Surveys of each year’s class are used for program improvement. Based on feedback, improvements included increased opportunities to learn statistical skills, networking, and leadership roles. The objective of this study was to report a separate 5-year survey to assess overall participant satisfaction with the program and career impact, particularly with respect to publishing, networking, and career opportunities.</p>
<h4>Design</h4>
<p> This cohort study sought to evaluate the impact of a mentored peer review program. The STROBE<sup>1</sup> reporting guideline was followed. An anonymous online questionnaire regarding the program was developed and emailed to graduates. The summative survey evaluated nomination pathway, mentor participation, leadership, and networking opportunities resulting from program participation as well as publication output.</p>
<h4>Results</h4>
<p> Ninety-six of the 182 graduates (52.8%) responded. Key survey responses are shown in <b>Table 25-0960</b>. Nearly all respondents reported enhanced understanding of the editorial process. Eighty-eight respondents (91.7%) had a generally positive experience and believed that the program improved the quality of their own publications and interpretation of others’ work. Seventeen respondents (17.7%) said they had improved understanding of statistical principles. Thirty-seven respondents (38.5%) published 5 or more manuscripts since graduating from the <i>Reviewer’s Guild</i>, 64 (66.7%) became involved with other journals (as ad hoc reviewers or editorial board members), and 16 (16.7%) of the highest performers received <i>Reviewer’s Guild</i> board or full editorial board appointments to the journal.</p>

![](https://assets.underline.io/markdown_image/1/image/ca2261e69c75f1d4ca1aa35e187cef59.png)

<h4>Conclusions</h4>
<p> The <i>Reviewer’s Guild</i> learning model appears to improve early career faculty and trainee peer review skills and leadership opportunities. While there are no comparisons with individuals who did not do the training, mentors saw improvement in review quality over the course of the program. In addition, graduates become loyal reviewers for the journal following graduation and are tracked as program graduates. As participants were surgeons, results may not be generalizable to other specialties. Such programs allow participants to become more involved in the editorial process and may potentially lead to journal leadership and career opportunities.<sup>2,3</sup></p>
<h4>References</h4>
<p>1. von Elm E, Altman DG, Egger M, Pocock SJ, Gøtzsche PC, Vandenbroucke JP; STROBE Initiative. The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies. <i>BMJ</i>. 2007;335(7624):806-808. doi:10.1136/bmj.39335.541782.AD</p>
<p>2. Isaacson RA, Bay SN, McCarty MM. Supporting the next generation of researchers: GENETICS peer review training program. <i>Sci Ed</i>. 2020;43:77. doi:10.36591/SE-D-4303-77</p>
<p>3. Lyons-Warren AM, Aamodt WW, Strowd R, Pieper KM, Merino JG. Assessment of a structured and mentored peer review curriculum on quality of peer review. Abstract presented at: Ninth International Congress on Peer Review and Scientific Publication; September 8-10, 2022; Chicago, IL. Accessed July 1, 2025. <a href="https://peerreviewcongress.org/wp-content/uploads/2022/11/Program2022.pdf"><span class="Hyperlink CharOverride-6">https://peerreviewcongress.org/wp-content/uploads/2022/11/Program2022.pdf</span></a></p>
<sup>1</sup>Division of Colon &amp; Rectal Surgery, University of Louisville, Louisville, KY, US, s0gala01@louisville.edu; <sup>2</sup>Price Institute for Surgical Research, University of Louisville, Louisville, KY, US.</p>
<h4>Conflict of Interest Disclosure</h4>
<p> Susan Galandiuk receives a stipend for services as editor of the journal Diseases of the Colon &amp; Rectum from the American Society of Colon and Rectal Surgeons.</p>


[V] Quality of an Educational Program to Empower Early Career Faculty and Trainees Through Mentored Training in Peer Review

<p>Chien Chou<sup>1</sup></p>
<h4>Objective </h4>
<p>This study surveyed how frequently Taiwanese researchers identify errors in others’ journal articles and their own, as well as how they address these errors.</p>
<h4>Design </h4>
<p>This cross-sectional study used a self-designed questionnaire from June to December 2023, categorizing errors into 7 dimensions: research design (6 items), data and information (8 items), writing and citation (12 items), authorship (2 items), conflicts of interest (2 items), journal publication (3 items), and others (1 item), totaling 34 items. <br />A 4-point Likert scale (often, occasionally, seldom, and never, ranging from 4 to 1) was used. The error-addressing methods were presented as questions, allowing multiple answers. Using snowball sampling, this study collected 593 valid responses from Taiwanese professors and researchers (312; 52.61%) as well as from PhD students with publication experience (281; 47.39%). Ethics approval was obtained from the National Yang Ming Chiao Tung University institutional review board before data collection. Content validity of the survey was confirmed by experts.</p>
<h4>Results </h4>
<p>The participants reported a higher frequency of identifying errors in others’ articles than those in their own (<i>t</i> = 17.88; 95% CI, 0.35-0.43; Cohen <i>d</i> = 0.67; <i>P</i> &lt; .001). Regarding identifying others’ errors, 22 items had a mean frequency above the theoretical mean of 2.5, in the categories of occasionally and seldom. The item with the highest frequency was spelling errors/omissions, followed by errors in the understanding or application of theory. For identifying one’s own errors, only the item spelling errors/omissions reached the theoretical mean of 2.5. The other 33 items, such as misunderstanding of previous research and errors in understanding theory application, ranged from 2.34 to 1.92, corresponding to seldom and never. In terms of addressing errors (multiple responses allowed; see <b>Table 25-0868</b>), 478 participants (80.6%) indicated that if they identify others’ errors, they would use them “as lessons to avoid making the same mistakes in the future,” 315 (53.1%) would take the errors “as examples to warn others,” and 179 (30.2%) would “just observe the errors and take no action.” As for addressing their own errors, 253 participants (42.7%) would “submit a proposed correction notice to the journal,” 232 (39.1%) would “take no action to correct the errors and use them as lessons,” 166 (28.0%) would take no action and use the errors “as examples to warn others.”</p>

![](https://assets.underline.io/markdown_image/1/image/89355b23266139edd8922606c3119dbb.png)

<h4>Conclusions </h4>
<p>Taiwanese researchers tend to identify errors more frequently in others’ work, viewing their own errors as cautionary examples for themselves rather than warnings for others. While correcting errors is a positive step, the tendency to be observant of others’ errors and more reflective to correct their own suggest that Taiwanese researchers tend to take errors more personally. The finding highlights the Committee on Publication Ethics principles of transparency and underscores the significance of upholding research integrity.</p>
<sup>1</sup>Institute of Education, National Yang Ming Chiao Tung University, Hsinchu, Taiwan, cchou@nycu.edu.tw.</p>
<h4>Conflict of Interest Disclosures</h4>
<p> None reported.</p>
<h4>Funding/Support</h4>
<p> This work was supported by the National Science and Technology Council, Taiwan (MOST110-2511-H-A49-008-MY4 and NSTC113-2750-V-A49-001-MY2).</p>
<h4>Role of the Funder/Sponsor</h4>
<p> The funder provided financial support only and had no role in the preparation of the abstract proposal.</p>


[V] Taiwanese Researchers' Perceptions of Errors and Their Coping Strategies

<p>Max Carlos Ramírez-Soto,<sup>1</sup> Laura Alvarado-Barbarán,<sup>1</sup> Dianeth Rojas-Naccha,<sup>1</sup> Ayda Luna-Mercado,<sup>1</sup> Arlet Arce-Zavala<sup>1</sup></p>
<h4>Objective</h4>
<p> Peer review plays a central role in selecting research proposals to obtain grants from science, technology, and innovation (STI) agencies.<sup>1</sup> Different criteria and questions are used during this process.<sup>2</sup> Given the time and resources involved, there is no available information on the reliability of the tools and review criteria used in the peer review of grant proposals. Peru’s National Program for Scientific Research and Advanced Studies (PROCIENCIA) uses a 2-tiered review process: an initial peer review to assess scientific merit, followed by a panel review to evaluate proposal relevance in line with funding priorities. Expert scientists are involved in both stages of the review process. Proposals may be improved prior to the final allocation of funding. In that context, we assess the reliability of the instruments used for the peer review of grant proposals at PROCIENCIA. </p>
<h4>Design</h4>
<p> This cross-sectional study assessed the construction and validation of 7 instruments used by PROCIENCIA for the peer review of grant applications between January 2022 and December 2024. Peer review instruments were applied in 7 funding calls (<b>Figure 25-1021</b>). Each financial instrument has distinct requirements and aims. The process of constructing and validating the instruments was carried out in the following stages: (1) literature review, (2) instrument construction, and (3) content validation. The literature search included studies on the criteria for reviewing STI grant applications, including the review criteria used by STI agencies. The findings from the literature review were used as evidence for the development of the evaluation criteria and questions included in the peer review instruments. We used a modified Delphi process to validate the content,<sup>3</sup> which included a review round by 8 to 15 expert panelists for each instrument. The experts were senior researchers with extensive experience in reviewing grant applications and scientific articles in various fields of scientific and technological knowledge. Each instrument contained 4 to 7 criteria (eg, knowledge of the researchers, relevance and coherence of the proposal, the scientific-technological aspect of the proposal, feasibility, results, and budget) as well as 3 to 8 questions. Content validation was performed at the overall instrument level (overall score) and for each criterion (individual score). Cronbach α was used to assess internal consistency; a value greater than .75 was considered acceptable.</p>

![](https://assets.underline.io/markdown_image/1/image/2dcedfe3054669882625d5c064f0f89e.png)

<h4>Results</h4>
<p> The 7 instruments used for the peer review of grant applications had an overall reliability index between .86 and .98 (<span class="CharOverride-4"><b>Figure 25-1021</b></span>). All instruments had acceptable reliability indices regarding assessment criteria (&gt;.75). The assessment criteria of the doctoral program, technological development, and improvement of laboratory equipment instruments had a higher reliability index (range, .89-1.00 and .92-1.00, respectively).</p>
<h4>Conclusions</h4>
<p> Our results show that PROCIENCIA uses validated and reliable instruments for the peer review of grant proposals. Validation and reliability of peer review instruments for STI funding is helpful in ensuring the integrity of the review process and in selecting research proposals for funding. Therefore, funding agencies should consider validating peer review instruments. </p>
<h4>References</h4>
<p>1. Hug SE, Aeschbach M. Criteria for assessing grant applications: a systematic review. <i>Palgrave Commun</i>. 2020;6(1):1-15. doi:10.1057/s41599-020-0412-9</p>
<p>2. Abdoul H, Perrey C, Amiel P, et al. Peer review of grant applications: criteria used and qualitative study of reviewer practices. <i>PLoS One</i>. 2012;7(9):e46054. doi:10.1371/journal.pone.0046054</p>
<p>3. Hsu C, Sandford BA. The Delphi technique: making sense of consensus. <i>Practl Assess Res Eval</i>. 2007;12(1):10. doi:10.7275/pdz9-th90</p>
<sup>1</sup>Beneficiary Selection Sub-Unit, National Program of Scientific Research and Advanced Studies (PROCIENCIA), National Council for Science, Technology and Innovation (CONCYTEC), Lima, Peru, mramirez@prociencia.gob.pe.</p>
<h4>Conflict of Interest Disclosures </h4>
<p>None reported.</p>
<h4>Acknowledgment </h4>
<p>The authors would like to thank Sixto Sánchez, president of CONCYTEC, and Dora Blitchtein Winicki De Levy, executive director of PROCIENCIA, for the technical support for the study.</p>


[V] Construction and Validation of Instruments for the Peer Review of Grant Proposals in Peru

<p>Rachel Claus<sup>1</sup></p>
<h4>Objective</h4>
<p> Research quality criteria guide grant applications and reviewer evaluations. When research crosses disciplinary boundaries, it requires more expansive quality criteria. Research evaluation is challenging in this context because concepts of research quality are rooted in disciplinary tradition.<sup>1</sup> Reviews are characterized by inconsistency and low agreement<sup>2</sup> and exacerbated by poorly defined criteria.<sup>3</sup> This presentation focuses on how consistently quality criteria are applied and interpreted by reviewers and explores reasons for inconsistency. </p>
<h4>Design </h4>
<p>All scoring data were collected from 2 noncompetitive review processes of multimillion-dollar research program proposals with aims to improve food security. In 2021, 32 research proposals were evaluated using 17 criteria on a standard 4-point Likert scale (0-3), with limited reviewer overlap. In 2024, 9 proposals were evaluated using 12 criteria, and 4 using 11 criteria, with no reviewer overlap. Each proposal was assessed by a panel of 3 reviewers who scored independently before discussing scores to reach consensus. In total, 45 proposals were evaluated by 80 individual subject matter experts across 45 panels. Score consistency was measured by discrepancy per criterion as the difference between the highest and lowest score awarded in a panel, and how many reviewers agreed on their individual assessments. The standard of consistency was met when at least 2 of 3 reviewers agreed, and the discrepancy between individual reviewer scores was less than or equal to 1 Likert scale point. A total of 696 panel-level measures of consistency were computed. Cross tabulations were used to identify frequencies of inconsistency by criterion. Interviews with reviewers were conducted to understand perceived reasons for individual review discrepancies and disagreement, and individual and panel-level score justifications were analyzed to explore criteria interpretations.</p>
<h4>Results</h4>
<p> There was a statistically significant relationship between consistency and the evaluation criteria (χ² = 61.013; <i>P</i> &lt; .001). Some criteria were more consistently applied than others. The frequency that each criterion met the standard of consistency is presented in <b>Table 25-1061</b>. Reviewers more frequently applied the following criteria inconsistently when evaluating proposals: comparative advantage (26.7%), monitoring, evaluation, and learning (24.4%), and overall theory of change (24.4%). Justified and transparent costing had the highest rate (71.9%) of inconsistency in 2021 but was not evaluated in the 2024 review cycle. Individual score discrepancies and disagreement were perceived by reviewers to result from diverse disciplinary expertise and a constructive way to achieve comprehensive quality assessments. More problematic reasons for discrepancies included misalignment of criteria to the application and different interpretations resulting from individual values and perceived abilities to make judgments.</p>

![](https://assets.underline.io/markdown_image/1/image/55ce83d15f0173abd5d1a75ab77f6b39.png)

<h4>Conclusions</h4>
<p> The preliminary results indicate scope for criteria clarification to improve consistency in interpretation and reliability of individual reviewer’s quality assessments of proposals. The approach can be adapted to test and inform improvements to quality criteria.</p>
<h4>References</h4>
<p>1. Defila R, Di Giulio A. Transdisciplinary development of quality criteria for transdisciplinary research. In: Regeer BJ, Klaassen P, Broerse JEW, eds. <i>Transdisciplinarity for Transformation</i>. Palgrave Macmillan, Cham; 2024. https://doi.org/10.1007/978-3-031-60974-9_5</p>
<p>2. Pier EL, Brauer M, Filut A, et al. Low agreement among reviewers evaluating the same NIH grant applications. <i>Proc Natl Acad Sci U S A</i>. 2018;115(12):2952-2957. doi:10.1073/pnas.1714379115</p>
<p>3. Abdoul H, Perrey C, Amiel P, et al. Peer review of grant applications: criteria used and qualitative study of reviewer practices. <i>PLoS One</i>. 2012;7(9):e46054. doi:10.1371/journal.pone.0046054</p>
<sup>1</sup>Royal Roads University, Victoria, BC, Canada, rachel.claus@royalroads.ca.</p>
<h4>Conflict of Interest Disclosures</h4>
<p> None reported.</p>
<h4>Funding/Support</h4>
<p> This research was supported by the Sustainability Research Effectiveness Program, the BC Graduate Scholarship, the Royal Roads Doctoral Scholarship, and the David Harris Flaherty Scholarship.</p>
<h4>Role of the Funder/Sponsor</h4>
<p> The research was done as part of the Sustainability Research Effectiveness Program, with input provided to the design of the study, review and approval of the abstract and input to the decision to submit the abstract for presentation. Scholarships provided general support without any role in design and conduct of the study; collection, management and interpretation of the data; preparation, review or approval of the abstract, and decision to submit the abstract for presentation.</p>


[V] Reviewers' Interpretation and Application of Research Quality Criteria in Grant Peer Review

<p>Joanie Sims Gould,<sup>1</sup> Anne Lasinsky,<sup>2</sup> Adrian Mota,<sup>3</sup> Karim M. Khan,<sup>1,2,4</sup> Clare L. Ardern<sup>5,6</sup></p>
<h4>Objective</h4>
<p> There is robust debate about the perceived strengths and weakness of grant peer review. Much of the research on issues in grant peer review is based on quantitative analysis of funding or scoring outcomes, which does not illuminate the experiences of peer review committee members. The objective of this study was to explore and understand the experiences and challenges faced by Canadian health research grant peer reviewers.</p>
<h4>Design</h4>
<p> This qualitative study received ethics approval from the University Behavioural Research Ethics Board. Study conduct and reporting followed the Consolidated Criteria for Reporting Qualitative Research (COREQ) guideline. Chairs, peer reviewers, and Scientific Officers of the Canadian Institutes of Health Research (CIHR) project grant competition peer review panels were interviewed. CIHR staff prepared a list of 50 potential participants who represented the 4 primary branches of CIHR research (biomedical, clinical, health systems and services, population health) from a public website. The inclusion criterion was having participated in a CIHR Project Grant competition peer review panel at least once as a peer reviewer, chair, or scientific officer. The researchers randomly selected names from the list and sent a recruitment email inviting participants to an online semistructured interview. The response rate was 36%. Two experienced qualitative researchers recruited and interviewed participants on a rolling basis from February to August 2022. The study team met biweekly to review the interview transcripts. All participants provided verbal consent for audio recording and reporting of quotes at the beginning of the interview. The interview guide was developed based on a priori concepts of peer review and the study team’s research experience in the field of grant peer review. There were questions about participants’ background, training in peer review, strengths and challenges of the review process, and managing conflict and bias. The analysis used a framework analysis approach.<sup>1</sup> Data were sifted, charted, and sorted based on key issues and themes to identify a thematic framework, compare and contrast themes, and explore similarities and differences.</p>
<h4>Results</h4>
<p> Eighteen participants were interviewed, all of whom were mid- or senior-career researchers (age 42-77 years); 11 participants (61.1%) were women and 7 (38.9%) were men. Twelve participants (66.7%) identified as White, 3 (16.7%) as South Asian, and 3 (16.7%) as other race or ethnicity. Participants identified 3 threats to grant peer review: (1) lack of training and limited opportunities to learn, (2) challenges in differentiating and rating applications of similar strength, and (3) relying on academic reputations and personal relationships in the review process to differentiate grant applications of a similar rating.</p>
<h4>Conclusions</h4>
<p> Experienced grant peer reviewers in the Canadian health funding context identified an absence of training and learning opportunities for peer review, difficulty differentiating between applications of similar strength, and an emphasis on academic reputations and personal relationships in rating applications for the Project Grant Competition.</p>
<h4>Reference</h4>
<p>1. Srivastava A, Thomson SB. Framework analysis: a qualitative methodology for applied policy research. <i>Journal of Administration and Governance</i>. 2009;72. Accessed May 31, 2025. <a href="https://ssrn.com/abstract=2760705"><span class="Hyperlink CharOverride-6">https://ssrn.com/abstract=2760705</span></a></p>
<sup>1</sup>Department of Family Practice, The University of British Columbia, Vancouver, British Columbia, Canada; <sup>2</sup>School of Kinesiology, The University of British Columbia, Vancouver, British Columbia, Canada; <sup>3</sup>Canadian Institutes of Health, Ottawa, Ontario, Canada; <sup>4</sup>Canadian Institutes of Health–Institute of Musculoskeletal Health and Arthritis, Vancouver, British Columbia, Canada; <sup>5</sup>Department of Physical Therapy, The University of British Columbia, Vancouver, British Columbia, Canada, clare.ardern@ubc.ca; <sup>6</sup>Sport and Exercise Medicine Research Centre, La Trobe University, Melbourne, Victoria, Australia.</p>
<h4>Conflict of Interest Disclosures</h4>
<p> Adrian Mota is acting vice president, Research–Programs for the Canadian Institute of Health Research (CIHR). Karim M. Khan is scientific director for the CIHR Institute of Musculoskeletal Health and Arthritis (2017-2025). No other disclosures were reported.</p>
<h4>Funding/Support </h4>
<p>This work was supported by a CIHR research operating grant (scientific directors) held by Karim M. Khan. </p>
<h4>Role of the Funder/Sponsor</h4>
<p> The CIHR had no role in design and conduct of the study. The CIHR did not participate in interpretation of the data or the preparation of the abstract and did not participate in the decision to submit the abstract for presentation.</p>


Downloads

Next from The 10th Peer Review Congress

[V] Trends in Peer Review Metrics at the Annals of African Surgery

Similar lecture

[V] Motivations to Participate in the Peer Review Process at the Journal of Urology

.css-70qvj9{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-align-items:center;-webkit-box-align:center;-ms-flex-align:center;align-items:center;}Downloads

Next from The 10th Peer Review Congress

[V] Trends in Peer Review Metrics at the Annals of African Surgery

Similar lecture

[V] Motivations to Participate in the Peer Review Process at the Journal of Urology

Downloads