United States

&lt;p&gt;Navita Goyal,&lt;sup&gt;1&lt;/sup&gt; Ivan Stelmakh,&lt;sup&gt;2&lt;/sup&gt; Nihar B. Shah,&lt;sup&gt;3&lt;/sup&gt; Hal Daumé III&lt;sup&gt;1&lt;/sup&gt;&lt;/p&gt;
&lt;h4&gt;Objective&lt;/h4&gt;
&lt;p&gt; Peer review often involves multiple reviewers to reduce bias and broaden perspectives.&lt;sup&gt;1,2&lt;/sup&gt; We examined the role of diversity among reviewers assigned to evaluate a paper as a factor in diversifying perspectives and improving the utility of the peer review process.&lt;sup&gt;3&lt;/sup&gt; We proposed 2 desiderata: reviews should cover most contents of the paper (high coverage) and reviews should add information not already present in other reviews (low redundancy). We hypothesized that reviews from diverse reviewers would exhibit higher coverage and lower redundancy.&lt;/p&gt;
&lt;h4&gt;Design&lt;/h4&gt;
&lt;p&gt; We conducted a cohort study using data from the International Conference on Machine Learning, a top-tier venue in computer science. Data were collected in February to April 2020 and analyzed January to April 2024. Reviewer diversity was defined along 5 binary dimensions: whether reviewers belonged to the same organization, same geographical region, similar seniority levels, similar research topics, or had common coauthors or papers. Organization and geographical data were obtained from reviewer profiles, seniority was based on h-index, coauthorship was determined from Google Scholar publications records, and topics were inferred using a topic model applied to the abstracts of reviewers’ prior publications. Seniority and topical diversity were binarized using median h-index and topical similarity scores, respectively. Coverage was measured by the breadth of review criteria addressed—specifically, the aspects of the paper discussed (eg, summary, motivation, originality, soundness) and the types of arguments used (eg, fact, request, reference, quote)—and lexical and semantic overlap between the reviews and the paper abstract. Redundancy was measured as the lexical and semantic overlap among different reviews. Both measures were computed using natural language processing tools. To isolate the relationship between diversity and review coverage and redundancy, we controlled for potential confounders, including reviewers’ expertise (measured as similarity between reviewer’s published papers and the submitted paper), reviewer characteristics (organization, location, topic, coauthors, and seniority), and paper content (by comparing diverse and nondiverse slates of reviewers for the same paper). We used a weighted linear regression model with review coverage or redundancy as the dependent variable and diversity and confounding factors as independent variables. Statistical significance was assessed using &lt;i&gt;t&lt;/i&gt; tests.&lt;/p&gt;
&lt;h4&gt;Results&lt;/h4&gt;
&lt;p&gt; This study included 4991 submitted papers and 3637 reviewers. Diversity in publication networks and seniority were associated with broader coverage of review criteria (&lt;span class=&quot;CharOverride-4&quot;&gt;Table 25-1066&lt;/span&gt;). Topical diversity was associated with a broader coverage of the paper. No significant association was found between coverage and organizational and geographical diversity. Except for geographical diversity, diversity in organizations, seniority, topics, or publications networks were associated with lower redundancy among reviews. Furthermore, publication network-based diversity alone was associated with varying perspectives (that is, low redundancy) within specific review criteria.&lt;/p&gt;
&lt;h4&gt;Conclusions&lt;/h4&gt;
&lt;p&gt; Our study revealed that, in addition to optimizing individual reviewer expertise, choosing diverse slates of reviewers can lead to better reviews with higher coverage and lower redundancy. The findings are limited to certain aspects of review utility, namely redundancy and coverage, operationalized through specific measures. We recognize that there may be other factors that are crucial, either generally or specific to a particular conference or journal, which we have not addressed.&lt;/p&gt;
&lt;h4&gt;References&lt;/h4&gt;
&lt;p&gt;1. Shah NB. Challenges, experiments, and computational solutions in peer review. Comm ACM. 2022;65(6):76-87. doi:10.1145/3528086&lt;/p&gt;
&lt;p&gt;2. Lee CJ, Sugimoto CR, Zhang G, Cronin B. Bias in peer review. J Am Soc Inf Sci Technol. 2013;64(1):2-17. doi: 10.1002/asi.22784&lt;/p&gt;
&lt;p&gt;3. Jackson SE, May KE, Whitney K, Guzzo RA, Salas E. Understanding the dynamics of diversity in decision making teams. Team Effectiveness Decision Making Organ. 1995;204:261.&lt;/p&gt;
&lt;sup&gt;1&lt;/sup&gt;University of Maryland, College Park, MD, US, navita@umd.edu; &lt;sup&gt;2&lt;/sup&gt;New Economic School, Moscow, Russia; &lt;sup&gt;3&lt;/sup&gt;Carnegie Mellon University, Pittsburgh, PA, US.&lt;/p&gt;
&lt;h4&gt;Conflict of Interest Disclosures&lt;/h4&gt;
&lt;p&gt; Nihar B. Shah is a member of the Peer Review Congress Advisory Board but was not involved in the review or decision for this abstract.&lt;/p&gt;
&lt;h4&gt;Funding/Support&lt;/h4&gt;
&lt;p&gt; This research was supported by National Science Foundation award 1942124 and Office of Naval Research award N000142212181.&lt;/p&gt;
&lt;h4&gt;Role of the Funder/Sponsor&lt;/h4&gt;
&lt;p&gt; The funders played no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the abstract; or decision to submit the abstract for presentation.&lt;/p&gt;
&lt;h4&gt;Additional Information&lt;/h4&gt;
&lt;p&gt; We thank the current and former members of the University of Maryland Computational Linguistics and Information Processing laboratory, especially Alexander Hoyle and Connor Baumler, for their useful suggestions and feedback on this work. This study was approved by the University of Maryland Institutional Review Board.&lt;/p&gt;


The 10th Peer Review Congress

Diversity Among Reviewers Assigned to Evaluate a Paper as a Factor in Diversifying Perspective and Improving the Peer Review Process in Computer Science

peer review process and models

editorial and peer review process

diversity and inclusion

The Tenth Peer Review Congress featured 3 days for presentations of new research into peer review and all aspects of scientific publication, bias, quality of reporting, and information access and dissemination.

This year’s meeting was hybrid, and all plenary sessions were livestreamed with opportunities to view presentations and engage with participants in person and virtually during and after the meeting. All recordings are now available on Underline for viewing.

<div style="position:relative;padding-top:max(60%,324px);width:100%;height:0;"><iframe style="position:absolute;border:none;width:100%;height:100%;left:0;top:0;" src="https://online.fliphtml5.com/ebtyf/mget/"  seamless="seamless" scrolling="no" frameborder="0" allowtransparency="true" allowfullscreen="true" ></iframe></div>


#### Why? 
Our aim is to encourage research into the quality and credibility of peer review and scientific publication, to establish the evidence base on which scientists can improve the conduct, reporting, and dissemination of scientific research.

#### Who? 
Participants include editors and publishers of scientific peer-reviewed journals, researchers, funders, bibliometric and informatics experts, information innovators, librarians, journalists, policymakers, ethicists, scientific information disseminators, and anyone interested in the progress of the scientific information enterprise and the quality of scientific evidence.

The Congress embraces a wide range of disciplines, including biomedicine, health and life sciences, applied sciences, basic sciences, physical and chemical sciences, mathematics, computer sciences, engineering, economics, and social sciences as well as new and emerging disciplines.

#### What? 
The Congress provides a forum for the presentation and discussion of new research into the quality and credibility of editorial and funding peer review, scientific and scholarly publication, research methods and reporting practices, identification and management of biases, transparency, reproducibility, access, publication and dissemination metrics, identification and management of threats to the integrity of science, and advances in information exchange.

**Follow us**

[![](https://assets.underline.io/markdown_image/1/image/662a22766b9788ed3766cb23cd156e32.png)](https://x.com/peerrevcongress)  [![](https://assets.underline.io/markdown_image/1/image/bb105afb1773a01fa169aa97ca1af49a.png)](https://www.youtube.com/channel/UCsG_0nKi3_Zfyv0yRPdxjWQ)  [![](https://assets.underline.io/markdown_image/1/image/034e1e70a194a9a4178fdf470ef796db.png)](https://bsky.app/profile/peerreviewcongress.bsky.social)



You need to log in with the email address you registered with. 



Please log in to explore this event.

Registration is required to attend this event, please [here](https://peerreviewcongress.org/registration-information/) to secure your ticket. Please check your spam/promotional inbox too.



Please register!

The Tenth Peer Review Congress features 3 days for presentations of new research into peer review and all aspects of scientific publication, bias, quality of reporting, and information access and dissemination.

<p>Navita Goyal,<sup>1</sup> Ivan Stelmakh,<sup>2</sup> Nihar B. Shah,<sup>3</sup> Hal Daumé III<sup>1</sup></p>
<h4>Objective</h4>
<p> Peer review often involves multiple reviewers to reduce bias and broaden perspectives.<sup>1,2</sup> We examined the role of diversity among reviewers assigned to evaluate a paper as a factor in diversifying perspectives and improving the utility of the peer review process.<sup>3</sup> We proposed 2 desiderata: reviews should cover most contents of the paper (high coverage) and reviews should add information not already present in other reviews (low redundancy). We hypothesized that reviews from diverse reviewers would exhibit higher coverage and lower redundancy.</p>
<h4>Design</h4>
<p> We conducted a cohort study using data from the International Conference on Machine Learning, a top-tier venue in computer science. Data were collected in February to April 2020 and analyzed January to April 2024. Reviewer diversity was defined along 5 binary dimensions: whether reviewers belonged to the same organization, same geographical region, similar seniority levels, similar research topics, or had common coauthors or papers. Organization and geographical data were obtained from reviewer profiles, seniority was based on h-index, coauthorship was determined from Google Scholar publications records, and topics were inferred using a topic model applied to the abstracts of reviewers’ prior publications. Seniority and topical diversity were binarized using median h-index and topical similarity scores, respectively. Coverage was measured by the breadth of review criteria addressed—specifically, the aspects of the paper discussed (eg, summary, motivation, originality, soundness) and the types of arguments used (eg, fact, request, reference, quote)—and lexical and semantic overlap between the reviews and the paper abstract. Redundancy was measured as the lexical and semantic overlap among different reviews. Both measures were computed using natural language processing tools. To isolate the relationship between diversity and review coverage and redundancy, we controlled for potential confounders, including reviewers’ expertise (measured as similarity between reviewer’s published papers and the submitted paper), reviewer characteristics (organization, location, topic, coauthors, and seniority), and paper content (by comparing diverse and nondiverse slates of reviewers for the same paper). We used a weighted linear regression model with review coverage or redundancy as the dependent variable and diversity and confounding factors as independent variables. Statistical significance was assessed using <i>t</i> tests.</p>
<h4>Results</h4>
<p> This study included 4991 submitted papers and 3637 reviewers. Diversity in publication networks and seniority were associated with broader coverage of review criteria (<span class="CharOverride-4">Table 25-1066</span>). Topical diversity was associated with a broader coverage of the paper. No significant association was found between coverage and organizational and geographical diversity. Except for geographical diversity, diversity in organizations, seniority, topics, or publications networks were associated with lower redundancy among reviews. Furthermore, publication network-based diversity alone was associated with varying perspectives (that is, low redundancy) within specific review criteria.</p>
<h4>Conclusions</h4>
<p> Our study revealed that, in addition to optimizing individual reviewer expertise, choosing diverse slates of reviewers can lead to better reviews with higher coverage and lower redundancy. The findings are limited to certain aspects of review utility, namely redundancy and coverage, operationalized through specific measures. We recognize that there may be other factors that are crucial, either generally or specific to a particular conference or journal, which we have not addressed.</p>
<h4>References</h4>
<p>1. Shah NB. Challenges, experiments, and computational solutions in peer review. Comm ACM. 2022;65(6):76-87. doi:10.1145/3528086</p>
<p>2. Lee CJ, Sugimoto CR, Zhang G, Cronin B. Bias in peer review. J Am Soc Inf Sci Technol. 2013;64(1):2-17. doi: 10.1002/asi.22784</p>
<p>3. Jackson SE, May KE, Whitney K, Guzzo RA, Salas E. Understanding the dynamics of diversity in decision making teams. Team Effectiveness Decision Making Organ. 1995;204:261.</p>
<sup>1</sup>University of Maryland, College Park, MD, US, navita@umd.edu; <sup>2</sup>New Economic School, Moscow, Russia; <sup>3</sup>Carnegie Mellon University, Pittsburgh, PA, US.</p>
<h4>Conflict of Interest Disclosures</h4>
<p> Nihar B. Shah is a member of the Peer Review Congress Advisory Board but was not involved in the review or decision for this abstract.</p>
<h4>Funding/Support</h4>
<p> This research was supported by National Science Foundation award 1942124 and Office of Naval Research award N000142212181.</p>
<h4>Role of the Funder/Sponsor</h4>
<p> The funders played no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the abstract; or decision to submit the abstract for presentation.</p>
<h4>Additional Information</h4>
<p> We thank the current and former members of the University of Maryland Computational Linguistics and Information Processing laboratory, especially Alexander Hoyle and Connor Baumler, for their useful suggestions and feedback on this work. This study was approved by the University of Maryland Institutional Review Board.</p>


poster

<p>James M. Zumel Dumlao,<sup>1</sup> Misha Teplitskiy<sup>1</sup></p>
<h4>Objective</h4>
<p> This study tested whether (1) peer reviewers from the same country as the corresponding author are more favorable compared with those from a different country and (2) corresponding authors have differential access to these same-country reviewers (SCRs). While reviewer geographic diversity was previously considered,<sup>1,2</sup> attempts to establish same-country preferences were confounded<sup>2</sup> or had small samples.<sup>3</sup></p>
<h4>Design</h4>
<p> This cross-sectional study (using the STROBE reporting guideline) took place from October 2022 to May 2025. Using metadata from 204,718 submissions to 60 STEM journals published by the Institute of Physics Publishing from 2018 to 2022, the study employed linear models with fixed effects and 
	t tests for significance at the 1% level. Standard errors were clustered at the level of the fixed effects. Only first-round reviews were included to avoid autocorrelation (2.10 first-round reviewers per submission on average; 14.67% were SCRs). Fixed effects limit analysis to within-group comparisons. For example, manuscript fixed effects control for time-invariant variation across manuscripts like quality and proportion of SCRs on the review panel. To test the first hypothesis, this study compared the likelihood of a positive review (outcome) between SCRs and non-SCRs (exposure) using manuscript and reviewer fixed effects. Review positivity is equal to 0 if a reviewer recommended to reject the submission and 1 otherwise. To test the second hypothesis, this study estimated the likelihood a reviewer is an SCR (outcome) by corresponding author country income category (exposure) using journal fixed effects and controls for logged team size and anonymization status. Of 156 corresponding author countries, 60 were high-income countries (HIC), 39 were upper-middle income (UMIC), and 57 were low- and lower-middle income (LLMIC), according to World Bank classification.</p>
<h4>Results</h4>
<p> SCRs were more likely to give positive reviews compared with non-SCRs on the same manuscript (4.78 percentage points higher [95% CI, 3.78-5.78]; n = 181,228). Estimates of relative SCR positivity were positive for all country income groups (<b>Figure 25-1099</b>). HIC authors received SCRs over twice as often as LLMIC authors (4.97% [95% CI, 3.50%-6.44%] vs 2.66% [95% CI, 1.41%-3.91%]; n = 237,823). Authors from countries most represented among reviewers—the US, China, and India—had the greatest likelihood of receiving SCRs (30.27% [95% CI, 30.26%-30.28%], 26.59% [95% CI, 24.41%-28.77%], and 13.69% [95% CI, 10.77%-16.61%], respectively; n = 237,823).</p>

![](https://assets.underline.io/markdown_image/1/image/5893761af38fd09800d0ef2976033283.png)

<h4>Conclusions</h4>
<p> Lack of geographical diversity among reviewers—possibly reflecting historical patterns of scientific production associated with country wealth—may introduce structural advantages if reviewers favor work from their own country, as this study demonstrated. However, diversification may be challenging if qualified reviewer capacity is low in underrepresented countries. Further research should examine generalizability across fields and test potential remedies.</p>
<h4>References</h4>
<p>1. Smith OM, Davis KL, Pizza RB, et al. Peer review perpetuates barriers for historically excluded groups. <i>Nat Ecol Evol</i>. 2023;7(4):512-523. doi:10.1038/s41559-023-01999-w</p>
<p>2. Murray D, Siler K, Larivière V, et al. Author-reviewer homophily in peer review. scientific communication and education <i>bioRxiv</i>. 2018. doi:10.1101/400515</p>
<p>3. Tomkins A, Zhang M, Heavlin WD. Reviewer bias in single- versus double-blind peer review. <i>Proc Natl Acad Sci USA</i>. 2017;114(48):12708-12713. doi:10.1073/pnas.1707323114</p>
<sup>1</sup>University of Michigan School of Information, Ann Arbor, MI, US, jamesmzd@umich.edu.</p>
<h4>Conflict of Interest Disclosures</h4>
<p> None reported.</p>
<h4>Funding/Support</h4>
<p> This study received funding from the Institute of Physics Publishing, Schmidt Futures, and Science for Progress Initiative.</p>
<h4>Role of the Funder/Sponsor</h4>
<p> The Institute of Physics Publishing was involved in the collection, management, analysis, and interpretation of the data and preparation, review, or approval of the abstract. Schmidt Futures and Science for Progress Initiative contributed personnel funding.</p>
<h4>Acknowledgment</h4>
<p> We thank the Institute of Physics Publishing, and particularly Kim Eggleton and Mikka Pers, for supporting this project. We thank Schmidt Futures and the Science for Progress Initiative for financial support.</p>


Geographical Representation of Author Country among Peer Reviewers and Publishing Success at 60 STEM Journals

<p>Mabel Chew,<sup>1</sup> Taissa Vila,<sup>2</sup> Jashelle Caga-Meller,<sup>3</sup> Zoë Mullan,<sup>4</sup> Diana Samuel<sup>5</sup></p>
<h4>Objective</h4>
<p> As part of The Lancet<i> </i>Group’s commitment to advancing equity, diversity, and inclusion, we implemented the Sex and Gender Equity in Research (SAGER) guidelines<sup>1</sup> in 2023, and <i>The Lancet</i> guidance on reporting race and ethnicity<sup>2</sup> in June 2024 across 24 Lancet Group journals. Although data exist on the reporting of race and ethnicity in published articles after similar guidelines were launched,<sup>3</sup> less is known about how authors view and engage with guidelines, which is crucial to implementing and refining editorial policy. Thus, we aimed to examine authors’ awareness of and responses to these guidelines and identify challenges to guideline implementation.</p>
<h4>Design</h4>
<p> Corresponding authors of research or review articles submitted to a <i>Lancet</i> journal, who had received a request for revision between July and December 2024, were invited to participate in an online survey in January 2025. The survey asked authors about their awareness of the guidelines, changes they made to their manuscripts in response to editorial advice, the ease with which they made those changes, any challenges encountered, and how these guidelines might influence their next project.</p>
<h4>Results</h4>
<p><i> </i>The survey response rate was 22% (484 of 2185 invitations). The largest proportion of respondents was from the US (68 [14%]), followed by China (65 [13%]), the UK (59 [12%]), Australia (30 [6%]), the Netherlands (27 [6%]), Germany (21 [4%]), Canada (19 [4%]), and Sweden (17 [4%]). Fifty-four percent identified as a man (n=262), 41% as a woman (n=197), 1% as non-binary or gender diverse (n=5), and 4% preferred not to say (n=20); (64%) were involved in research and/or development, and 191 (40%) were senior researchers or in middle management. Of the 484 respondents, 193 (40%) and 220 (46%) were aware of the sex and gender and race and ethnicity reporting guidelines before submitting, respectively. A total of 246 respondents (51%) and 266 (55%), respectively, were not required to collect these data, and 9 (2%) and 33 (7%) were not permitted to collect these data. Among 153 respondents (32%) who amended their manuscript as a result of the sex and gender guidance, 104 found this easy or very easy to do. Fewer respondents (90 [19%]) made changes in response to the race and ethnicity guidance, with 55 finding this easy or very easy. <b>Box 25-1122</b> includes open-text responses on challenges in addressing these guidelines. Approximately one-half of respondents said they were likely or very likely to do things differently in their next project as a result of these guidelines (255 [53%] and 221 [46%] for each guidance, respectively).</p>

<h4>Conclusions</h4>
<p> Less than one-half of the authors surveyed reported awareness of sex and gender or race and ethnicity reporting guidelines before submission, suggesting that increasing awareness could enhance engagement. Most authors who amended their manuscript in response to these guidelines found this to be easy. The challenges identified provide opportunities for journals to refine editorial processes.</p>

![](https://assets.underline.io/markdown_image/1/image/f556edaec2af5850b8b81ae3c23ff545.png)

<h4>References</h4>
<p>1. Heidari S, Babor TF, De Castro P, Tort S, Curno M. Sex and gender equity in research: rationale for the SAGER guidelines and recommended use. <i>Res Integr Peer Rev</i>. 2016;1:2. doi:10.1186/s41073-016-0007-6</p>
<p>2. Chew M, Samuel D, Mullan Z, Kleinert S; Lancet Group for Racial Equity (GRACE). The Lancet Group’s new guidance to authors on reporting race and ethnicity. <i>Lancet</i>. 2024;403(10442):2360-2361. doi:10.1016/S0140-6736(24)01081-X</p>
<p>3. Flanagin A, Cintron MY, Christiansen SL, et al. Comparison of reporting race and ethnicity in medical journals before and after implementation of reporting guidance, 2019-2022. <i>JAMA Netw Open</i>. 2023;6(3):e231706. doi:10.1001/jamanetworkopen.2023.1706</p>
<sup>1</sup><i>The Lancet, </i>Elsevier Australia, Chatswood, Sydney, Australia, mabel.chew@lancet.com; <sup>2</sup><i>The Lancet Regional Health—Americas, </i>Rio de Janeiro, Brazil; <sup>3</sup><i>The Lancet Regional Health—Western Pacific</i>, Sydney, Australia; <sup>4</sup><i>The Lancet Global Health</i> London, UK; <sup>5</sup><i>The Lancet Digital Health</i>, London, UK.</p>
<h4>Conflict of Interest Disclosures</h4>
<p> Mabel Chew is a member and former co-chair of The Lancet Group for Racial Equity and has received funding for conference travel expenses from the World Conference on Research Integrity, Committee on Publication Ethics, and Nuffield Department of Primary Care Health Sciences. Taissa Vila is co-chair of and a member of The Lancet Group for Racial Equity. Jashelle Caga-Meller is honorary clinical senior lecturer at the Faculty of Medicine and Health, University of Sydney, Australia, and a member of The Lancet Group for Racial Equity. Diana Samuel is a member and former co-chair of The Lancet Group for Racial Equity, and a member and former Chair of the European Association of Science Editors’ (EASE) EDI Committee.</p>
<h4>Acknowledgments </h4>
<p>We thank all authors who responded to the survey; Marco Conforti for database support; Louise Hall and Adrian Mulligan for their help in conducting the survey and in preliminary reporting and analysis; Pooja Jha, Rupa Sarkar, Lan-Lan Smith, Richard Horton, and members of the <i>Lancet’s</i> Group for Racial Equity (GRacE) for advice and support.</p>


Author Responses to Editorial Guidance on Reporting of Sex, Gender, Race and Ethnicity Data

<p>Harleen K. Marwah,<sup>1</sup> Abarna Pearl,<sup>1</sup> Christos P. Kotanidis,<sup>1,2,3</sup> Sarah Gorey,<sup>1,4</sup> Darren Taichman,<sup>1,5</sup> Mary Beth Hamel<sup>1,6</sup></p>
<h4>Objective</h4>
<p> Trials increasingly include participants from low- and middle-income countries (LMICs),<sup>1</sup> but authorship representation may not be proportional. LMIC-affiliated authors on trials recruiting from LMICs offer important perspectives, which have implications for equity, expertise, and capacity building. Prior data have looked at specialty journals or non-Western country categories,<sup>2</sup> but author representation specifically in trials with participants from LMICs is not known. This study evaluated trends over time in LMIC-affiliated authorship among trials recruiting participants from LMICs.</p>
<h4>Design</h4>
<p> Using PubMed, we extracted trials published during alternating months (6 mo/y) in the years 2008, 2013, 2018, and 2023 from 4 high-impact medical journals: the <i>New England Journal of Medicine</i>, <i>The Lancet</i>, <i>JAMA</i>, and <i>BMJ</i>. Four investigators manually identified trials that recruited participants from LMICs, defined by the Wellcome Trust<i> </i>as countries with low-income or middle-income economies<i>.</i><sup>3</sup> We designed and calculated 3 metrics for each trial: (1) an enrolling country ratio (the number of recruiting LMICs divided by the total number of recruiting countries), (2) an author ratio (the number of LMIC-affiliated authors divided by the total number of authors in a trial), and (3) a representation index (RI) (the author ratio divided by the enrolling country ratio). The RI controlled for the proportion of recruiting LMICs in each trial. An RI less than 1 indicated that the proportion of LMIC-affiliated authors was lower than the proportion of recruiting LMICs. An RI of 1 indicated that the 2 ratios were exactly proportionate.</p>
<h4>Results</h4>
<p> The PubMed search yielded 865 trials, 302 of which recruited participants from LMICs. Of these, 139 (46.0%) were published in the <i>New England Journal of Medicine</i>, 117 (38.7%) in <i>The Lancet</i>, 35 (11.6%) in <i>JAMA</i>, and 11 (3.6%) in <i>BMJ</i>. The number and proportion of trials recruiting participants from LMICs increased between 2008 (52/174, 29.9%) and 2023 (117/262, 44.7%). The mean enrolling country ratio decreased over time, from 0.62 in 2008 to 0.41 in 2023 (<b>Figure 25-1131</b>, blue line). The mean author ratio decreased after 2008 and then remained similar; in 2008, it was 0.34 and in 2023, it was 0.23 (<b>Figure 25-1131</b>, gray line). Between 2008 and 2023, the RI increased modestly from 0.49 to 0.53 (<b>Figure 25-1131</b>, teal line).</p>

![](https://assets.underline.io/markdown_image/1/image/8b793884c95e0b12aa0c86e9d286c00d.png)

<h4>Conclusions</h4>
<p> This study highlights that LMIC-affiliated authorship in LMIC-recruiting trials has increased modestly over time but remains disproportionate to LMIC participant recruitment sites. Limitations of the study include the lack of information on the proportion of LMIC-recruited participants for each trial. This information could be a stronger indicator as to what LMIC-affiliated authorship representation should look like. Also, LMIC definitions may change over time and by defining organization. While there has been modest improvement in author representation, increased attention may help promote representative and ethical research.</p>
<h4>References</h4>
<p>1. Number of clinical trials by year, country, WHO region and income group (1999-2024). World Health Organization. December 2024. Accessed April 15, 2025. <a href="https://www.who.int/observatories/global-observatory-on-health-research-and-development/monitoring/number-of-clinical-trials-by-year-country-who-region-and-income-group"><span class="Hyperlink CharOverride-6">https://www.who.int/observatories/global-observatory-on-health-research-and-development/monitoring/number-of-clinical-trials-by-year-country-who-region-and-income-group</span></a></p>
<p>2. De Oliveira-Gomes D, Guilliod C, Gebran K, et al. Equity and representation in cardiology research: a comprehensive analysis of authorship from low and low-middle income countries in cardiology journals. <i>Curr Probl Cardiol</i>. 2024;49(8):102667. doi:10.1016/j.cpcardiol.2024.102667</p>
<p>3. Low- and middle-income countries. Wellcome Trust. Accessed January 8, 2025. <a href="https://wellcome.org/grant-funding/guidance/prepare-to-apply/low-and-middle-income-countries"><span class="Hyperlink CharOverride-6">https://wellcome.org/grant-funding/guidance/prepare-to-apply/low-and-middle-income-countries</span></a></p>
<sup>1</sup><i>New England Journal of Medicine</i>, Boston, MA, US, hmarwah@nejm.org; <sup>2</sup>Heart and Vascular Center, Division of Cardiovascular Medicine, Brigham and Women’s Hospital, Harvard Medical School, Boston, MA, US; <sup>3</sup>Division of Cardiovascular Medicine, Radcliffe Department of Medicine, University of Oxford, UK; <sup>4</sup>Stroke Clinical Trials Network Ireland, University College Dublin Clinical Research Centre, Mater Misericordiae University Hospital, Dublin, Ireland; <sup>5</sup>Department of Medicine, Division of Pulmonary, Allergy and Critical Care Medicine, University of Pennsylvania School of Medicine, Philadelphia, PA, US; <sup>6</sup>Division of General Medicine and Primary Care, Department of Medicine, Harvard Medical School, Beth Israel Deaconess Medical Center, Boston, MA, US.</p>
<h4>Conflict of Interest Disclosures</h4>
<p> Harleen K. Marwah, Abarna Pearl, Christos P. Kotanidis, and Sarah Gorey are editorial fellows at the <i>New England Journal of Medicine</i> (2024-2025). Darren Taichman is a deputy editor and Mary Beth Hamel is executive editor of the <i>New England Journal of Medicine</i>. No other disclosures were reported.</p>


Representation of Authors From Low- and Middle-Income Countries (LMIC) in Trials With Participants From LMICs

<p><span lang="fr-FR">Vincent Kipkorir,</span><sup>1</sup><span lang="fr-FR"> Godfrey Sama,</span><sup>1</sup><span lang="fr-FR"> Mumba Chalwe,</span><sup>1</sup><span lang="fr-FR"> Tihitena Negussie,</span><sup>1</sup><span lang="fr-FR"> Michael Mwachiro,</span><sup>1</sup><span lang="fr-FR"> Robert Parker,</span><sup>1</sup><span lang="fr-FR"> Seke Kazuma,</span><sup>1</sup><span lang="fr-FR"> Stella Itungu,</span><sup>1</sup><span lang="fr-FR"> Abebe Bekele</span><sup>1</sup></p>
<h4>Objective</h4>
<p> Underrepresentation along geographical and gender lines has historically been observed in academic surgery.<sup>1</sup> However, efforts have been made in recent years to ensure equity and inclusion in the surgical learning environment to increase female representation and ensure geographically diversified research engagements.<sup>2</sup> We set out to explore the evolution of authorship diversity in African surgical research using publications featured in an African surgical journal.</p>
<h4>Design</h4>
<p> This was a retrospective cross-sectional study. Data were collected from all articles published in the <i>East and Central African Journal of Surgery</i> between 2003 and 2023. Article metadata and author biodata were extracted from each article for subsequent analysis. Author gender was established by 2 independent reviewers through an online search, confirmed by a minimum of 2 web pages including LinkedIn, editorial profiles, institutional websites, and ResearchGate. For original articles, case series, and case reports, geographical location was determined by the primary study site. For the other articles, we used the author’s institutional affiliation.</p>
<h4>Results </h4>
<p>A total of 889 articles were included featuring 3001 authors; 2609 (86.9%) were male and 392 (13.1%) were female. In total, 92 females (3.1%) were designated as first authors, 93 (3.1%) as corresponding authors, and 90 (3.0%) as last authors. There was a predominance of males in the positions of first, last, and corresponding author. Despite this, the number of females represented in these categories had a steady increase over the years. The gender disparity ratio increased in the first 15 years, then dropped in the last 5 years. There were 2523 African authors (84.1%), with Ethiopia having the largest number of authors (155 [5.2%]). The distribution according to country and the trends in female authorship are presented in <b>Figure 25-1149</b>.</p>

![](https://assets.underline.io/markdown_image/1/image/21e0e5a9b54c9d6917f17d43e8d66dbb.png)

<h4>Conclusions </h4>
<p>Our findings show disparities in the ratio of male to female authors, with a substantial male predominance. While it is encouraging to note the steady increase in the number of female authors over the years, a wide gap remains. The steady rise may be attributed to the overall rise in female surgeons following the recent establishment of women in surgery associations and better advocacy and funding toward empowerment of female surgeons in Africa. With a female surgical workforce density of about 11% in College of Surgeons of East, Central and Southern Africa member countries, further efforts are needed to ensure that these potential female authors are well represented, especially as first, last, and corresponding authors.<sup>3</sup> Strategies currently in place to foster surgical research in Africa need to be sustained to maintain the current trajectory of African representation in the global surgical research landscape. This study is limited by the subjective nature of ascertaining gender and location; however, the level of bias was reduced by having 2 independent assessors.</p>
<h4>References</h4>
<p>1. Rathna RB, Biswas J, D’Souza C, Joseph JM, Kipkorir V, Dhali A. Authorship diversity in general surgery-related Cochrane systematic reviews: a bibliometric study. <i>BJS</i>. 2023;110(8):989-990. doi:10.1093/bjs/znad117</p>
<p>2. Burks CA, Russell TI, Goss D, et al. Strategies to increase racial and ethnic diversity in the surgical workforce: a state of the art review. <i>Otolaryngol Head Neck Surg</i>. 2022;166(6):1182-1191.</p>
<p>3. Osei-Kuffour D, Banda CH, Campion A, et al. The evolution of the specialist surgeon workforce in East, Central and Southern Africa. World J Surg. 2025;49(4):946-954. doi:10.1002/wjs.12545</p>
<sup>1</sup><i>East and Central African Journal of Surgery</i>, College of Surgeons of East, Central and Southern Africa; managing_editor@cosecsa.org.</p>
<h4>Conflict of Interest Disclosures </h4>
<p>None reported.</p>


Evolution of Authorship Diversity in African Surgical Research Over 2 Decades

<p>Tais Freire Galvão,<sup>1,2</sup> Everton Nunes da Silva,<sup>2</sup> Jorge Otávio Maia Barreto,<sup>3</sup> Marcus Tolentino Silva<sup>4</sup></p>
<h4>Objective</h4>
<p> Detailed descriptions of journals’ workflows exist,<sup>1</sup> but empirical studies assessing their effects are scarce. In 2024, a new workflow was implemented at <i>Epidemiologia e Serviços de Saúde: revista do SUS</i> (<i>RESS</i>) aimed at getting timely and quality reviews. This study assessed the effect of a new workflow on the journal’s time to decision-making.</p>
<h4>Design</h4>
<p> This before and after study using historical controls assessed submissions to <i>RESS</i> from June to December 2024 (intervention) and June to December 2023 (control), followed up until January 2025. The intervention started in June 2024, based on inclusion of a peer review administrator in charge of reviewers’ assignments and adding new and trained associate editors<sup>2</sup>; previous evaluation by the editor in chief (scope) and scientific editor (methodological quality); and checklists to standardize reviews.<sup>3</sup> The comparator was the previous workflow, centralized on the associate editor’s assignment of reviewers, and final decisions made by the editor in chief and a member of the editorial committee. Outcomes were processing time from submission to decision, number of reviewers invited, and reviews received. We summarized means and SDs or medians and IQRs. Normality was tested using the Shapiro-Wilk test and square root or logarithmic transformations preceded t-tests, with the Levene test used to assess variances’ homogeneity. In unsuitable t-tests, the Mann-Whitney test was applied. Residual normality and heteroscedasticity were examined by Breusch-Pagan test. We adjusted by the length of the paper (full or short) to minimize potential confounding.</p>
<h4>Results</h4>
<p> Out of 928 submissions in 2024, 218 proceeded to peer review (140 ongoing until cohort closure) and 78 had a decision (36 accepted, 42 rejected). In 2023, from 970 submissions, 77 were peer reviewed and resulted in 28 acceptances and 49 rejections (<i>P</i> = .22). Mean (SD) time in days to decision was significantly shorter in the intervention (62.8 [40.5]) than in the controls (130.1 [64.5]; <i>P</i> &lt; .001), which was also faster in the intervention according to acceptance (92.8 [39.8] vs 133.5 [61.6]; <i>P</i> = .002) or rejection (37.1 [16.0] vs 128.1 [66.7]; <i>P</i> &lt; .001). The mean (SD) number of invited reviewers per manuscript was higher in the intervention (10.1 [6.3]) than in the controls (7.8 [5.5]; <i>P</i> = .02), without significant difference according to decision when compared with the historical controls (approved, 10.0 [7.0] vs 6.3 [2.8]; <i>P</i> = .07; rejected, 10.3 [5.8] vs 8.6 [6.5]; adjusted <i>P</i> = .11). The median (IQR) number of reviews received was also higher (4 [4-4] vs 2 [2-3]; <i>P</i> &lt; .001), and was similarly higher when comparing decisions across groups (approved, 4 [4-4] vs 2 [2-3]; <i>P</i> &lt; .001; rejected, 4 [4-4] vs 2 [2-3]; <i>P</i> &lt; .001) (<b>Table 25-0909</b>).</p>

![](https://assets.underline.io/markdown_image/1/image/c5b3c1bf721bf5ccfd8c212b70f07e17.png)

<h4>Conclusions</h4>
<p> The new workflow reduced the duration of editorial processing and allowed a higher number of peer reviews to inform decision-making. The outcomes were not defined before the intervention was planned; the length of follow-up, new editors, and submission system were confounders that may have influenced the findings.</p>
<h4>References</h4>
<p>1. Mark H, Ragon T, Funning G, et al. Editorial workflow of a community-led, all-volunteer scientific journal: lessons from the launch of Seismica. <i>Seismica</i>. 2023;2(2):1091. doi:10.26443/seismica.v2i2.1091</p>
<p><span lang="pt-PT">2. Galvão TF, Silva EN, Araújo WN, Barreto JOM. Editorial improvement in </span><i>Epidemiologia e Serviços de Saúde</i><span lang="pt-PT"> in 2024. </span><i>Epidemiol Serv Saude</i><span lang="pt-PT">. 2024;33:e20241002. doi:</span><a href="https://doi.org/10.1590/S2237-96222024v33e20241002.en"><span class="Hyperlink CharOverride-6" lang="pt-PT">10.1590/S2237-96222024v33e20241002.en</span></a></p>
<p><span lang="pt-PT">3. Silva MT, Galv</span><span lang="ar-SA">ã</span><span lang="pt-PT">o TF. Systematization of peer review in </span><i>Epidemiologia e Serviços de Saúde</i><span lang="pt-PT">. </span><i>Epidemiol Serv Saude</i><span lang="pt-PT">. 2024;33:e20241001. doi:</span><a href="https://doi.org/10.1590/S2237-96222024v33e20241001.en"><span class="Hyperlink CharOverride-6" lang="pt-PT">10.1590/S2237-96222024v33e20241001.en</span></a></p>
<p class="abstract_conflict" lang="fr-FR"><sup>1</sup>Faculdade de Ciências Farmacêuticas<span lang="pt-PT">, Universidade Estadual de Campinas, Campinas, Brazil, taisgalvao@gmail.com; </span><sup>2</sup><span lang="pt-PT">Secretaria de Vigilância em Saúde e Ambiente, Ministério da Saúde</span>, Brasília, Brazil; <sup>3</sup><span lang="pt-PT">Faculdade d</span>e Ciências e Tecnologias em Saúde, Universidade de Brasília, Brasília, Brazil; <sup>4</sup><span lang="pt-PT">Gerência Regional de Brasília, Fundação Oswaldo Cruz, </span>Brasília, Brazil; <sup>5</sup><span lang="pt-PT">Faculdade d</span>e Ciências da Saúde, Universidade de Brasília, Brasília, Brazil.</p>
<h4>Conflict of Interest Disclosures </h4>
<p>Tais Freire Galv<span lang="ar-SA">ã</span>o, Everton Nunes da Silva, and Jorge Otávio Maia Barreto are editors of <i>Epidemiologia e Serviços de Saúde: revista do SUS</i>. Marcus Tolentino Silva has no conflicts of interest to declare. Tais Freire Galv<span lang="ar-SA">ã</span>o is a member of the Peer Review Congress Advisory Board but was not involved in the review or decision for this abstract.</p>


Effects of Peer Review and Editorial Workflows in Decision-Making at a Diamond Open Access Journal

<p>Hillary Wnfried Ramirez,<sup>1,2,3</sup> Malena Chiaborelli,<sup>1,2,3</sup> Christof M. Schönenberger,<sup>1</sup> Katie Mellor,<sup>4,5</sup> Alexandra N. Griessbach,<sup>1</sup> Paula Dhiman,<sup>4,6</sup> Pooja Gandhi,<sup>7</sup> Szimonetta Lohner,<sup>8,9</sup> Arnav Agarwal,<sup>10,11</sup> Ayodele Odutayo,<sup>5,12</sup> Michael M. Schlussel,<sup>4,6</sup> Philippe Ravaud,<sup>13,14</sup> David Moher,<sup>15,16 </sup>Matthias Briel,<sup>1,10</sup> Isabelle Boutron,<sup>13,14</sup> Sally Hopewell,<sup>4</sup> Sara Schroter,<sup>17,18 </sup>Benjamin Speich<sup>1,4</sup></p>
<h4>Objective </h4>
<p>Two randomized controlled trials (RCTs) conducted at journal level have shown that reminding peer reviewers about the 10 most important and underreported reporting items did not improve the reporting quality in published articles.<sup>1</sup> With this pooled in-depth analysis of peer reviewer reports, we aimed to assess at what stage the intervention failed.</p>
<h4>Design</h4>
<p> A subsample of peer reviewer reports from the control group (receiving no reminder) and the intervention group (receiving a reminder of the 10 most important reporting items) were analyzed. In brief, 2 blinded authors independently extracted from peer reviewer reports how many of the 10 key reporting items were flagged by peer reviewers for clarification. The main outcome of this analysis was the mean proportion of the 10 selected reporting items for which at least 1 peer reviewer requested clarification, assessed at the manuscript level. Furthermore, we assessed how many requested changes were later adequately reported in published articles.</p>
<h4>Results </h4>
<p>Across the RCTs, we had access to peer reviewer reports for 533 manuscripts (265 in the intervention group, assessing comments from 740 peer reviewers; and 268 in the control group, assessing comments from 719 peer reviewers). Our results indicate that reviewers in the intervention group requested clarification on more reporting items than those in the control group. Overall, reviewers in the intervention group flagged 21.1% of the 10 reporting items for clarification compared with 13.1% in the control group (mean difference, 8.0 percentage points (pp); 95% CI, 4.9-11.1 pp). However, the overall mean difference between groups was diluted from 8.0 to 4.2 pp when only assessing accepted and published articles and decreased even further to 2.6 pp when only considering changes that were then implemented by authors of manuscripts (<b>Table 25-0956</b>). Approximately 55% of reporting items that were criticized by peer reviewers were later adequately reported in the published article (intervention group, 173 of 308 [56.2%]; control group, 105 of 197 [53.3%]).</p>

![](https://assets.underline.io/markdown_image/1/image/80fe50628cb162743e45faadd6604b8b.png)

<h4>Conclusions</h4>
<p> Reminding peer reviewers to check reporting items increased their focus on reporting guidelines, leading to more reporting-related requests in their review reports. However, the effect was diluted during the peer review process (particularly due to rejected articles and requests not being implemented by authors). Journals should therefore make sure that requested clarifications are adequately addressed in revised manuscripts.</p>
<h4>Reference</h4>
<p>1. Speich B, Mann E, Schönenberger CM, et al. Reminding peer reviewers of reporting guideline items to improve completeness in published articles: primary results of 2 randomized trials. <i>JAMA Netw Open</i>. 2023;6(6):e2317651.</p>
<sup>1</sup>CLEAR Methods Center, Division of Clinical Epidemiology, Department Clinical Research, University Hospital Basel, University of Basel, Basel, Switzerland, benjamin.speich@usb.ch; <sup>2</sup>Swiss Tropical and Public Health Institute, Basel, Switzerland; <sup>3</sup>University of Basel, Basel, Switzerland; <sup>4</sup>Centre for Statistics in Medicine, Nuffield Department of Orthopaedics, Rheumatology and Musculoskeletal Sciences, University of Oxford, Oxford, UK; <sup>5</sup>Clinical Outcomes Assessment, Clarivate, London, UK; <sup>6</sup>The EQUATOR Network, Oxford, UK; <sup>7</sup>Department of Communication Sciences and Disorders, Faculty of Rehabilitation Medicine, University of Alberta, Edmonton, AL, Canada; <sup>8</sup>Cochrane Hungary, Medical School, University of Pécs, Pécs, Hungary; <sup>9</sup>MTA–PTE Lendület “Momentum” Evidence in Medicine Research Group, Department of Public Health Medicine, Medical School, University of Pécs, Pécs, Hungary; <sup>10</sup>Department of Health Research Methods, Evidence, and Impact, McMaster University, Hamilton, ON, Canada; <sup>11</sup>Division of General Internal Medicine, Department of Medicine, McMaster University, Hamilton, ON, Canada; <sup>12</sup>Division of Nephrology, Toronto General Hospital, University Health Network, Toronto, ON, Canada; <sup>13</sup>Centre d’Épidémiologie Clinique, Hôpital Hôtel-Dieu, Assistance Publique Hôpitaux de Paris, Paris, France;<sup>14</sup>Université de Paris, CRESS, Inserm, INRA, Paris, France; <sup>15</sup>Centre for Journalology, Clinical Epidemiology Program, Ottawa Hospital Research Institute, Ottawa, Ontario, Canada; <sup>16</sup>Faculty of Medicine, School of Epidemiology and Public Health, University of Ottawa, Ottawa, Ontario, Canada; <sup>17</sup><i>The BMJ</i>, London, UK; <sup>18</sup>Faculty of Public Health &amp; Policy, London School of Hygiene &amp; Tropical Medicine, London, UK.</p>
<h4>Conflict of Interest Disclosures</h4>
<p> Benjamin Speich and Matthias Briel reported receiving unrestricted grants from Moderna for studies unrelated to the presented work. Sara Schroter is employed by BMJ Publishing Group. Katie Mellor is employed by Clarivate. David Moher, Sally Hopewell, and Isabelle Boutron are members of the Consolidated Standards for Reporting Trials (CONSORT) executive board and authors of the CONSORT 2010 statement. David Moher is an author of the Standard Protocol Items: Recommendations for Interventional Trials (SPIRIT) 2013 statement. David Moher, Michael M. Schlussel, Paula Dhiman, and Philippe Ravaud are members of the Enhancing the Quality and Transparency of Research (EQUATOR) network. Isabelle Boutron and David Moher are members of the Peer Review Congress Advisory Board but were not involved in the review or decision for this abstract. No other disclosures were reported.</p>
<h4>Funding/Support </h4>
<p>Benjamin Speich was supported by a Return Postdoc.Mobility (P4P4PM_194496) grant from the Swiss National Science Foundation. Christof M. Schönenberger was funded by the Janggen Pöhn Foundation and the Swiss National Science Foundation (MD-PhD grant No. 323530_221860). Szimonetta Lohner was supported by the Hungarian Academy of Sciences (MTA) within the framework of the Lendület Programme.</p>
<h4>Role of the Funder/Sponsor</h4>
<p> The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.</p>
<h4>Additional Information</h4>
<p> Both trials (CONSORT-PR and SPIRIT-PR) were prospectively registered on Open Science Framework (<a href="https://osf.io/c4hn8"><span class="Hyperlink CharOverride-6">https://osf.io/c4hn8</span></a> and <a href="https://osf.io/z2hm9"><span class="Hyperlink CharOverride-6">https://osf.io/z2hm9</span></a>).</p>


Reminding Peer Reviewers to Comment on Reporting Items as Instructed by the Journal? An Analysis of 2 Randomized Trials

<p>Mario Malički,<sup>1,2,3</sup> Ana Jerončić,<sup>4</sup> Gerben ter Riet,<sup>5,6</sup> Lex Bouter,<sup>7,8</sup> John P. A. Ioannidis,<sup>2,3,9,10</sup> IJsbrand Jan Aalbersberg,<sup>11 </sup>Steven N. Goodman<sup>1,2,3,9,12</sup> </p>
<h4>Objective </h4>
<p>Previous research has indicated a knowledge gap on changes brought on by peer review and journal publishing.<sup>1</sup> We are conducting a living evidence synthesis of studies that analyzed differences between manuscript versions (eg, preprinted or submitted versions vs peer reviewed journal versions).</p>
<h4>Design </h4>
<p>This is a living synthesis of studies with meta-analyses of proportions (following PRISMA guidelines for reporting).<sup>2</sup> Studies were identified based on authors’ knowledge of the field, from 18 previous systematic reviews on peer review,<sup>3</sup> and by checking all research presented at the peer review conferences. All references and citations of identified studies (using Google Scholar up to January 2025) were checked. For all included studies that reported on changes, we extracted descriptive variables (including year of publication, sampling method, change comparison method, and data, wherever available, on how many analyzed version-pairs had changes for 11 outcomes). We assessed risk of bias for each outcome based on 3 domains: sample selection, measurement reliability, and conflicts of interest. To pool the data, we used random effects meta-analysis with Freeman-Tukey transformed proportions and reported the <i>I</i><sup>2</sup> index of heterogeneity.</p>
<h4>Results </h4>
<p>We identified 67 studies published from 1978 through the end of 2024, of which 31 (46%) analyzed changes between preprint and journal versions, 23 (34%) between submitted and accepted or published versions, 8 (12%) between rejected versions and those later published in other journals, 4 (6%) between multiple version sets, and 1 (1.5%) between different preprinted versions. Most studies, 45 (67%), analyzed changes manually, 13 (19%) used computational methods, and 9 (12%) combined manual and computational methods. The median number of analyzed version-pairs was 113 (IQR, 51-429). By discipline, 43 (64%) studies looked only at health research, 6 (9%) at life sciences, 6 (9%) at social sciences, 4 (6%) at physical sciences, and 8 (12%) at multiple disciplines. Meta-analyses showed the highest frequency of changes in title, authorship, conflicts of interest, and numerical results, with the lowest frequency in study conclusions (<b>Table 25-1077</b>).</p>

![](https://assets.underline.io/markdown_image/1/image/e3c3d82a0d010bd96adfaadbf9984c49.png)

<h4>Conclusions </h4>
<p>Current evidence indicates that while different manuscript sections experience varying degrees of changes from their submitted, preprinted, or rejected versions to their peer-reviewed journal version, the primary conclusions rarely change. Our results in this stage are limited by not conducting a systematic search of bibliographic databases and lack of (identified) data points from many subdisciplines, especially arts and humanities, as well as studies that explored possible factors associated with the extent of changes (eg, researchers’ or editors’ career stage or prestige, quality of initial manuscript version).</p>
<h4>References</h4>
<p>1. Tennant JP, Ross-Hellauer T. The limitations to our understanding of peer review. <i>Res Integr Peer Rev</i>. 2020;5:6. doi:10.1186/s41073-020-00092-1</p>
<p>2. Page MJ, McKenzie JE, Bossuyt PM, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. <i>BMJ</i>. 2021;372:n71. doi:10.1136/bmj.n71</p>
<p>3. European Association of Science Editors. What is peer review? December 28, 2024. Accessed July 15, 2025. <a href="https://ease.org.uk/communities/peer-review-committee/peer-review-toolkit/What-is-peer-review/"><span class="Hyperlink CharOverride-6">https://ease.org.uk/communities/peer-review-committee/peer-review-toolkit/What-is-peer-review/</span></a></p>
<sup>1</sup>Stanford Program on Research Rigor and Reproducibility (SPORR), Stanford University, Stanford, California, US, mmalicki@stanford.edu; <sup>2</sup>Department of Epidemiology and Population Health, Stanford University School of Medicine, Stanford, California, US; <sup>3</sup>Meta-Research Innovation Center at Stanford (METRICS), Stanford University, Stanford, CA, US; <sup>4</sup>Department of Research in Biomedicine and Health, University of Split School of Medicine, Split, Croatia; <a id="_idTextAnchor069"></a><sup>5</sup>Urban Vitality Centre of Expertise, Amsterdam University of Applied Sciences, Amsterdam, The Netherlands; <sup>6</sup>Amsterdam University Medical Centers, Department of Cardiology, Amsterdam, The Netherlands; <sup>7</sup>Department of Philosophy, Faculty of Humanities, Vrije Universiteit, Amsterdam, The Netherlands; <sup>8</sup>Amsterdam University Medical Centers, Department of Epidemiology and Data Science, Amsterdam, The Netherlands; <a id="_idTextAnchor070"></a><sup>9</sup>Department of Medicine, Stanford University School of Medicine, Stanford, California, US<a id="_idTextAnchor071"></a>; <sup>10</sup>Department of Biomedical Data Science, Stanford University School of Medicine, Stanford, California, US; <sup>11</sup>Independent scholar; <sup>12</sup>Department of Health Policy, Stanford University School of Medicine, Stanford, California, US.</p>
<p class="abstract_conflict" lang="fr-FR"><span class="text-bold">Conflict of Interest Disclosures </span>Mario Malički was a co–editor in chief of Research Integrity and Peer Review, in which 1 of the included studies was published (he was not involved in the processing or decision making for that study, and Research Integrity and Peer Review full peer review histories are available to the public on the journal website). Mario Malički was also an external reviewer for 1 study. Gerben ter Riet is a co-author of 1 of the included studies and Steven N. Goodman is a co-author of 1 of the included studies. In both of these cases, data extractions and primary data analyses were done by Mario Malički and Ana Jerončić. IJsbrand Jan Aalbersberg was a Senior Vice President of Research Integrity at Elsevier. John P. A. Ioannidis, Lex Bouter, and Steven N. Goodman are members of the Peer Review Congress Advisory Board but were not involved in the review or decision for this abstract.</p>
<h4>Funding/Support</h4>
<p> The work on this project started in 2020, and in 2020 Elsevier funding was awarded to Stanford University for a 1-year METRICS postdoctoral position that supported Dr Malički’s work. From 2021 to 2025, Dr Malički’s work has been supported by the Stanford School of Medicine Research Office, which also supports the work of Dr Goodman. At the start of the project, IJsbrand Jan Aalbersberg was an employee of Elsevier.</p>
<h4>Acknowledgments </h4>
<p>In 2022’s Peer Review Congress, we presented preliminary findings (<a href="https://peerreviewcongress.org/abstract/a-synthesis-of-studies-on-changes-manuscripts-underwent-between-submission-or-preprint-posting-and-peer-reviewed-journal-publication/"><span class="Hyperlink CharOverride-6">https://peerreviewcongress.org/abstract/a-synthesis-of-studies-on-changes-manuscripts-underwent-between-submission-or-preprint-posting-and-peer-reviewed-journal-publication/</span></a>) that covered 25 studies published between 1990 and 2021. The current results include studies published before 1990 and up to January 2025.</p>
<h4>Additional Information</h4>
<p> A website with computationally reproducible results from this living review will be available soon.</p>


Differences Between Manuscript Versions: A Living Review and Series of Meta-Analyses

<p>Melecia Miller,<sup>1</sup> <span lang="it-IT">Vera Nezgovorova,</span><sup>2</sup> Ilana Kersh,<sup>3</sup> Mohamed I. Elsaid,<sup>4</sup> Marina Broitman<sup>2</sup></p>
<h4>Objective</h4>
<p> This study’s goal was to develop a way to assess patient review quality based on author and editor reactions to reviews. We used patient reviews of draft final research reports (DFRRs) of Patient Centered Outcomes Research Institute (PCORI)–funded research that were peer reviewed by clinical, statistical, and patient reviewers.<sup>1</sup> Review quality ratings were based on prior research on PCORI peer review and other research on patient reviewers.<sup>2,3</sup> We hypothesized that specific, substantive changes to the report and patient-focused comments would be associated with a favorable mention of the patient review by the editor as well as a substantive response from the author.</p>
<h4>Design</h4>
<p> Pilot testing on 13 patient reviews for 13 clinical research DFRRs submitted from 2021 to 2022 was completed in 2025. We conducted a content analysis of patient reviewer comments and developed a coding scheme based on prior research. Coding was performed at the sentence level and included 2 categories: (1) specificity and constructiveness of the comment and (2) patient focus of the comment (<b>Table 25-1110</b>). Each review was coded independently by 2 raters each for category 1 and category 2, with a final Cohen κ of 0.83 and 0.66, respectively, indicating moderate to substantial interrater agreement. Coding discrepancies between reviewers received the most conservative (ie, lowest value) code. Raters assessed whether the editor mentioned the patient review at least once in their review summary and either encouraged (code 1) or discouraged (code 2) an author response vs no mention (code 0). Author responses to reviewer comments were coded by paragraph indicating whether authors responded directly to patient-reviewer comments (code 1) or made changes to the report because of the comments (code 2) vs neither (code 0). We used hierarchical modeling to calculate the odds of editors encouraging authors to respond (code 1 vs codes 0 or 2) and mixed-effects logistic regression to calculate the odds of authors responding substantively (codes 1 or 2 vs code 0) to patient reviews with more specific, constructive comments.</p>
<h4>Results</h4>
<p> <b>Table 25-1110</b> shows counts for associations between category 1 and category 2. More specific and constructive comments and patient-focused comments were not associated with higher odds of the editor mentioning the patient review (N = 13; odds ratio [OR], 0.36; 95% CI, 0.28-0.45 and OR, 0.65; 95% CI 0.54-0.78, respectively). Patient review paragraphs with more than 50% specific and constructive comments were associated with higher odds of authors responding substantively, compared with other comment types (N = 147; OR, 4.00; 95% CI, 1.67-9.59; <i>P</i> = .002).</p>
<a href="/wp-content/uploads/2025_conference/tables/prc25-1110t1.png"> <img loading="lazy" src="/wp-content/uploads/2025_conference/tables/prc25-1110t1.png"> </a>
<h4>Conclusions</h4>
<p> This pilot study shows the feasibility of establishing codes rating the quality of patient reviews and determining whether higher quality patient reviews are associated with mentions by the editor and author responses. The results are limited because of the sample size and the uniqueness of DFRRs.</p>
<h4>References</h4>
<p>1. Broitman M, Sox HC, Slutsky J. A model for public access to trustworthy and comprehensive reporting of research. <i>JAMA</i>. 2019;321(15):1453-1454. doi:10.1001/jama.2019.2807</p>
<p>2. Ghazaryan E, Broitman M, Sox H. Differences in the style and quantity of reviewer comments in structured vs unstructured peer review forms. Poster presented at: Ninth International Congress on Peer Review and Scientific Publication; September 10, 2022; Chicago, IL.</p>
<p>3. Huml AM, Albert JM, Beltran JM, et al. Community members as reviewers of medical journal manuscripts: a randomized controlled trial. <i>J Gen Intern Med</i>. 2022;38(6):1393-401. doi:10.1007/s11606-022-07802-z</p>
<sup>1</sup>KnowledgeWorks Global, Ltd., 410 MN-25, Brainerd, MN, US; <span lang="it-IT">mmiller@originreview.org; </span><sup>2</sup>Patient-Centered Outcomes Research Institute, Washington, DC, US; <sup>3</sup>University of Pittsburgh, Pittsburgh, PA, US; <sup>4</sup>The Ohio State University College of Medicine, Columbus, Ohio, US.</p>
<h4>Conflict of Interest Disclosures </h4>
<p>None reported.</p>


Quality of Patient Reviewer Comments and Association With Author and Editor Responses

<p>Matthew B. Stanbrook,<sup>1</sup> Shannon Charlebois,<sup>1</sup> George A. Tomlinson,<sup>1</sup> Meredith Weinhold<sup>1</sup></p>
<h4>Objective</h4>
<p> The opportunity to appeal a journal’s decision to reject a manuscript enhances procedural fairness toward authors and allows editors to correct errors in judgment but wastes time for both authors and editors when unsuccessful. We sought to identify factors associated with outcomes of appeals of research manuscripts at the <i>Canadian Medical Association Journal</i> (<i>CMAJ</i>), a general medical journal that explicitly highlights the option to appeal in rejection letters.</p>
<h4>Design</h4>
<p> We conducted a retrospective cohort study of all appeal requests for manuscripts initially rejected after submission to the Research section of <i>CMAJ</i> between 2007 and 2024. We obtained data on manuscript characteristics and editorial decisions from <i>CMAJ</i>’s manuscript submission database (ScholarOne). Three editors independently reviewed the text of the initial manuscript decision letters to identify reasons for rejection, with disagreements resolved by consensus. We included manuscripts rejected at any stage of the editorial process. The primary outcome was whether appeal requests were granted or denied. A secondary outcome was whether appealed manuscripts were ultimately published. We estimated associations of multiple prespecified characteristics with appeal outcomes using bayesian logistic regression.</p>
<h4>Results</h4>
<p> Among 14,351 submitted manuscripts, 13,419 were rejected, and 616 of these rejections (4.6%) were appealed. Among appeals, 214 (34.7%) were granted, of which 58 (27.1% of granted appeals and 9.4% of all appeals) were ultimately published. Compared with manuscripts rejected without external peer review, granted appeals were more likely for manuscripts rejected after peer review without editorial discussion (adjusted odds ratio [AOR], 2.71; 95% credible interval [CrI], 1.40-5.27) but similarly likely for manuscripts rejected after peer review and editorial discussion (AOR, 1.19; 95% CrI, 0.74-1.91) (<b>Table 25-1161</b>). Granted appeals were more likely if rejection letters included positive comments (AOR, 1.82; 95% CrI, 1.21-2.76) or cited lack of clarity (AOR, 1.84; 95% CrI, 1.12-3.04) and less likely if corresponding authors were based outside Canada (AOR, 0.38; 95% CrI, 0.24-0.59). The odds of an appeal being granted also varied significantly based on the individual editor making the appeal decision (median AOR, 1.90; 95% CrI, 1.24-4.37). Among factors associated with appeals being granted, only positive comments (AOR, 1.98; 95% CrI, 1.10-3.59) were also significantly associated with successful publication after appeal. However, appealed manuscripts rejected after peer review and editorial discussion were more likely to be published than were those rejected without peer review (AOR, 4.93; 95% CrI, 2.47-10.25).</p>

![](https://assets.underline.io/markdown_image/1/image/ca4a766cb379b500fce70628c15c12cb.png)

<h4>Conclusions</h4>
<p> Characteristics of authors, editors, editorial process, and decision communication were associated with successful appeals of initially rejected research manuscripts. Lack of awareness of the small fraction of appeals that ultimately achieve publication may raise false expectations among authors.</p>
<sup>1</sup><i>Canadian Medical Association Journal </i>(<i>CMAJ</i>), Toronto, Ontario, Canada, matthew.stanbrook@cmaj.ca.</p>
<h4>Conflict of Interest Disclosures</h4>
<p> All authors are employees or paid consultants of CMAJ. Matthew B. Stanbrook, Shannon Charlebois, and George A. Tomlinson participated in editorial decision making for some of the manuscripts included in the study. No other disclosures were reported.</p>


Factors Associated With Outcomes of Appeals of Manuscripts Initially Rejected by a General Medical Journal

<p>Daniel S. Evanko,<sup>1</sup> Deondre Jordan<sup>1</sup></p>
<h4>Objective </h4>
<p>Journals ask authors to comply with many reporting requirements to improve study transparency, reproducibility, and reuse. Manual monitoring and enforcement of requirements is difficult, and reporting checklists or vendor-based solutions can be untargeted, burdensome, expensive, or of limited effectiveness. For example, since June 2020 the American Association for Cancer Research has used a methods review tool (SciScore) to help improve author compliance with reporting requirements at its 10 journals. This tool greatly increased author assignment of research resource identifiers (RRIDs),<sup>1</sup> but compliance plateaued at approximately 30% of original research articles published in the association journals. This cross-sectional study describes a simple, inexpensive, automated approach to further overcome limitations of common approaches and quantitatively evaluates its performance using 3 example interventions.</p>
<h4>Design </h4>
<p>Custom Python scripts coupled with structured language queries were used for targeted extraction of manuscript data from the EJournal Press journal submission system. For RRID reporting compliance, once every day the submission system was queried for manuscripts that completed initial quality control checks and did not contain the text RRID. A templated e-mail alert was sent to the corresponding author that explained what RRIDs are, why they are important, and how to add them. The system was also adapted with a custom supervised machine-learning method to identify studies that generated new sequencing data or that were computationally intensive to encourage authors to deposit sequence data and provide computer code used in their study and used to send different alerts for these manuscripts at different designated points in the editorial process.</p>
<h4>Results </h4>
<p>The new automated RRID process was activated August 6, 2024, at all the association journals, resulting in missing-RRID alerts sent for 84% (4949 of 5899) of manuscripts at completion of initial quality control checks through the end of 2024. Performance was assessed by searching for “RRID” in published articles as done previously.<sup>1</sup> An immediate sharp increase in articles containing RRIDs was observed that quickly increased the percentage of published research articles containing RRIDs from 33% (176 of 526) in April through July to 65% (330 of 505) in November through February <b>(Figure 25-1181</b>). Targeted alerts to authors of manuscripts asking them to deposit sequencing data (sent after peer review started) or supply code in a computational science software (Code Ocean) capsule (sent after a revise decision was sent to the author) were turned on on August 26, 2024, and resulted in sequence deposition alerts sent for 43% (1416 of 3261) of manuscripts and Code Ocean alerts sent for 39% (614 of 1594) of manuscripts through April 30, 2025. The sequencing alert triggered more than 100 author e-mail communications about deposition, demonstrating author engagement with this request and allowing resolution early in the editorial process. The percentage of accepted manuscripts with linked Code Ocean capsules increased from 1.4% (21 of 1555) in 2024 to 4.6% (25 of 547) in the January to May 2025 period, a more than 3-fold increase.</p>

![](https://assets.underline.io/markdown_image/1/image/87ac32761435115531ebf1a5f6ddec09.png)

<h4>Conclusions </h4>
<p>This fully automatic and highly targeted process engages with authors in a context-dependent way at predefined times in the editorial process that are appropriate for the specific communication, thus achieving substantial author engagement and improved compliance with journal requirements.</p>
<h4>Reference</h4>
<p>1. Roelandse M, Ozykurt IB, Evanko D, Bandrowski A. Assessing the effectiveness of SciScore in supporting the reproducibility of scientific research. <i>Sci Ed</i>. 2023;46:46-52.</p>
<sup>1</sup>American Association for Cancer Research, Philadelphia, PA, US, daniel.evanko@aacr.org.</p>
<h4>Conflict of Interest Disclosures</h4>
<p> Daniel S. Evanko is a member of the Peer Review Congress Advisory Board but was not involved in the review or decision for this abstract.</p>


Next from The 10th Peer Review Congress

Geographical Representation of Author Country among Peer Reviewers and Publishing Success at 60 STEM Journals

Similar lecture

Peer Reviews of Peer Reviews: A Randomized Controlled Trial and Other Assessments