United States

&lt;p&gt;Laura Caquelin,&lt;sup&gt;1&lt;/sup&gt; Rachel Heyard,&lt;sup&gt;2&lt;/sup&gt; Stephanie Zellers,&lt;sup&gt;3&lt;/sup&gt; Hanno Würbel,&lt;sup&gt;4&lt;/sup&gt; Gustav Nilsonne&lt;sup&gt;1&lt;/sup&gt;&lt;/p&gt;
&lt;h4&gt;Objective&lt;/h4&gt;
&lt;p&gt; Computational reproducibility is vital to research quality, enabling others to replicate analyses and achieve the same results. Despite its importance, it is rarely assessed systematically. This study aims to evaluate whether computational reproducibility review during the publication process improves reproducibility compared with standard peer review. We hypothesize that this intervention will improve reproducibility, encourage code sharing, and reduce errors.&lt;/p&gt;
&lt;h4&gt;Design&lt;/h4&gt;
&lt;p&gt; This randomized controlled trial was designed to enroll manuscripts submitted to partnering journals that meet inclusion criteria, including open data availability and inferential analysis. Manuscripts will be randomized 1:1 to peer review with computational reproducibility review (intervention group) or standard peer review without intervention (control group). The computational reproducibility review involves reproducing the essential statistical results using shared data (with or without the code). Feedback is provided to authors in the intervention group during the peer review process. The reproducibility of the manuscripts randomized to the control group will be assessed after publication only. The primary outcome is the proportion of manuscripts for which we successfully reproduced the essential statistical results after publication. Secondary outcomes include rates of code sharing, overt errors identified, time required for the review, publication timelines, and categorized reproducibility issues.&lt;/p&gt;
&lt;h4&gt;Results&lt;/h4&gt;
&lt;p&gt; This study began in January 2025, with a preregistered protocol openly available on the Open Science Framework.&lt;sup&gt;1&lt;/sup&gt; Recruitment is ongoing, and as of April 2025, 28 manuscripts submitted to the partnering journal GigaSicence have been screened, with 17 deemed eligible. The project is scheduled to run through September 2026, with the goal of including 200 manuscripts. This presentation will outline key insights gained during the study’s implementation and preliminary results.&lt;/p&gt;
&lt;h4&gt;Conclusions&lt;/h4&gt;
&lt;p&gt; This study fills an important gap in assessing research quality by testing a practical way to improve computational reproducibility. If successful, the results could support wider use of reproducibility reviews in academic journals, leading to more reliable and trustworthy science.&lt;/p&gt;
&lt;h4&gt;Reference&lt;/h4&gt;
&lt;p&gt;1. Caquelin L, Heyard R, Zellers S, et al. A study protocol for an intervention study using computational reproducibility testing to improve reproducibility. Open Science Framework. Updated January 29, 2025. doi:10.17605/OSF.IO/3Y8FP&lt;/p&gt;
&lt;sup&gt;1&lt;/sup&gt;Karolinska Institutet, Stockholm, Sweden, laura.caquelin@ki.se; &lt;sup&gt;2&lt;/sup&gt;Center for Reproducible Science at the Epidemiology, Biostatistics and Prevention Institute, University of Zurich, Zurich, Switzerland; &lt;sup&gt;3&lt;/sup&gt;Institute for Molecular Medicine Finland, University of Helsinki, Helsinki, Finland; &lt;sup&gt;4&lt;/sup&gt;Division of Animal Welfare, University of Bern, Bern, Switzerland.&lt;/p&gt;
&lt;h4&gt;Conflict of Interest Disclosures&lt;/h4&gt;
&lt;p&gt; None reported.&lt;/p&gt;
&lt;h4&gt;Funding/Support&lt;/h4&gt;
&lt;p&gt; This study is part of the iRISE (Improving Reproducibility in Science) project, which has received funding from the European Union under the Horizon Europe program (grant agreement 101094853).&lt;/p&gt;
&lt;h4&gt;Role of the Funder/Sponsor&lt;/h4&gt;
&lt;p&gt; The funder had no role in the data collection, analysis, or conclusions of the abstract.&lt;/p&gt;


The 10th Peer Review Congress

[V] Testing Computational Reproducibility Review in Editorial Workflows of Academic Journals: A Randomized Controlled Trial from the European iRISE project

peer review process and models

quality of the literature

reproducibility

The Tenth Peer Review Congress featured 3 days for presentations of new research into peer review and all aspects of scientific publication, bias, quality of reporting, and information access and dissemination.

This year’s meeting was hybrid, and all plenary sessions were livestreamed with opportunities to view presentations and engage with participants in person and virtually during and after the meeting. All recordings are now available on Underline for viewing.

<div style="position:relative;padding-top:max(60%,324px);width:100%;height:0;"><iframe style="position:absolute;border:none;width:100%;height:100%;left:0;top:0;" src="https://online.fliphtml5.com/ebtyf/mget/"  seamless="seamless" scrolling="no" frameborder="0" allowtransparency="true" allowfullscreen="true" ></iframe></div>


#### Why? 
Our aim is to encourage research into the quality and credibility of peer review and scientific publication, to establish the evidence base on which scientists can improve the conduct, reporting, and dissemination of scientific research.

#### Who? 
Participants include editors and publishers of scientific peer-reviewed journals, researchers, funders, bibliometric and informatics experts, information innovators, librarians, journalists, policymakers, ethicists, scientific information disseminators, and anyone interested in the progress of the scientific information enterprise and the quality of scientific evidence.

The Congress embraces a wide range of disciplines, including biomedicine, health and life sciences, applied sciences, basic sciences, physical and chemical sciences, mathematics, computer sciences, engineering, economics, and social sciences as well as new and emerging disciplines.

#### What? 
The Congress provides a forum for the presentation and discussion of new research into the quality and credibility of editorial and funding peer review, scientific and scholarly publication, research methods and reporting practices, identification and management of biases, transparency, reproducibility, access, publication and dissemination metrics, identification and management of threats to the integrity of science, and advances in information exchange.

**Follow us**

[![](https://assets.underline.io/markdown_image/1/image/662a22766b9788ed3766cb23cd156e32.png)](https://x.com/peerrevcongress)  [![](https://assets.underline.io/markdown_image/1/image/bb105afb1773a01fa169aa97ca1af49a.png)](https://www.youtube.com/channel/UCsG_0nKi3_Zfyv0yRPdxjWQ)  [![](https://assets.underline.io/markdown_image/1/image/034e1e70a194a9a4178fdf470ef796db.png)](https://bsky.app/profile/peerreviewcongress.bsky.social)



You need to log in with the email address you registered with. 



Please log in to explore this event.

Registration is required to attend this event, please [here](https://peerreviewcongress.org/registration-information/) to secure your ticket. Please check your spam/promotional inbox too.



Please register!

The Tenth Peer Review Congress features 3 days for presentations of new research into peer review and all aspects of scientific publication, bias, quality of reporting, and information access and dissemination.

<p>Laura Caquelin,<sup>1</sup> Rachel Heyard,<sup>2</sup> Stephanie Zellers,<sup>3</sup> Hanno Würbel,<sup>4</sup> Gustav Nilsonne<sup>1</sup></p>
<h4>Objective</h4>
<p> Computational reproducibility is vital to research quality, enabling others to replicate analyses and achieve the same results. Despite its importance, it is rarely assessed systematically. This study aims to evaluate whether computational reproducibility review during the publication process improves reproducibility compared with standard peer review. We hypothesize that this intervention will improve reproducibility, encourage code sharing, and reduce errors.</p>
<h4>Design</h4>
<p> This randomized controlled trial was designed to enroll manuscripts submitted to partnering journals that meet inclusion criteria, including open data availability and inferential analysis. Manuscripts will be randomized 1:1 to peer review with computational reproducibility review (intervention group) or standard peer review without intervention (control group). The computational reproducibility review involves reproducing the essential statistical results using shared data (with or without the code). Feedback is provided to authors in the intervention group during the peer review process. The reproducibility of the manuscripts randomized to the control group will be assessed after publication only. The primary outcome is the proportion of manuscripts for which we successfully reproduced the essential statistical results after publication. Secondary outcomes include rates of code sharing, overt errors identified, time required for the review, publication timelines, and categorized reproducibility issues.</p>
<h4>Results</h4>
<p> This study began in January 2025, with a preregistered protocol openly available on the Open Science Framework.<sup>1</sup> Recruitment is ongoing, and as of April 2025, 28 manuscripts submitted to the partnering journal GigaSicence have been screened, with 17 deemed eligible. The project is scheduled to run through September 2026, with the goal of including 200 manuscripts. This presentation will outline key insights gained during the study’s implementation and preliminary results.</p>
<h4>Conclusions</h4>
<p> This study fills an important gap in assessing research quality by testing a practical way to improve computational reproducibility. If successful, the results could support wider use of reproducibility reviews in academic journals, leading to more reliable and trustworthy science.</p>
<h4>Reference</h4>
<p>1. Caquelin L, Heyard R, Zellers S, et al. A study protocol for an intervention study using computational reproducibility testing to improve reproducibility. Open Science Framework. Updated January 29, 2025. doi:10.17605/OSF.IO/3Y8FP</p>
<sup>1</sup>Karolinska Institutet, Stockholm, Sweden, laura.caquelin@ki.se; <sup>2</sup>Center for Reproducible Science at the Epidemiology, Biostatistics and Prevention Institute, University of Zurich, Zurich, Switzerland; <sup>3</sup>Institute for Molecular Medicine Finland, University of Helsinki, Helsinki, Finland; <sup>4</sup>Division of Animal Welfare, University of Bern, Bern, Switzerland.</p>
<h4>Conflict of Interest Disclosures</h4>
<p> None reported.</p>
<h4>Funding/Support</h4>
<p> This study is part of the iRISE (Improving Reproducibility in Science) project, which has received funding from the European Union under the Horizon Europe program (grant agreement 101094853).</p>
<h4>Role of the Funder/Sponsor</h4>
<p> The funder had no role in the data collection, analysis, or conclusions of the abstract.</p>


poster

<p>Griffin Hughes,<sup>1</sup> Andrew Tran,<sup>1</sup> Sydney Marouk,<sup>1</sup> Eli Paul,<sup>1</sup> Matt Vassar<sup>1,2</sup></p>
<h4>Objective</h4>
<p> The overinterpretation of research findings—otherwise known as “spin”—is commonly assessed in clinical trials and synthesis designs of interventional evidence.<sup>1,2</sup> However, spin in diagnostic evidence has received less attention and has fewer published works.<sup>3</sup> Therefore, we conducted a systematic review aimed to assess spin in pathology’s diagnostic accuracy meta-analyses.</p>
<h4>Design </h4>
<p>Using systematic review methodology, we searched PubMed (MEDLINE) on September 21, 2023, for diagnostic test accuracy meta-analyses published in top 20 pathology journals. To be included for data extraction, studies must have (1) been a systematic review with meta-analysis of primary studies identified through a literature search, (2) pooled diagnostic accuracy effects for a quantitative summary effect estimation, and (3) been published in a top 20 pathology journal, defined by Google Scholar Metrics’ h5-index. We excluded studies that (1) did not represent systematic reviews with meta-analyses; (2) did not quantitatively pool diagnostic accuracy measures; (3) were primarily prognostic in their aims; (4) were published before the publication of the Quality Assessment of Diagnostic Accuracy Studies quality tool;<sup> </sup>or (5) represented publication types such as erratum, retractions, or corrigendum. We derived 10 items of actual spin and 9 items of potential spin from previously published methodology assessing overinterpretation of results in quantitative diagnostic synthesis.<sup>3</sup> Studies with conclusions interpreted as positive were assessed for actual spin, whereas all studies were assessed for potential spin. Authors screened and extracted relevant data from sample studies in a masked duplicate fashion to reduce extraction errors. A third author was available to resolve discrepancies. We conducted descriptive statistics, including frequencies and percentages, using R in RStudio.</p>
<h4>Results</h4>
<p> We identified 207 articles from PubMed for potential inclusion. After screening, 55 abstracts and full texts were available for full data extraction, with 80% (44 of 55) having positive conclusions germane to accuracy or clinical implications. At least 1 item of spin was present in every included article. Most positive conclusions in abstracts (75% [33 of 44]) and full texts (79.6% [35 of 44]) did not adequately reflect pooled estimates (<b>Table 25-1166</b>). A total of 34.5% of studies (19 of 55) used nonrecommended statistical approaches for pooling accuracy measures, reflecting potential spin. All 55 studies reported CIs within their full text for precision interpretation.</p>
<a href="/wp-content/uploads/2025_conference/tables/prc25-1166t1.png"> <img loading="lazy" src="/wp-content/uploads/2025_conference/tables/prc25-1166t1.png"> </a>
<h4>Conclusions</h4>
<p> Diagnostic test accuracy meta-analyses published within the top 20 pathology journals consistently overinterpret their findings. Authors should ensure that their findings are properly contextualized within predetermined diagnostic performance criteria. Additionally, authors should ensure that summary estimates are pooled using appropriate and robust methods that maintain the correlated nature of sensitivity and specificity.</p>
<h4>References</h4>
<p>1. Boutron I, Dutton S, Ravaud P, Altman DG. Reporting and interpretation of randomized controlled trials with statistically nonsignificant results for primary outcomes. JAMA. 2010;303(20):2058-2064. doi:10.1001/jama.2010.651</p>
<p>2. Yavchitz A, Ravaud P, Altman DG, et al. A new classification of spin in systematic reviews and meta-analyses was developed and ranked according to the severity. J Clin Epidemiol. 2016;75:56-65. doi:<a href="https://doi.org/10.1016/j.jclinepi.2016.01.020"><span class="Hyperlink CharOverride-6">10.1016/j.jclinepi.2016.01.020</span></a></p>
<p>3. McGrath TA, Bowdridge JC, Prager R, et al. Overinterpretation of research findings: evaluation of “spin” in systematic reviews of diagnostic accuracy studies in high-impact factor journals. Clin Chem. 2020;66(7):915-924. doi:<a href="https://doi.org/10.1093/clinchem/hvaa093"><span class="Hyperlink CharOverride-6">10.1093/clinchem/hvaa093</span></a></p>
<sup>1</sup>Office of Medical Student Research, Oklahoma State University Center for Health Sciences, Tulsa, OK, US, griffinhughesresearch@gmail.com; <sup>2</sup>Department of Psychiatry and Behavioral Sciences, Oklahoma State University Center for Health Sciences, Tulsa, OK, US.</p>
<h4>Conflict of Interest Disclosures </h4>
<p>Matt Vassar receives funding from the National Institute on Drug Abuse, the National Institute on Alcohol Abuse and Alcoholism, the US Office of Research Integrity, Oklahoma Center for Advancement of Science &amp; Technology, and internal grants from Oklahoma State University Center for Health Sciences outside the present work. All other authors have nothing to disclose.</p>


Assessment of Spin Among Diagnostic Accuracy Meta-Analyses Published in Top Pathology Journals: A Systematic Review

<p>Niklas Bobrovitz,<sup>1,2</sup> Harriet Ware,<sup>2</sup> Corson Johnstone,<sup>3</sup> Juliane Kennett,<sup>3</sup> Stephana Julia Moss,<sup>4</sup> Liam Whalen-Browne,<sup>3</sup> Faizan Khan,<sup>5</sup> Benjamin Fletcher,<sup>2</sup> Daniel J. Niven,<sup>3,6,7</sup> Henry T. Stelfox<sup>3,6,8</sup></p>
<h4>Objective </h4>
<p>To synthesize evidence on the prevalence and measurement of reproducibility in health sciences research.</p>
<h4>Design </h4>
<p>This was a systematic review with meta-analysis registered a priori.<sup>1</sup> We searched MEDLINE, Embase, Web of Science, the Cochrane Controlled Trials Registry and Database of Systematic Reviews, and grey literature published from January 1, 2000, to May 31, 2023. We included English-language articles in the health sciences that reported the prevalence of reproducible research or the prevalence of studies dedicated to reproducing research. We included studies published after 2000 to ensure relevance to current practice. Two authors independently screened articles in duplicate. Extractions were completed by one reviewer and verified by a second. The prevalence of reproducible research was defined as<i> </i>the proportion of research findings or studies for which the methods could be exactly repeated (methodological), the results could be corroborated by a study using the same or similar methods (results), or qualitatively similar conclusions could be made based on a study reproduction (inferential). Risk of bias was assessed using a modified Joanna Briggs Institute checklist. We meta-analyzed estimates using random effects models. To explore between-study heterogeneity, we conducted univariable and multivariable meta-regression using generalized linear mixed models. The level of significance was set at <i>P</i> &lt; .05.</p>
<h4>Results </h4>
<p>A total of 177 studies published between 2001 and 2023 were included, covering all health sciences domains: biomedical/preclinical (n = 56), clinical (n = 106), health/social care services (n = 2), population health (n = 5), and multiple domains (n = 8). Studies reported on 3 types of reproducibility: methodological (n = 68), results (n = 129), and inferential (n = 5). There were 7 methodological approaches for conducting reproducibility research and 38 metrics to quantify reproducibility. The pooled estimated prevalence of reproducible research was 36.2% (95% CI, 29.7%-43.3%; <i>I</i><sup>2</sup> = 99.8%; n = 180 estimates) with variation by research domain: 30.3% (95% CI, 20.3%-42.6%; n = 59 estimates) for biomedical/preclinical, 38.3% (95% CI, 30.0%-47.4%; n = 106 estimates) for clinical, 66.6% (95% CI, 20.7%-93.9%; n = 3 estimates) for health/social care services, and 36.5% (95% CI, 0.0%-100.0%; n = 3 estimates) for population health. There was a nonsignificant decline over time between 2001 and 2023 of −4.7% (95% CI −10.1 to 1.1%; <i>P</i> = .11) per year in the odds of reproducibility (<b>Figure 25-1041</b>). The pooled estimated prevalence of studies dedicated to reproducing research was 1.4% (95% CI, 0.40%-4.10%; <i>I</i><sup>2</sup> = 99.2%; n = 40 estimates). Most estimates were at moderate risk of bias (n = 193), with some at high (n = 25) or low (n = 2) risk. Meta-regression of publication year, research domain, type of reproducibility, metric to quantify reproducibility, and risk of bias explained minimal estimate heterogeneity.</p>

![](https://assets.underline.io/markdown_image/1/image/58c52ca1e8ab42d080707ad8bf3acf01.png)

<h4>Conclusions </h4>
<p>We found that an estimated one-third of health sciences research was reproducible and 1 in 100 studies were dedicated to reproducing research. Further investigation is needed to understand heterogeneity in estimates. Standardized strategies for quantifying reproducibility should be developed. Reproducibility studies are needed in health/social care services and population health research.</p>
<h4>Reference</h4>
<p>1. Bobrovitz N. The reproducibility of health research. Open Science Framework. 2021. <a href="https://file:///Users/jamft/Creative%20Cloud%20Files%20Company%20Account%20American%20Medical%20Association%20joe.amft@ama-assn.org%20D656564062986A6B0A495CB8@d81f414362986a64495eb4.e/JAmft%20Shared/CURRENT%20WORK/SPECIAL%20PROJECTS/PRC25/text/osf.io/3fhd9"><span class="Hyperlink CharOverride-6">osf.io/3fhd9</span></a></p>
<sup>1</sup>Department of Emergency Medicine, Cumming School of Medicine,<i> </i>University of Calgary, Canada, njhbobro@ucalgary.ca; <sup>2</sup>Centre for Health Informatics, Cumming School of Medicine,<i> </i>University of Calgary, Canada; <sup>3</sup>Department of Critical Care Medicine, Cumming School of Medicine,<i> </i>University of Calgary, Canada; <sup>4</sup>Department of Pediatrics, Division of Infectious Diseases, IWK Health Centre, Dalhousie University, Canada; <sup>5</sup>Hotchkiss Brain Institute, Cumming School of Medicine, University of Calgary, Canada; <sup>6</sup>O’Brien Institute for Public Health, University of Calgary, Canada; <sup>7</sup>Department of Community Health Sciences, Cumming School of Medicine,<i> </i>University of Calgary, Canada; <sup>8</sup>Faculty of Medicine and Dentistry, University of Alberta, Canada.</p>
<h4>Conflict of Interest Disclosures</h4>
<p> None reported. </p>
<h4>Funding/Support </h4>
<p>The work was funded by the Canadian Institutes of Health Research. </p>
<h4>Role of the Funder/Sponsor</h4>
<p> The funding body had no role in the design or conduct of the study; collection, management, analysis, or interpretation of the data; preparation, review, or approval of the abstract; or decision to submit the abstract for presentation.</p>


[V] Prevalence of Reproducible Health Sciences Research: A Systematic Review and Meta-Analysis

<p>Benjamin Djulbegovic,<sup>1</sup> Iztok Hozo,<sup>2</sup> Renata Iskander,<sup>5</sup> Austin J. Parish,<sup>3,4</sup> Jonathan Kimmelman,<sup>5</sup> John P. A. Ioannidis<sup>4</sup></p>
<h4>Objective</h4>
<p> Randomized clinical trials (RCTs) are commonly considered important for detecting small treatment benefits. However, previous studies have shown they are equally likely to detect large (“dramatic”) treatment effects.<sup>1-3</sup> Forecasting the likelihood of future large treatment effects is critical for informing policies on resource allocation in conducting human RCTs, including cancer trials.</p>
<h4>Design </h4>
<p>We conducted a systematic review to inform generalized Pareto distribution (GPD) under extreme value theory to predict future maximum treatment effects based on data from the past 65 years. We included consecutive cancer RCTs (“cohorts”) identified by funders or trial registries designed to minimize publication bias and analyzed all trials regardless of publication status. Five such cohorts have been described in the literature to date. </p>
<h4>Results </h4>
<p>Between 1955 and 2018, a total of 716 RCTs testing 984 experimental vs standard treatments in 349,947 patients were conducted and published by 2022, averaging approximately 20 RCTs per year. The shape parameter of the GPD had positive values, indicating no upper limit on maximum treatment effects. We found that the treatment with the largest effect in the past had an odds ratio (OR) of 45 (95% CI, 2-1008). If effect patterns remain the same and a similar pace of performing RCTs continues, the largest predicted future effect over the next 50 years would be an OR of 23 (95% CI, 4-43) (<span class="CharOverride-4"><b>Figure 25-0942</b></span>). We estimated a 20% probability of detecting new treatment effects with an OR greater than 50 in the next 50 years. We also found that conducting more RCTs (40 or 60 per year) would double or triple the probability of detecting breakthrough treatments with dramatic effects.</p>

![](https://assets.underline.io/markdown_image/1/image/e757b092f8aa49cda20b7b719367bd63.png)

<h4>Conclusions</h4>
<p> Our analysis indicates there may be no upper bound on the maximum discoverable treatment effects in cancer RCTs, but effect estimates will likely remain within the range of those observed between 1955 and 2022. Conducting more RCTs would accelerate the detection of treatments with large effects.</p>
<h4>References</h4>
<p>1. Djulbegovic B, Kumar A, Glasziou P, Miladinovic B, Chalmers I. Medical research: Trial unpredictability yields predictable therapy gains. <i>Nature</i>. 2013;500(7463):395-396. doi:10.1038/500395a</p>
<p>2. Hozo I, Djulbegovic B, Parish AJ, Ioannidis JPA. Identification of threshold for large (dramatic) effects that would obviate randomized trials is not possible. <i>J Clin Epidemiol</i>. 2022;145:101-111. doi:10.1016/j.jclinepi.2022.01.016</p>
<p>3. Djulbegovic B, Kumar A, Soares HP, et al. Treatment success in cancer: new cancer treatment successes identified in phase 3 randomized controlled trials conducted by the National Cancer Institute-sponsored cooperative oncology groups, 1955 to 2006. <i>Arch Intern Med</i>. 2008;168(6):632-642. doi:10.1001/archinte.168.6.632</p>
<sup>1</sup>Medical University of South Carolina, Division of Medical Hematology and Oncology, Department of Medicine, Charleston, SC, US, djulbegov@musc.edu; <sup>2</sup>Department of Mathematics, Indiana University Northwest, Gary, IN, US; <sup>3</sup>Department of Emergency Medicine, Lincoln Medical Center, Bronx, NY; <sup>4</sup>Stanford Prevention Research Center, Department of Medicine, Stanford University School of Medicine; Department of Epidemiology and Population Health, Stanford University School of Medicine; Department of Biomedical Data Science, Stanford University School of Medicine; Department of Statistics, Stanford University School of Humanities and Sciences, Meta-Research Innovation Center at Stanford (METRICS), Stanford University, Stanford, CA, US; <sup>5</sup> Department of Equity, Ethics and Policy School of Population and Global Health, McGill University, Montreal, QC, Canada.</p>
<h4>Conflict of Interest Disclosures </h4>
<p>John P. A. Ioannidis is a member of the Peer Review Congress Advisory Board but was not involved in the review or decision for this abstract. No other disclosures reported. </p>
<h4>Additional Information </h4>
<p>Datasets for this work were obtained with support of grants from the US National Institute of Health: R01CA140408, R01NS044417, R01NS052956, and R01CA133594 (Benjamin Djulbegovic). </p>


[V] Estimation of an Upper Limit on the Maximum Effect That Can Be Detected in Randomized Trials of Cancer Therapeutics

<p>Ambar Khan,<sup>1</sup> Ania Bobrowska,<sup>2</sup> Chloe Coelho,<sup>2</sup> Hannah Frost,<sup>2</sup> Swati Kumar,<sup>2</sup> Hannah Russell,<sup>3</sup> Anna Noel-Storr,<sup>4</sup> Molly Murton<sup>2</sup></p>
<h4>Objective</h4>
<p> This study aimed to evaluate a traditional (validated search filter) vs machine learning (classifier) approach for identifying randomized clinical trials (RCTs), in the context of a real-life systematic literature review (SLR) project. </p>
<h4>Design</h4>
<p> A real-life SLR was conducted following standard practices as recommended by the Cochrane Handbook,<sup>1</sup> covering all identification stages from electronic database searches to full-text review, to identify RCTs on fibrodysplasia ossificans progressiva (FOP; a rare disease). Two approaches were compared for the searches conducted in January 2025: (1) combining population search terms with the Scottish Intercollegiate Guidelines Network (SIGN) RCT search filter (“filter” approach) and (2) using population search terms only and running the results through the Cochrane RCT classifier at a 99% recall threshold (“classifier” approach). All subsequent stages were conducted in an identical manner by the same review team, with both abstracts and full texts dual-screened by independent reviewers against prespecified eligibility criteria. The time taken for each approach was measured and reviewer feedback was gathered via a modified NASA Task Load Index. Performance metrics for each approach were calculated, including the accuracy, sensitivity, specificity, and precision. McNemar test was used to assess statistical significance of the differences in the performance metrices, and a <i>P</i> value &lt;.05 was considered statistically significant. </p>
<h4>Results</h4>
<p> Prior to application of the classifier or filter, 9588 records were found for the FOP population. After application of the classifier or filter, 2327 and 1582 abstracts were identified as RCTs, respectively. At the end of the full-text review stage for each approach, the same 10 papers reporting on FOP RCTs were identified, demonstrating equal efficacy in terms of finding relevant papers in the context of a real-life SLR. However, when looking more generally at the classification of RCT vs non-RCT publications within the 9588 records (irrespective of ultimate inclusion in the SLR), sensitivity was significantly higher for the classifier vs the filter (McNemar test <i>P</i> &lt; .001; <b>Table 25-0997</b>). The trade-off for higher sensitivity of the classifier was that it had a significantly lower specificity and therefore required 31% more time than the filter approach (25.79 vs 17.86 hours). This also resulted in a less positive user experience in effort and frustration domains. </p>

![](https://assets.underline.io/markdown_image/1/image/bf3eeb47d6679394a2ede2d090402f99.png)

<h4>Conclusions</h4>
<p> For SLRs, where the identification of all relevant evidence is crucial, the performance of the classifier was superior to a traditional search filter. However, this was at the cost of increased time and reduced specificity. Further testing of the application of machine learning classifiers to augment literature review processes in real-life projects (including other, larger disease areas) is required. The risk of bias in this research, stemming from the researchers being unblinded to the approach, should also be noted.</p>
<h4>Reference</h4>
<p>1. Higgins J, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, et al. <i>Cochrane Handbook for Systematic Reviews of Interventions Version 6.3</i>. Cochrane; 2022.</p>
<sup>1</sup>Costello Medical, London, UK; <sup>2</sup>Costello Medical, Cambridge, UK, molly.murton@costellomedical.com; <sup>3</sup>Costello Medical, Manchester, UK; <sup>4</sup>Cochrane, London, UK.</p>
<h4>Conflict of Interest Disclosures</h4>
<p> Ambar Khan, Ania Bobrowska, Chloe Coelho, Hannah Frost, Swati Kumar, Hannah Russell, and Molly Murton reported being employees of Costello Medical. No other disclosures were reported.</p>
<h4>Funding/Support</h4>
<p> This study was funded by Costello Medical. </p>
<h4>Role of the Funder/Sponsor</h4>
<p> Costello Medical supported all aspects of the study, including the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the abstract; and the decision to submit the abstract for presentation.</p>


[V] Performance and Practicality of a Randomized Clinical Trial Classifier in Systematic Literature Reviews vs a Traditional Approach

<p>Ashrah Maleki,<sup>1</sup> Niina Sormanen,<sup>1</sup></span><span lang="fi-FI"> Kim Holmberg<sup>1</sup></p>
<h4>Objective </h4>
<p>While the alternative metric (altmetric) visibility of retracted publications has been studied,<sup>1</sup> less is known about how media and social media engage with corrections issued after retractions. This study examines how different types of retracted and corrected scientific publications engage scholarly and public audiences across altmetric platforms, focusing on how the reasons for retraction influence visibility. We distinguish between misinformation (unintentional error), disinformation (intentional deception), and disputed cases and assess their relative representation in altmetric activity.</p>
<h4>Design</h4>
<p> This cross-sectional study analyzes altmetric data of retracted and corrected publications from multiple sources. Data were extracted in November and December 2024 from Retraction Watch, Scopus, and Altmetric.com (and updated in June 2025). The dataset comprised (1) 34,191 DOIs of original articles (later retracted), (2) 31,935 DOIs of retraction notices, (3) 1276 DOIs of correction notices, and (4) corrected versions (ie, retracted articles followed by a correction). To ensure clarity in altmetric attribution, we excluded records in which a single DOI was assigned to both the original article and its corresponding retraction or correction notice. Retraction reasons were coded using a 6-zone classification framework (based on Sormanen et al<sup>2</sup>) that distinguishes both intent and (nonscholarly) consequence: critical disinformation (eg, data falsification or fabrication), critical misinformation (eg, unreliable analyses or results), inconsequential disinformation (eg, duplication or plagiarism), and inconsequential misinformation (eg, miscommunication or lack of approvals), along with disputed (eg, expressions of concern) and ambiguous cases. Each publication was counted using a severity-exclusive method: when multiple reasons were present, the record was assigned to the most severe applicable category (<b>Table 25-1043</b>). Platform-specific mention counts were analyzed across 16 altmetric platforms, including news media, blogs, X/Twitter, Mendeley, Dimensions, and policy documents. Engagement levels were compared across document types, and how these varied by reason category were examined.</p>

![](https://assets.underline.io/markdown_image/1/image/57d662ca8d5035770bc3c9ba4b2b75db.png)

<h4>Results </h4>
<p>Among the 37,451 records, original articles received the most absolute attention: 41% had an Altmetric Attention Score (AAS) greater than zero, 43% were saved in Mendeley, and 28% appeared on X. Retraction notices were less visible (32% AAS &gt; 0; 26% Mendeley). This means that retracted studies, often based on unreliable or deceptive findings, tend to receive more attention than the notices meant to correct them. Correction-related records showed higher proportional attention (51% of notices and 70% of retractions had an AAS greater than 0) but remained rare and largely absent from news (17% and 8%, respectively) and policy documents (6% and 1%, respectively). Among retraction notices, the most frequent reasons were critical disinformation (27%), inconsequential disinformation (27%), and critical misinformation (21%), yet these categories each saw only 8% to 10% with an AAS greater than 0. </p>
<h4>Conclusions </h4>
<p>Retractions receive more attention than corrections, even in serious cases. Corrections remain mostly invisible in public-facing media, underscoring the need for better metadata and targeted communication strategies.</p>
<h4>References</h4>
<p>1. Serghiou S, Marton RM, Ioannidis JP. Media and social media attention to retracted articles according to Altmetric. <i>PLoS One</i>. 2021;16(5):e0248625. doi:10.1371.journal.pone.0248625</p>
<p>2. Sormanen N, Holmberg K, Maleki A. Characterizing retraction reasons: types of mis- and disinformation in the scientific publications. Research in progress.</p>
<sup>1</sup>Department of Social Research, University of Turku, Turku, Finland, ashraf.maleki@utu.fi.</p>
<h4>Conflict of Interest</h4>
<p> None reported.</p>
<h4>Funding </h4>
<p>This research is supported by 2 projects: the European Media and Information Fund, focusing on unreliable science and the role of media regulation, and the Research Council of Finland, examining the impact of scientific misinformation.</p>
<h4>Acknowledgment </h4>
<p>We sincerely appreciate Professor Mike Thelwall (University of Sheffield) for his support in acquiring and sharing the Retraction Watch dataset.</p>


[V] Altmetric Footprint of Retracted and Corrected Publications: The Role of Misinformation and Disinformation

<p>Emilie Gunn,<sup>1</sup> Kelly Brooks,<sup>1</sup> Stephanie Valladao<sup>1</sup></p>
<h4>Objective </h4>
<p>Prior research has shown the effects of shortening a review deadline on overall time to review.<sup>1</sup> We wanted to determine whether extending the standard 2-week deadline for peer review in a scholarly journal would have positive outcomes. In particular, we wanted to test the hypothesis that offering a 21-day deadline would increase the reviewer conversion rate, thus reducing the number of reviewers invited before getting 2 reviewers to agree. A secondary end point was the number of days it took reviewers to complete their reviews.</p>
<h4>Design </h4>
<p>In this pilot study, 593 review invitations for 95 manuscripts were sent from <i>JCO Oncology Practice</i> between July 1, 2024, and September 30, 2024, with a 21-day deadline. This was compared with 606 invitations for 116 manuscripts sent between July 1, 2023, and September 30, 2023, with a 14-day deadline. Only Original Reports and Reviews were studied because these article types require at least 2 reviewers. Using the Editorial Manager submission system, editors invited reviewers using the normal process, except that reviewers were offered 21 days instead of the industry-standard 14 to complete their reviews. Reviewers received information regarding the 21-day time frame at initial invitation, at the time of review invitation acceptance, and via reminders 1 week and 2 days prior to the deadline. The goal for each paper was to obtain 2 reviewers.</p>
<h4>Results </h4>
<p>When the time allowed to submit a review was increased to 21 days, the number of reviewers invited per manuscript increased by 0.5 reviewers (10.4%). The time it took reviewers to accept a review invitation increased by 0.1 days (12.5%), measured as the number of days from the date the invitation was sent to the date the reviewer agreed <br />(<b>Table 25-0847</b>). The percentage of late reviews decreased by 5% (from 27% to 22%). The median (IQR) time to submit a review was 19 (11-33) days, compared with 13 (5-18) days during the control period. Additionally, late reviews were submitted an average of 11.1 days late, a 29.1% increase. However, median (IQR) time from submission to first decision decreased by 4.7% to 41 (31-55) days from 43 (30-65.5) days.</p>

![](https://assets.underline.io/markdown_image/1/image/280d4f142fd24710f891e59a1d23acc4.png)

<h4>Conclusions </h4>
<p>Extending the 2-week deadline for peer review did not decrease the number of reviewers invited before getting 2 to agree. When the allowed time was increased, the time taken to review increased as well. Although there were fewer late reviews overall, the reviews submitted past the deadline were later than those submitted under the 14-day deadline. We ultimately recommended that the journal continue with the current 2-week deadline for reviewers. Further research is needed in this area.</p>
<h4>Reference</h4>
<p>1. Cochran A, Guertin L. Affecting editor behavior with data: a case study. <i>Sci Editor</i>. 2014;37(1):117-119.</p>
<sup>1</sup>American Society of Clinical Oncology, Alexandria, VA, US, emilie. gunn@asco.org. <h4>Conflict of Interest Disclosures</h4> None reported. <h4>Acknowledgment</h4> We thank Jeffrey Peppercorn for his help with designing this abstract and Scott Marchese for his statistical expertise.



Results of Testing the Gold Standard 2-Week Reviewer Deadline

<p>Emmanuel A. Zavalis,<sup>1</sup>Love Ahnström,<sup>2</sup><Natasha Olsson,<sup>2</sup>Gustav Nilsonne<sup>2,3</sup></p>
<p><h4>Objective</h4> The objective of this study was to assess lead times and application decisions using the national database of the Swedish Ethical Review Authority since its inception. We also aimed to assess the effect of the COVID-19 ethical review fast-track implemented in spring 2020. Prior studies of ethical review have been limited to single or few institutions.<sup>1,2</sup></p>
<p><h4>Design</h4> This retrospective analysis examined 21,962 administrative records of all ethical review applications submitted in Sweden between 2021 and 2023. The applications were categorized by type (eg, amendments, clinical trial, or processing sensitive data) and sponsor (eg, health care systems, universities, or corporations). Kaplan-Meier curves were constructed for analysis of time to decision. Adherence to statutory deadlines was assessed for amendments (35 days) and drug trials (60 days).<sup>3</sup> Log-rank test was used for statistical testing with α = .05 as a significance threshold.</p>
<p><h4>Results</h4> Following exclusions (eg, incomplete applications, unpaid fees, or withdrawals), 20,440 applications remained: 5866 from 2021, 6926 from 2022, and 7648 from 2023. Submissions were primarily from health care systems (10,456 [51%]) and universities (8681 [42%]), with corporations accounting for 578 applications (3%). Amendment applications constituted the largest category of applications (51% in 2021, 50% in 2022, and 44% in 2023). Applications in the “processing of personal data” category doubled from 736 to 1574 over the study period (<b>Figure 25-0966</b>, A). Of the 20,440 applications, initial approvals accounted for 68% (n = 13,864), further information or change to the project was requested in 24% (n = 5001) of cases, and 1.5% (n = 313) were rejected. Final decisions resulted in approvals for 91% (n = 18,516), with rejections being rare (2%), as seen in <b>Figure 25-0966</b>, B. Geographic variation was observed, with Stockholm having the shortest (34 days) and Lund the longest (43 days) median lead times. COVID-19 studies were associated with expedited processing times (median time to decision, 51 vs 63 days; P &lt; .001). While 27% of amendment applications were without a decision after 35 days, 39% of original applications as well as 58% of drug trials were without a decision after 60 days (<b>Figure 25-0966</b>, C).</p>

![](https://assets.underline.io/markdown_image/1/image/0bbf11d39a2fb72d73a5e55c57749cc8.png)

<p><span class="text-bold"><h4>Conclusions</h4> </span>The Swedish Ethical Review Authority mainly handled amendments. Lead times were longer than statutory deadlines for more than half of drug trials and for a quarter of amendments. The processing times showed geographic variations, and the fast-track for COVID-19 showed a significant association with processing times. Further studies of the qualitative aspects of ethical reviews’ impact on research plans as well as the nature of the applications for amendments would be of interest for a better understanding of the value introduced and the harms prevented by ethical review.</p>
<h4>References</h4>
<p>1. Hall DE, Hanusa BH, Stone RA, Ling BS, Arnold RM. Time required for institutional review board review at one Veterans Affairs medical center. <i>JAMA Surg</i>. 2015;150(2):103. doi:10.1001/jamasurg.2014.956</p>
<p>2. Varley PR, Feske U, Gao S, et al. Time required to review research protocols at 10 Veterans Affairs institutional review boards. <i>J Surg Res</i>. 2016;204(2):481-489. doi:10.1016/j.jss.2016.06.004</p>
<p>3. Sveriges Riksdag. Ordinance (2003:615) on ethical review of research involving humans. Accessed January 18, 2025. <a href="https://www.riksdagen.se/sv/dokument-och-lagar/dokument/svensk-forfattningssamling/forordning-2003615-om-etikprovning-av-forskning_sfs-2003-615/">https://www.riksdagen.se/sv/dokument-och-lagar/dokument/svensk-forfattningssamling/forordning-2003615-om-etikprovning-av-forskning_sfs-2003-615/</a></p>
<sup>1</sup>Department of Physiology and Pharmacology, Karolinska Institutet, Stockholm, Sweden; <sup>2</sup>Department of Clinical Neuroscience, Karolinska Institutet, Stockholm, Sweden, gustav.nilsonne@ki.se; <sup>3</sup>Department of Psychology, Stockholm University, Stockholm, Sweden. <h4>Conflict of Interest Disclosures</h4> None reported.



Analysis of Decisions and Lead-Time in Ethical Review Boards in Sweden

<p>Christopher S. Cotton,<sup>1</sup> Abid Alam,<sup>1</sup> Sophie Tosta,<sup>2</sup> Timothy G. Buchman,<sup>3</sup> David M. Maslove<sup>4,5,6</sup></p>
<h4>Objective</h4>
<p> Peer review is a cornerstone of academic publishing, providing a bulwark against the dissemination of flawed or unethical research. Historically, peer review has been carried out by experts on a volunteer basis. Financial compensation for peer reviewers has been discussed but with little empirical evidence to inform the debate.<sup>1</sup> A single experimental study<sup>2</sup> has examined this question, finding that for peer review at an economics journal, offering a $100 gift card to reviewers reduced median review time by 8 days. No experimental study to our knowledge has looked at the effect of paying for peer review at a medical journal. Our objective was to determine the effect of a cash incentive on the rate of fulfillment of peer review requests at a medical journal.</p>
<h4>Design </h4>
<p>We conducted a quasi-randomized experimental study at <i>Critical Care Medicine</i>, a specialty medical journal. We divided the experimental period (September 2023 to March 2024) into alternating 2-week blocks, during which all peer review requests sent out to prospective reviewers included either an offer of $250 to complete the peer review assignment (incentive blocks) or no offer (control blocks). Only original research reports were included in the study, with the monetary incentive applying only to the initial review. The primary outcome was the rate of invitation –to completed review conversion, defined as the ratio of reviews submitted divided by the number of reviewer invitations sent out. Review quality was a secondary outcome.</p>
<h4>Results</h4>
<p> During the study, 131 manuscripts went out for review, with a median (IQR) of 5 (2-8) reviewer invitations per manuscript. Of 715 reviewer invitations, 414 (58%) included an incentive offer. There was no difference in the proportion of invitations that were accepted (control vs incentive, 144 of 301 [48%] vs 218 of 414 [53%]; <i>P </i>= .20), but a difference was seen in the primary outcome, with reviewers in the intervention group having a greater odds of completing the review compared with those in the control group (control vs incentive, 127 of 301 [42%] vs 206 of 414 [50%]; odds ratio, 1.36; 95% CI. 1.01-1.84). A Cox proportional hazards model showed that incentivized invitations were fulfilled slightly faster (median [IQR] time, 11 [6-14] days vs 12 [7-14] days; hazard ratio, 1.30; 95% CI, 1.04-1.62). There was no difference in the quality of reviewer reports, as adjudicated by the handling editors (control vs incentive, median [IQR] score on a 100-point scale, 75 [70-85] vs 75 [66-90]).</p>

![](https://assets.underline.io/markdown_image/1/image/7a9a7744d70d458b4a46f19a2b4e1870.png)

<h4>Conclusions</h4>
<p> Our findings suggest that cash payments can incentivize peer review at a medical journal, potentially providing a method to help journals expedite peer review in situations like public health emergencies. Important policy questions remain around cost-benefit considerations; extrapolating from the rate of reviews required during the study period, we estimate that offering $250 for each review would cost <i>Critical Care Medicine </i>nearly $150,000 annually. Other considerations include sources of funding for paid reviews, the effect of nonmonetary incentives, and the potential unintended consequences of paid reviews.</p>
<h4>References</h4>
<p>1. Cheah PY, Piasecki J. Should peer reviewers be paid to review academic papers? <i>Lancet</i>. 2022;399(10335):P1601. doi: 10.1016/S0140-6736(21)02804-X</p>
<p>2. Chetty R, Saez E, Sándor L. What policies increase prosocial behavior? an experiment with referees at the Journal of Public Economics. <i>J Econ Perspect</i>. 2104;28(3):169-188. doi: 10.1257/jep.28.3.169</p>
<sup>1</sup>Department of Economics, Queen’s University, Kingston, ON, Canada; <sup>2</sup>Society of Critical Care Medicine, Mount Prospect, IL, US; <sup>3</sup>Emory Critical Care Center, Emory University School of Medicine, Atlanta, GA, US; <sup>4</sup>Department of Critical Care Medicine, Queen’s University, Kingston, ON, Canada, david.maslove@queensu.ca; <sup>5</sup>Department of Medicine, Queen’s University, Kingston, ON, Canada, <sup>6</sup>Kingston Health Sciences Centre, Kingston, ON, Canada. <h4>Conflict of Interest Disclosures</h4> David M. Maslove was an associate editor at <i>Critical Care Medicine</i> at the time the study was conducted. Timothy G. Buchman was the editor-in-chief at </i>Critical Care Medicine</i> at the time the study was conducted. Sophie Tosta is the managing editor of <i>Critical Care Medicine</i>. <h4>Funding/Support</h4> This study was supported by a grant from the Government of Canada’s New Frontiers in Research Fund (grant No. NFRFR-2021-00335). David M. Maslove is supported by a New Clinician Scientist Award from the Southeastern Ontario Academic Medical Association. Christopher S. Cotton receives research funding as the Jarislowsky-Deutsch Chair in Economic & Financial Policy at Queen’s University. <h4>Role of Funder/Sponsor</h4> No funder had a role in the design, analysis, or interpretation of the study. <h4>Acknowledgment</h4> The authors wish to thank the associate editors and senior editors at <i>Critical Care Medicine</i>, as well as the editorial assistants at the journal, and the staff at the Society of Critical Care Medicine for facilitating the study.



Monetary Incentives for Peer Review at a Medical Journal: A Quasi-Randomized Experimental Study

<p>Sara Schroter,<sup>1,2</sup> Rebecca Harmston,<sup>3</sup> Emma Doble,<sup>1</sup> Sophie Cook,<sup>1</sup> Amy Price<sup>4,5</sup></p>
<h4>Objective</h4>
<p> Calls for patient partners to be paid for their contributions to the health sector are growing.<sup>1</sup> <i>The BMJ</i> invites patients and the public, as well as academic and clinical peer reviewers, to review manuscripts.<sup>2</sup> On November 28, 2024, <i>BMJ</i> announced it would remunerate its patient and public reviewers from 2025.<sup>3</sup>We surveyed these reviewers to capture their perspectives on the introduction of remuneration and overall experience of reviewing.</p>
<h4>Design</h4>
<p> Survey of current patient and public reviewers (those who had completed a review in the previous 3 years) on SurveyMonkey on December 11, 2024. We asked reviewers about their experiences of reviewing and their perspectives on the introduction of remuneration (£50 or an online subscription to a BMJ journal was provided for each completed review).</p>
<h4>Results</h4>
<p>We received a response from 183 of 267 invited reviewers (69%). Independent <i>t</i> tests showed that respondents, compared with nonresponders, received more invitations (mean: 14.0 vs 10.3; <i>P </i>= .01) and completed more reviews (mean: 7.7 vs 3.7; <i>P</i> &lt; .001), but there was no difference in 5-point review quality score (mean: 3.8 vs 3.8; <i>P </i>= .81). The majority of respondents were based in the UK and the US. Respondents were positive about their experience of reviewing: 84% (n = 154) reported it as a very good or good experience. The primary motivation for 93% of reviewers (n = 170) was contributing patient perspectives to the research process. Regarding remuneration, 87 (48%) and 58 (32%) indicated they would be more likely to review a manuscript if they were offered £50 or a 12-month online subscription to one of our journals, respectively. However, approximately one-third said these incentives would not make them more likely to review (60 [33%] payment; 71 [39%] journal subscription), and 30 (16%) and 48 (26%), respectively, were unsure. Half of the participants (n = 93 [51%]) thought offering the choice of these incentives would help attract a more diverse sample of reviewers, 69 (38%) were unsure, and 15 (8%) did not. Views on remuneration were divergent (<b>Table 25-0908</b>) with some seeing it as a recognition of value and others considering it unnecessary. While 107 (58%) had no concerns about introducing payment for patient and public reviewers, 37 (20%) did have concerns and 33 (18%) were unsure. Concerns included potential conflicts of interest, tax implications and effort of declaring, impact on means-tested benefits, effect on review quality and integrity, people becoming reviewers for the wrong reasons, and high foreign exchange banking fees. Others viewed the remuneration as a modest token unlikely to influence quality. Respondents emphasized the importance of optional incentives to accommodate individual preferences and suggested monitoring the quality of impact after implementation of remuneration.</p>

![](https://assets.underline.io/markdown_image/1/image/e8c666f657a7ccf5964c0a603485b888.png)

<h4>Conclusions</h4>
<p> <i>BMJ</i>’s patient and public reviewers hold diverse views on reviewer remuneration, highlighting the importance of providing flexible, optional incentive choices to accommodate varying individual needs, values, and preferences.</p>
<h4>References </h4>
<p>1. Asante K. Remuneration of African patient partners is an important tool for health justice. <i>BMJ</i>. 2024;387:q2675. doi:10.1136/bmj.q2675</p>
<p>2. Schroter S, Price A, Flemyng E, et al.<i> </i>Perspectives on involvement in the peer-review process: surveys of patient and public reviewers at two journals. <i>BMJ Open.</i> 2018;8(9):e023357. doi:10.1136/bmjopen-2018-023357</p>
<p>3. Doble E, Schroter S, Price A, et al. <i>The BMJ</i> will remunerate patient and public reviewers. <i>BMJ</i>. 2024;387:q2581. doi:10.1136/bmj.q2581</p>

<sup>1</sup>BMJ, BMA House, Tavistock Square, London, UK, sschroter
@bmj.com; <sup>2</sup>Faculty of Public Health and Policy, London School of Hygiene and Tropical Medicine, London, UK; <sup>3</sup>Patient and public reviewer, BMJ, London, UK; <sup>4</sup>Dartmouth Health, NH, US; <sup>5</sup>Colorado School of Public Health, CO, US.
<h4>Conflict of Interest Disclosures</h4> Sara Schroter and Sophie Cook are employed by BMJ, Emma Doble is a freelance patient editor for BMJ, Amy Price was a freelance patient editor for BMJ, and Rebecca Harmston is a patient and public reviewer for BMJ.
<h4>Acknowledgment</h4> We thank Layla Abdulbaki, Shahad Al Mashjari, Shahd Rihawi, Alaa Almuhaini, Hygieia Jacob, Munya Alshaalani, Aliaa Bhahzad, Manvi Upadhaya, Zainab Almutawa, Sarah Khan, Adriana Goracci, and Adrian Yee for piloting the questionnaire, providing feedback on the clarity of the questions, and reviewing the work for readability and relevance. We appreciate all of BMJ’s patient and public reviewers who participated in the study.


Exploring Views on Remuneration for Review: A Survey of BMJ' s Patient and Public Reviewers

<p>Simine Vazire<sup>1</sup></p>
<h4>Importance</h4>
<p> Scientific institutions, including journals, should work to earn trust from the scientific community and from the public. With prominent threats to trust in science around the world, it is especially important to make clear why scientific institutions are worthy of trust. Too often, journals’ reputations are unearned—based on flawed metrics, such as impact factors—or simply the inertia of prestige. But journal prestige can and should be earned.</p>
<h4>Observations</h4>
<p> There are many things journals can do to give the community concrete, verifiable indicators of their priorities. First, journals can thoroughly vet the articles they publish to ensure they are accurate, well-calibrated, and transparently reported. Second, journals can invest in postpublication critique and correction so that when they inevitably publish some things that are wrong or miscalibrated, they correct themselves. Third, journals can encourage audits of their published articles, such as those conducted by the Institute for Replication. </p>
<h4>Conclusions</h4>
<p> Scientific journals will sometimes publish research that turns out to be wrong. That is not a reason to distrust journals or the peer review process. But journals do have an obligation to take reasonable steps to vet research before it is published and to make those efforts visible and verifiable to readers. Journals that invest in and facilitate both prepublication and postpublication quality checks, error detection, and correction are the ones that deserve the most trust and the most prestige.</p>

<sup>1</sup>Melbourne School of Psychological Sciences, University of Melbourne, Parkville, Australia, simine@gmail.com.
<h4>Conflict of Interest Disclosures</h4> None reported.


Next from The 10th Peer Review Congress

Assessment of Spin Among Diagnostic Accuracy Meta-Analyses Published in Top Pathology Journals: A Systematic Review

Similar lecture

Use of an Open Science Checklist and Reproducibility of Findings: A Randomized Controlled Trial