United States

&lt;p&gt;Christopher C. Winchester,&lt;sup&gt;1&lt;/sup&gt; Mark J. Rolfe,&lt;sup&gt;1&lt;/sup&gt; William T. Gattrell,&lt;sup&gt;2&lt;/sup&gt; Patricia Logullo,&lt;sup&gt;3&lt;/sup&gt; Keith Goldman,&lt;sup&gt;4&lt;/sup&gt; Amy Price,&lt;sup&gt;5&lt;/sup&gt; Paul Blazey,&lt;sup&gt;6&lt;/sup&gt; Esther J. van Zuuren,&lt;sup&gt;7&lt;/sup&gt; Niall Harrison&lt;sup&gt;8&lt;/sup&gt;&lt;/p&gt;
&lt;h4&gt;Objective &lt;/h4&gt;
&lt;p&gt;The ACcurate COnsensus Reporting Document (ACCORD) guideline supports the reporting of biomedical studies involving any consensus method.&lt;sup&gt;1&lt;/sup&gt; We evaluated its uptake by bibliometric analysis and qualitative and quantitative assessment of implementation activities 1 year post publication.&lt;/p&gt;
&lt;h4&gt;Design&lt;/h4&gt;
&lt;p&gt; An implementation plan&lt;sup&gt;2&lt;/sup&gt; was established by the ACCORD Steering Committee before guideline publication, including developing an explanation and elaboration (E&amp;amp;E) document, writing journal editorials, presenting at congresses, and encouraging journals and publishers to include ACCORD in author instructions. Guideline views and downloads (via &lt;i&gt;PLOS Medicine&lt;/i&gt;,&lt;sup&gt;1&lt;/sup&gt; the publishing journal) and citations (Dimensions, plus additional references from the Steering Committee’s reference libraries) were obtained for the period from January 23, 2024, to January 22, 2025. Likely journal and publisher beneficiaries of ACCORD were identified by a literature search to determine the 10 journals that published the most consensus studies (Web of Science for 2020-2023) and other publications at the discretion of the Steering Committee. Relevant journal and publisher stakeholders were contacted directly or indirectly (eg, via journal websites). Activities were evaluated against the implementation&lt;i&gt; &lt;/i&gt;plan&lt;sup&gt;2&lt;/sup&gt;; journal and publisher responses were described.&lt;/p&gt;
&lt;h4&gt;Results&lt;/h4&gt;
&lt;p&gt; In the 1 year following publication, the ACCORD guideline was viewed 13,752 times, downloaded (full-text PDF) 3790 times, and cited 105 times (5 citations [4.8%] were in articles by ACCORD authors, including the ACCORD E&amp;amp;E&lt;sup&gt;3&lt;/sup&gt;). Sources of guideline citations included consensus studies (67.6% [71 of 105]), reviews (11.4% [12 of 105]), study protocols (8.6% [9 of 105]), other reporting standards and guidelines (6.7% [7 of 105]), and editorials (3.8% [4 of 105]). Of the journals contacted by ACCORD Steering Committee members (between August 24, 2023 [preprint publication], and January 22, 2025), &lt;i&gt;BMJ Open&lt;/i&gt;, &lt;i&gt;British Journal of Dermatology&lt;/i&gt;, and &lt;i&gt;Journal of Clinical Epidemiology &lt;/i&gt;mandated the use of ACCORD for consensus studies, while the &lt;i&gt;British Journal of Sports Medicine&lt;/i&gt;,&lt;i&gt; Internal Medicine Journal&lt;/i&gt;, and &lt;i&gt;Pragmatic and Observational Research &lt;/i&gt;encouraged its use. &lt;i&gt;PLOS &lt;/i&gt;has confirmed its intention to include the guideline in their author instructions. Although a further 4 journals have expressed interest in ACCORD and 1 publisher has offered to raise awareness via a webinar, they are yet to commit to its inclusion in author instructions; 2 journals and publishers have not responded to inquiries.&lt;/p&gt;
&lt;h4&gt;Conclusions &lt;/h4&gt;
&lt;p&gt;One year after publication, the ACCORD guideline has been cited and used to inform the design and reporting of consensus-based research. However, journal and publisher adoption has been limited, which may affect long-term uptake. Given the resources invested in developing reporting guidelines and their potential to improve reporting, the role of biomedical journals and publishers in their adoption warrants wider discussion.&lt;/p&gt;
&lt;h4&gt;References&lt;/h4&gt;
&lt;p&gt;1. Gattrell WT, Logullo P, van Zuuren EJ, et al. ACCORD (ACcurate COnsensus Reporting Document): a reporting guideline for consensus methods in biomedicine developed via a modified Delphi. &lt;i&gt;PLOS Med&lt;/i&gt;. 2024;21:e1004326. doi:10.1371/journal.pmed.1004326&lt;/p&gt;
&lt;p&gt;2. Gattrell WT, Hungin AP, Price A, et al. ACCORD guideline for reporting consensus-based methods in biomedical research and clinical practice: a study protocol. &lt;i&gt;Res Integr Peer Rev&lt;/i&gt;&lt;span lang=&quot;de-DE&quot;&gt;. 2022;7:3. doi:10.1186/s41073-022-00122-0&lt;/span&gt;&lt;/p&gt;
&lt;p&gt;&lt;span lang=&quot;de-DE&quot;&gt;3. Logullo P, van Zuuren EJ, Winchester CC, et al. &lt;/span&gt;ACcurate COnsensus Reporting Document (ACCORD) explanation and elaboration: guidance and examples to support reporting consensus methods. &lt;i&gt;PLOS Med&lt;/i&gt;. 2024;21:e1004390. doi:10.1371/journal.pmed.1004390&lt;/p&gt;
&lt;sup&gt;1&lt;/sup&gt;Oxford PharmaGenesis, Oxford, UK, chris.winchester@pharmagenesis.com; &lt;sup&gt;2&lt;/sup&gt;Independent Medical Communications Professional, Oxfordshire, UK; &lt;sup&gt;3&lt;/sup&gt;Bodleian Libraries, University of Oxford, Oxford, UK; &lt;sup&gt;4&lt;/sup&gt;Medical Affairs + Health Impact, AbbVie, North Chicago, IL, US; &lt;sup&gt;5&lt;/sup&gt;Dartmouth Institute for Health Policy &amp;amp; Clinical Practice (TDI), Geisel School of Medicine, Dartmouth College, Hanover, NH, US; &lt;sup&gt;6&lt;/sup&gt;School of Kinesiology, Department of Medicine, University of British Columbia, Vancouver, British Columbia, Canada; &lt;sup&gt;7&lt;/sup&gt;Leiden University Medical Centre, Leiden, the Netherlands; &lt;sup&gt;8&lt;/sup&gt;OPEN Health Communications, London, UK.&lt;/p&gt;
&lt;h4&gt;Conflict of Interest Disclosures &lt;/h4&gt;
&lt;p&gt;Christopher C. Winchester is an employee, director, and shareholder of Oxford PharmaGenesis; a director of Oxford Health Policy Forum CIC; a trustee of the Friends of the National Library of Medicine; and an associate fellow of Green Templeton College, University of Oxford. Mark J. Rolfe is an employee of Oxford PharmaGenesis. William T. Gattrell was an independent medical communications professional at the time of this study and is currently an employee of Bristol Myers Squibb. Keith Goldman is an employee of AbbVie. Niall Harrison is an employee of OPEN Health Communications.&lt;/p&gt;
&lt;h4&gt;Funding/Support &lt;/h4&gt;
&lt;p&gt;Medical writing, editorial, and project management support for this abstract were provided by Oxford PharmaGenesis.&lt;/p&gt;
&lt;h4&gt;Acknowledgments&lt;/h4&gt;
&lt;p&gt; Medical writing support was provided by Alison Chisholm, with editorial support from Jenny Thorp, and administrative support was provided by Mehraj Ahmed, Ryan Gamble, and Jessica Miller.&lt;/p&gt;
&lt;h4&gt;Additional Information &lt;/h4&gt;
&lt;p&gt;No authors were reimbursed for participating in the initiative.&lt;/p&gt;


The 10th Peer Review Congress

Challenges in achieving uptake and journal endorsement of the ACcurate COnsensus Reporting Document (ACCORD) guideline

and scientometrics

informatics

reporting guidelines

dissemination of information

bibliometrics

The Tenth Peer Review Congress featured 3 days for presentations of new research into peer review and all aspects of scientific publication, bias, quality of reporting, and information access and dissemination.

This year’s meeting was hybrid, and all plenary sessions were livestreamed with opportunities to view presentations and engage with participants in person and virtually during and after the meeting. All recordings are now available on Underline for viewing.

<div style="position:relative;padding-top:max(60%,324px);width:100%;height:0;"><iframe style="position:absolute;border:none;width:100%;height:100%;left:0;top:0;" src="https://online.fliphtml5.com/ebtyf/mget/"  seamless="seamless" scrolling="no" frameborder="0" allowtransparency="true" allowfullscreen="true" ></iframe></div>


#### Why? 
Our aim is to encourage research into the quality and credibility of peer review and scientific publication, to establish the evidence base on which scientists can improve the conduct, reporting, and dissemination of scientific research.

#### Who? 
Participants include editors and publishers of scientific peer-reviewed journals, researchers, funders, bibliometric and informatics experts, information innovators, librarians, journalists, policymakers, ethicists, scientific information disseminators, and anyone interested in the progress of the scientific information enterprise and the quality of scientific evidence.

The Congress embraces a wide range of disciplines, including biomedicine, health and life sciences, applied sciences, basic sciences, physical and chemical sciences, mathematics, computer sciences, engineering, economics, and social sciences as well as new and emerging disciplines.

#### What? 
The Congress provides a forum for the presentation and discussion of new research into the quality and credibility of editorial and funding peer review, scientific and scholarly publication, research methods and reporting practices, identification and management of biases, transparency, reproducibility, access, publication and dissemination metrics, identification and management of threats to the integrity of science, and advances in information exchange.

**Follow us**

[![](https://assets.underline.io/markdown_image/1/image/662a22766b9788ed3766cb23cd156e32.png)](https://x.com/peerrevcongress)  [![](https://assets.underline.io/markdown_image/1/image/bb105afb1773a01fa169aa97ca1af49a.png)](https://www.youtube.com/channel/UCsG_0nKi3_Zfyv0yRPdxjWQ)  [![](https://assets.underline.io/markdown_image/1/image/034e1e70a194a9a4178fdf470ef796db.png)](https://bsky.app/profile/peerreviewcongress.bsky.social)



You need to log in with the email address you registered with. 



Please log in to explore this event.

Registration is required to attend this event, please [here](https://peerreviewcongress.org/registration-information/) to secure your ticket. Please check your spam/promotional inbox too.



Please register!

The Tenth Peer Review Congress features 3 days for presentations of new research into peer review and all aspects of scientific publication, bias, quality of reporting, and information access and dissemination.

<p>Christopher C. Winchester,<sup>1</sup> Mark J. Rolfe,<sup>1</sup> William T. Gattrell,<sup>2</sup> Patricia Logullo,<sup>3</sup> Keith Goldman,<sup>4</sup> Amy Price,<sup>5</sup> Paul Blazey,<sup>6</sup> Esther J. van Zuuren,<sup>7</sup> Niall Harrison<sup>8</sup></p>
<h4>Objective </h4>
<p>The ACcurate COnsensus Reporting Document (ACCORD) guideline supports the reporting of biomedical studies involving any consensus method.<sup>1</sup> We evaluated its uptake by bibliometric analysis and qualitative and quantitative assessment of implementation activities 1 year post publication.</p>
<h4>Design</h4>
<p> An implementation plan<sup>2</sup> was established by the ACCORD Steering Committee before guideline publication, including developing an explanation and elaboration (E&amp;E) document, writing journal editorials, presenting at congresses, and encouraging journals and publishers to include ACCORD in author instructions. Guideline views and downloads (via <i>PLOS Medicine</i>,<sup>1</sup> the publishing journal) and citations (Dimensions, plus additional references from the Steering Committee’s reference libraries) were obtained for the period from January 23, 2024, to January 22, 2025. Likely journal and publisher beneficiaries of ACCORD were identified by a literature search to determine the 10 journals that published the most consensus studies (Web of Science for 2020-2023) and other publications at the discretion of the Steering Committee. Relevant journal and publisher stakeholders were contacted directly or indirectly (eg, via journal websites). Activities were evaluated against the implementation<i> </i>plan<sup>2</sup>; journal and publisher responses were described.</p>
<h4>Results</h4>
<p> In the 1 year following publication, the ACCORD guideline was viewed 13,752 times, downloaded (full-text PDF) 3790 times, and cited 105 times (5 citations [4.8%] were in articles by ACCORD authors, including the ACCORD E&amp;E<sup>3</sup>). Sources of guideline citations included consensus studies (67.6% [71 of 105]), reviews (11.4% [12 of 105]), study protocols (8.6% [9 of 105]), other reporting standards and guidelines (6.7% [7 of 105]), and editorials (3.8% [4 of 105]). Of the journals contacted by ACCORD Steering Committee members (between August 24, 2023 [preprint publication], and January 22, 2025), <i>BMJ Open</i>, <i>British Journal of Dermatology</i>, and <i>Journal of Clinical Epidemiology </i>mandated the use of ACCORD for consensus studies, while the <i>British Journal of Sports Medicine</i>,<i> Internal Medicine Journal</i>, and <i>Pragmatic and Observational Research </i>encouraged its use. <i>PLOS </i>has confirmed its intention to include the guideline in their author instructions. Although a further 4 journals have expressed interest in ACCORD and 1 publisher has offered to raise awareness via a webinar, they are yet to commit to its inclusion in author instructions; 2 journals and publishers have not responded to inquiries.</p>
<h4>Conclusions </h4>
<p>One year after publication, the ACCORD guideline has been cited and used to inform the design and reporting of consensus-based research. However, journal and publisher adoption has been limited, which may affect long-term uptake. Given the resources invested in developing reporting guidelines and their potential to improve reporting, the role of biomedical journals and publishers in their adoption warrants wider discussion.</p>
<h4>References</h4>
<p>1. Gattrell WT, Logullo P, van Zuuren EJ, et al. ACCORD (ACcurate COnsensus Reporting Document): a reporting guideline for consensus methods in biomedicine developed via a modified Delphi. <i>PLOS Med</i>. 2024;21:e1004326. doi:10.1371/journal.pmed.1004326</p>
<p>2. Gattrell WT, Hungin AP, Price A, et al. ACCORD guideline for reporting consensus-based methods in biomedical research and clinical practice: a study protocol. <i>Res Integr Peer Rev</i><span lang="de-DE">. 2022;7:3. doi:10.1186/s41073-022-00122-0</span></p>
<p><span lang="de-DE">3. Logullo P, van Zuuren EJ, Winchester CC, et al. </span>ACcurate COnsensus Reporting Document (ACCORD) explanation and elaboration: guidance and examples to support reporting consensus methods. <i>PLOS Med</i>. 2024;21:e1004390. doi:10.1371/journal.pmed.1004390</p>
<sup>1</sup>Oxford PharmaGenesis, Oxford, UK, chris.winchester@pharmagenesis.com; <sup>2</sup>Independent Medical Communications Professional, Oxfordshire, UK; <sup>3</sup>Bodleian Libraries, University of Oxford, Oxford, UK; <sup>4</sup>Medical Affairs + Health Impact, AbbVie, North Chicago, IL, US; <sup>5</sup>Dartmouth Institute for Health Policy &amp; Clinical Practice (TDI), Geisel School of Medicine, Dartmouth College, Hanover, NH, US; <sup>6</sup>School of Kinesiology, Department of Medicine, University of British Columbia, Vancouver, British Columbia, Canada; <sup>7</sup>Leiden University Medical Centre, Leiden, the Netherlands; <sup>8</sup>OPEN Health Communications, London, UK.</p>
<h4>Conflict of Interest Disclosures </h4>
<p>Christopher C. Winchester is an employee, director, and shareholder of Oxford PharmaGenesis; a director of Oxford Health Policy Forum CIC; a trustee of the Friends of the National Library of Medicine; and an associate fellow of Green Templeton College, University of Oxford. Mark J. Rolfe is an employee of Oxford PharmaGenesis. William T. Gattrell was an independent medical communications professional at the time of this study and is currently an employee of Bristol Myers Squibb. Keith Goldman is an employee of AbbVie. Niall Harrison is an employee of OPEN Health Communications.</p>
<h4>Funding/Support </h4>
<p>Medical writing, editorial, and project management support for this abstract were provided by Oxford PharmaGenesis.</p>
<h4>Acknowledgments</h4>
<p> Medical writing support was provided by Alison Chisholm, with editorial support from Jenny Thorp, and administrative support was provided by Mehraj Ahmed, Ryan Gamble, and Jessica Miller.</p>
<h4>Additional Information </h4>
<p>No authors were reimbursed for participating in the initiative.</p>


poster

<p>Aidan G. Cashin,<sup>1,2</sup> Harrison J. Hansford,<sup>1,2</sup> Miguel A. Hernán,<sup>3,4,5</sup> Sonja A. Swanson,<sup>3,4,6</sup> Hopin Lee,<sup>7,8</sup> Matthew D. Jones,<sup>1,2</sup> Issa J. Dahabreh,<sup>3,4,5,9</sup> Barbra A. Dickerman,<sup>3,4</sup> Matthias Egger,<sup>10,11,12</sup> Xabier Garcia-Albeniz,<sup>3,13</sup> Robert M. Golub,<sup>14</sup> Nazrul Islam,<sup>15,16</sup> Sara Lodi,<sup>3,17</sup> Margarita Moreno-Betancur,<sup>18,19</sup> Sallie-Anne Pearson,<sup>20</sup> Sebastian Schneeweiss,<sup>21</sup> Melissa K. Sharp,<sup>22</sup> Jonathan A. C. Sterne,<sup>12,23,24</sup> Elizabeth A. Stuart,<sup>25</sup> James H. McAuley<sup>1,2</sup></p>
<h4>Objective</h4>
<p><i> </i>When randomized trials are unavailable or not feasible, data from observational studies can be used, if assumptions hold, to answer causal questions about the comparative effects of interventions by emulating a hypothetical pragmatic randomized trial (target trial). The reporting of studies that emulate a target trial is inconsistent and often incomplete, limiting transparency and reproducibility. To address this knowledge gap, we developed consensus-based guidance for reporting analyses of observational data that aim to estimate causal effects by explicitly emulating a target trial.</p>
<h4>Design</h4>
<p> The TARGET (Transparent Reporting of Observational Studies Emulating a Target Trial)<sup>1</sup> guideline was developed using the Enhancing the Quality and Transparency of Health Research (EQUATOR) framework.<sup>2</sup> This included (1) a systematic review<sup>3</sup> of reporting practices in published studies that explicitly aimed to emulate a target trial, (2) a 2-round online survey (August 2023-March 2024; 18 expert participants from 6 countries) to generate agreement on the importance of candidate items selected from previous research and to identify additional items, (3) a 3-day expert consensus meeting (June 2024; 18 panelists) to refine the scope of the guideline and draft the checklist, and (4) an internal and external review and piloting activity with stakeholders and potential users (n = 108; September 2024-February 2025). The checklist was then refined based on feedback and finalized.</p>
<h4>Results</h4>
<p> The 21-item TARGET checklist was organized into 6 sections (abstract, introduction, methods, results, discussion, other information) (<b>Table 25-0970</b>). The TARGET guideline was intended to be general, with a focus on nonrandomized studies of interventions explicitly emulating a parallel-group, individually randomized target trial, with adjustment for baseline confounders. Key recommendations were to (1) summarize the causal question and the reason for emulating a target trial; (2) clearly specify the target trial protocol (ie, causal estimand, identifying assumptions, data analysis plan) and how these components were mapped to the observational data; and (3) for each causal estimand, report the estimate obtained and its precision, along with findings from additional analyses to assess the sensitivity of estimates to assumptions and design and analysis choices.</p>

![](https://assets.underline.io/markdown_image/1/image/1044ea19b2369c6f6dadcd84058920d2.png)

<h4>Conclusions</h4>
<p> The TARGET guideline provides recommendations on reporting of studies explicitly emulating a target trial. Use of the guideline should facilitate transparency in reporting to improve peer review and help researchers, clinicians, and other readers interpret and use the results.</p>
<h4>References</h4>
<p>1. Hansford HJ, Cashin AG, Jones MD, et al. Development of the TrAnsparent ReportinG of observational studies Emulating a Target trial (TARGET) guideline. <i>BMJ Open</i>. 2023;13(9):e074626. doi:10.1136/bmjopen-2023-074626</p>
<p>2. Moher D, Schulz KF, Simera I, Altman DG. Guidance for developers of health research reporting guidelines. <i>PLoS Med</i>. 2010;7(2):e1000217. doi:10.1371/journal.pmed.1000217</p>
<p>3. Hansford HJ, Cashin AG, Jones MD, et al. Reporting of observational studies explicitly aiming to emulate randomized trials. <i>JAMA Netw Open</i>. 2023;6(9):e2336023. doi:10.1001/jamanetworkopen.2023.36023</p>
<sup>1</sup>School of Health Sciences, Faculty of Medicine &amp; Health, University of New South Wales, Sydney, Australia, a.cashin.neura.edu.au; <sup>2</sup>Centre for Pain IMPACT, Neuroscience Research Australia, Sydney, Australia; <sup>3</sup>CAUSALab, Harvard T.H. Chan School of Public Health, Boston, MA, US; <sup>4</sup>Department of Epidemiology, Harvard T.H. Chan School of Public Health, Boston, MA, US; <sup>5</sup>Department of Biostatistics, Harvard T.H. Chan School of Public Health, Boston, MA, US; <sup>6</sup>Department of Epidemiology, University of Pittsburgh, Pittsburgh, PA, US; <sup>7</sup>University of Exeter Medical School, Exeter, UK; <sup>8</sup>IQVIA, London, UK; <sup>9</sup>Richard A. and Susan F. Smith Center for Outcomes Research, Beth Israel Deaconess Medical Center and Harvard Medical School, Boston, MA, US; <sup>10</sup>Institute of Social and Preventive Medicine, University of Bern, Bern, Switzerland; <sup>11</sup>Centre for Infectious Disease Epidemiology and Research, Faculty of Health Sciences, University of Cape Town, Cape Town, South Africa; <sup>12</sup>Department of Population Health Sciences, Bristol Medical School, University of Bristol, Bristol, UK; <sup>13</sup>RTI Health Solutions, Barcelona, Spain; <sup>14</sup>Northwestern University Feinberg School of Medicine, Chicago, IL, US; <sup>15</sup>Oxford Population Health, Big Data Institute, University of Oxford, Oxford, UK; <sup>16</sup>Faculty of Medicine, University of Southampton, Southampton, UK; <sup>17</sup>Department of Biostatistics, Boston University School of Public Health, Boston, MA, US; <sup>18</sup>Clinical Epidemiology &amp; Biostatistics Unit, Murdoch Children’s Research Institute, Royal Children’s Hospital, Parkville, Melbourne, VIC, Australia; <sup>19</sup>Department of Paediatrics, The University of Melbourne, Parkville, Australia; <sup>20</sup>School of Population Health, Faculty of Medicine &amp; Health, UNSW Sydney, Sydney, Australia; <sup>21</sup>Division of Pharmacoepidemiology, Department of Medicine, Brigham and Women’s Hospital, Harvard Medical School, Boston, MA, US; <sup>22</sup>Department of Public Health and Epidemiology, School of Population Health, RCSI University of Medicine and Health Sciences, Dublin, Ireland; <sup>23</sup>NIHR Bristol Biomedical Research Centre, Bristol, UK; <sup>24</sup>Health Data Research UK South-West, Bristol, UK; <sup>25</sup>Department of Mental Health, Johns Hopkins Bloomberg School of Public Health, Baltimore, MD, US.</p>
<h4>Conflict of Interest Disclosures </h4>
<p>Miguel A. Hernán is an advisor to ProPublica and Adigens Health, in which he owns equity, and a member of ADIA Lab’s advisory board. Issa J. Dahabreh is a consultant to Moderna on work related to target trial emulation. Sebastian Schneeweiss is participating in investigator-initiated grants to Brigham and Women’s Hospital from Boehringer Ingelheim, Takeda, and UCB unrelated to the topic of this study; is an advisor to and owns equity in Aetion, a software manufacturer; and is an advisor to Temedica, a patient-oriented data generation company.</p>
<h4>Funding/Support</h4>
<p> This work received no direct funding. The TARGET consensus meeting was supported through proceeds from a course on causal inference (December 4-7, 2023; Sydney, Australia). Aidan G. Cashin was supported by an Australian National Health and Medical Research Council (NHMRC) Investigator Grant (2010088). Harrison J. Hansford was supported by an Australian NHMRC Postgraduate Scholarship (2021950), a PhD Top-Up Scholarship from Neuroscience Research Australia, and was a Neuroscience Research Australia PhD Pearl sponsored by Sandra Salteri, AO. Miguel A. Hernán was funded by a National Institutes of Health (NIH) grant (R37 AI102634). Issa J. Dahabreh was supported by a Patient-Centered Outcomes Research Institute (PCORI) award (ME-2021C2-22365) and an NIH award (R01HL136708). Barbra A. Dickerman was supported by a grant from the NIH (R00 CA248335). Matthias Egger was supported by grants from the NIH (R01 AI152772-01, 5U01-AI069924-05) and the Swiss National Science Foundation (32FP30-174281). Nazrul Islam was supported by grants from the National Institute for Health and Care Research (NIHR; HDRUK2022.0313) and the UK Office for National Statistics (2002563). Margarita Moreno-Betancur was supported by an Australian NHMRC Investigator Grant (2009572). Sebastian Schneeweiss was supported by the NIH (R01-HL141505, R01-AR080194) and the US Food and Drug Administration (HHSF223201710146C). Jonathan A. C. Sterne was supported by the NIHR Bristol Biomedical Research Centre and by Health Data Research UK. Elizabeth A. Stuart was supported by grants from the NIH (R01 MH126856) and PCORI (ME-2020C3-21145). James H. McAuley was supported by an Australian NHMRC Investigator Grant (2010128).</p>
<h4>Role of the Funder/Sponsor </h4>
<p>The funders had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation review, or approval of the abstract; and decision to submit the abstract for publication.</p>
<h4>Additional Information </h4>
<p>Aidan G. Cashin and Harrison J. Hansford are co–first authors.</p>
<h4>Acknowledgment </h4>
<p>We thank and acknowledge the contributions of all participants of the internal and external piloting who were not compensated for their contributions.</p>


Transparent Reporting of Observational Studies Emulating a Target Trial: The TARGET Guideline

<p>Mette B. Engmose,<sup>1,2</sup> An-Wen Chan,<sup>3</sup> Kerry Dwan,<sup>4</sup> Carsten Hinrichsen,<sup>5</sup> Asbjørn Hróbjartsson,<sup>1,2</sup> David Moher,<sup>6</sup> Matthew J. Page,<sup>7</sup> Larissa Shamseer,<sup>8</sup> Lesley A. Stewart,<sup>4</sup> Camilla H. Nejstgaard<sup>1,2</sup></p>
<h4>Objective </h4>
<p>Systematic review protocols often adhere incompletely to the Preferred Reporting Items for Systematic review and Meta-Analysis Protocols (PRISMA-P) 2015.<sup>1</sup> Reporting guidelines need to be regularly updated to incorporate methodological developments, author feedback, and changed research context. We aimed to inform the update of PRISMA-P 2015<sup>2</sup> by exploring protocol authors’ practices and experiences with the reporting guideline.</p>
<h4>Design </h4>
<p>In a cross-sectional study, we investigated adherence to PRISMA-P 2015 and interviewed protocol authors. We randomly sampled 100 systematic review protocols from May 2021 to May 2024: 50 protocols published in PubMed-indexed journals and 50 protocols uploaded to or registered in OSF/PROSPERO. Two authors independently assessed the 26 PRISMA-P 2015 items as fully, partially, or not reported (or not applicable). We analyzed protocol adherence overall and at the item level. We invited 43 corresponding authors from our sample of 100 systematic review protocols for interviews on their experiences with using PRISMA-P 2015. The selection of authors was based on maximum variation and snowball sampling and personal network. We used a piloted, semistructured interview guide involving 3 predefined themes: level of experience, views on using PRISMA-P 2015, and reflections on the guideline’s strengths and weaknesses. We applied framework analysis to the interview transcripts. The study was reported according to Standards for Reporting Qualitative Research (SRQR).<sup>3</sup></p>
<h4>Results </h4>
<p>The PubMed-indexed protocols fully adhered to a median of 60% (range, 36%-80%) of PRISMA-P 2015 items. The equivalent median for OSF/PROSPERO protocols was 45% (range, 25%-88%). In both types of protocols, items not adhered to in more than 25% of the protocols were related to protocol amendments (item 4), role of the funder (item 5c), methods for planned summary other than quantitative (item 15d), meta-bias(es) (item 16), and confidence in the cumulative evidence (item 17) (<b>Table 25-1033</b>). In both types of protocols, 11 items partially adhered to in more than 25% of the protocols were lacking, eg, descriptions of procedures for data selection from multiple reports of the same study (item 11c). Of the 43 invited authors, 15 (9 men, 6 women) participated in the interviews. From the predefined themes, several suggestions for the PRISMA-P 2015 update emerged. Some suggestions regarded adding or modifying existing content, eg, to report conflicts of interest or how to report data synthesis when no meta-analysis was planned; other suggestions were more generic, eg, to add links to the Elaboration &amp; Explanation paper.</p>

![](https://assets.underline.io/markdown_image/1/image/8a19657504ae87562517e7d38d9665ea.png)

<h4>Conclusions</h4>
<p> Adherence to PRISMA-P 2015 was inadequate in 50 systematic review protocols from PubMed-indexed journals and especially in 50 protocols from OSF/PROSPERO. Aspects often not adhered to were protocol amendments, role of the funder, methods for planned summary other than quantitative, meta-bias(es), and confidence in the cumulative evidence. The interviewed authors suggested several modifications of the guideline. Findings from this study will inform the update of PRISMA-P 2015.</p>
<h4>References</h4>
<p>1. Frost AD, Hróbjartsson A, Nejstgaard CH. Adherence to the PRISMA-P 2015 reporting guideline was inadequate in systematic review protocols. <i>J Clin Epidemiol</i>. 2022;150:179-187. doi:10.1016/j.jclinepi.2022.07.002</p>
<p>2. Nejstgaard CH, Shamseer L, Chan A-W, et al. Updating the PRISMA-P reporting guideline for systematic review protocols. OSF. April 10, 2024. <a href="https://osf.io/2znc5/"><span class="Hyperlink CharOverride-6">https://osf.io/2znc5/</span></a></p>
<p>3. O’Brien BC, Harris IB, Beckman TJ, et al. Standards for reporting qualitative research: a synthesis of recommendations. <i>Acid Med</i>. 2014;89(9):1245-1251. doi:10.1097/acm.0000000000000388</p>
<sup>1</sup>Cochrane Denmark &amp; Centre for Evidence-Based Medicine Odense (CEBMO), Department of Clinical Research, University of Southern Denmark, Odense, Denmark, mengmose@health.sdu.dk; <sup>2</sup>Open Patient data Exploratory Network (OPEN), Odense University Hospital, Odense, Denmark; <sup>3</sup>Department of Medicine, Women’s College Research Institute, University of Toronto, Toronto, Ontario, Canada; <sup>4</sup>Centre for Reviews and Dissemination, University of York, York, UK; <sup>5</sup>National Institute of Public Health, University of Southern Denmark, Odense, Denmark; <sup>6</sup>Centre for Journalology, Clinical Epidemiology Program, Ottawa Hospital Research Institute, Ottawa, Ontario, Canada; <sup>7</sup>Methods in Evidence Synthesis Unit, School of Public Health and Preventive Medicine, Monash University, Melbourne, Australia; <sup>8</sup>Knowledge Translation Program, Li Ka Shing Knowledge Institute, St Michael’s Hospital, Unity Health Toronto, Toronto, Ontario, Canada.</p>
<h4>Conflict of Interest Disclosures </h4>
<p>An-Wen Chan, Kerry Dwan, Asbjørn Hróbjartsson, David Moher, Matthew J. Page, Larissa Shamseer, Lesley A. Stewart, and Camilla H. Nejstgaard are members of the steering group for the update of the PRISMA-P 2015 reporting guideline. All other authors declare no conflicts of interest related to financial interests, activities, relationships, and affiliations, including employment, affiliation, funding and grants received or pending, consultancies, honoraria or payment, speakers’ bureaus, stock ownership or options, expert testimony, royalties, donation of medical equipment, or patents planned, pending, or issued. An-Wen Chan and David Moher are members of the Peer Review Congress Advisory Board but were not involved in the review or decision for this abstract.</p>
<h4>Funding/Support </h4>
<p>This project did not receive any external funding; therefore, design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the abstract; and decision to submit the abstract for presentation were not influenced by any funder or sponsor.</p>


Author Practices and experiences With PRISMA-P 2015

<p>Cath Chamberlain,<sup>1</sup> Peter Craig,<sup>2</sup> Luis Gabriel Cuervo,<sup>3</sup> Omar Dewidar,<sup>4</sup> H. N. Ellingwood,<sup>5</sup> Elizabeth Ghogomu,<sup>6</sup> Billie-Jo Hardy,<sup>7</sup> Tanya Horsley,<sup>8</sup> Sonya Faber,<sup>9</sup> Cindy Feng,<sup>10</sup> Damian Francis,<sup>11</sup> Sarah Funnell,<sup>12</sup> Alison Krentel,<sup>9</sup> Janet Jull,<sup>12</sup> Elizabeth Kristjansson,<sup>13</sup> Julian Little,<sup>9</sup> Loveline Lum Niba,<sup>14</sup> Tamara Kredo,<sup>15</sup> Zack Marshall,<sup>16</sup> Lawrence Mbuagbaw,<sup>17</sup> Michael Johnson Mahande,<sup>18</sup> Stuart Nicholls,<sup>19</sup> Miriam Nkangu Nguilefem,<sup>9</sup> Ekwaro Obuku,<sup>20</sup> Oyekola Oloyede,<sup>21</sup> Ebenezer Owusu-Addo,<sup>22</sup> Kevin Pottie,<sup>23</sup> Jacqueline Ramke,<sup>24</sup> Alison Riddle,<sup>6</sup> Anita Rizvi,<sup>13</sup> Janet Hatcher Roberts,<sup>9</sup> Larissa Shamseer,<sup>25</sup> Melissa Sharp,<sup>26</sup> Janice Tufte,<sup>27</sup> Peter Tugwell,<sup>28</sup> Xiaoqin Wang,<sup>29</sup> Laura Weeks,<sup>30</sup> Charles Wisonge,<sup>31</sup> Luke Wolfenden,<sup>32</sup> Taryn Young,<sup>33</sup> Vivian Welch<sup>6</sup></p>
<h4>Objective</h4>
<p> Advancing health equity requires improvements in the conduct, analysis, and reporting of health research. Many observational studies are relevant to health equity because they use data that can advance our understanding of health inequities, including the systemic structures that perpetuate them. The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) guideline for the reporting of observational studies is highly endorsed by journal editors; however, the consideration of health equity has not been explicitly addressed in the STROBE statement. As a result, editors have been limited in their ability to promote the reporting of health equity data and considerations. The STROBE-Equity extension provides actionable guidance for equity-relevant observational studies to fill this gap.</p>
<h4>Design </h4>
<p>The development of STROBE-Equity adhered to a published protocol, informed by the Enhancing the Quality and Transparency of Health Research (EQUATOR) Network’s methodological framework including a scoping review of existing guidance, methods study, key informant interviews, and a global online survey. The project team was diverse across multiple domains, including lived experience of health inequity, gender, career stage, age, ethnicity, Indigenous background, disciplines, place of residence, and different decision-making roles. We recruited team members from our international network of patients, researchers, editors, and practitioners who had prior involvement in observational studies or expertise in reporting guidelines, biostatistics, knowledge translation, and patient engagement. We applied an integrated knowledge translation approach to (1) assess the reporting of health equity in published observational studies through a scoping review of available guidance, methodological review of observational studies, and key informant interviews; (2) seek wide international input on draft items; and (3) establish consensus amongst knowledge users and researchers through a 2-day consensus meeting. The development of the reporting guideline occurred between January 2020 and February 2024, with consensus achieved in May 2024.</p>
<h4>Results </h4>
<p>Our process identified 10 extension items for the STROBE statement to offer guidance for reporting of health equity (<b>Table 25-1136</b>). The items involve the following: (1) facilitating the identification of health equity relevance, (2) describing the purpose of the study in relation to health equity, (3) describing team composition, (4) engagement with individuals experiencing inequities relevant to the condition of interest, (5) sampling and recruitment of relevant populations, (6) participant selection, (7) analytical methods, (8) describing participant flow for relevant populations, (9) describing the characteristics of study participants, and (10) interpretation of study findings in relation to health equity.</p>
<a href="/wp-content/uploads/2025_conference/tables/prc25-1136t1.png"> <img loading="lazy" src="/wp-content/uploads/2025_conference/tables/prc25-1136t1.png"> </a>
<h4>Conclusions </h4>
<p>The use of the STROBE-Equity extension alongside the original STROBE statement aims to promote transparent reporting of health equity data and considerations in observational research. Journal editors endorsing the use of this guideline can help advance health equity through improved reporting, ultimately contributing to research that informs more equitable policies and interventions.</p>
<sup>1</sup>University of Melbourne, Australia; <sup>2</sup>MRC/CSO Social and Public Health Sciences Unit, School of Health and Wellbeing, University of Glasgow, UK; <sup>3</sup>Pan American Health Organization, Washington, DC, US; <sup>4</sup>University of Toronto, Canada; <sup>5</sup>Carleton University, Canada; <sup>6</sup>Bruyère Health Research Institute, Canada; <sup>7</sup>Dalla Lana School of Public Health, University of Toronto, Canada; <sup>8</sup>Royal College of Physicians and Surgeons, Canada; <sup>9</sup>School of Epidemiology and Public Health, University of Ottawa, Canada; <sup>10</sup>Dalhousie University, Canada; <sup>11</sup>PhD/School of Health and Human Performance, Center for Health and Social Issues, Georgia College, Milledgeville, GA, US; <sup>12</sup>Queens University, Canada; <sup>13</sup>School of Psychology, University of Ottawa, Canada; <sup>14</sup>Department of Public Health, Faculty of Health Sciences, The University of Bamenda, Cameroon; <sup>15</sup>South Africa Medical Research Council, South Africa; <sup>16</sup>Department of Community Health Sciences, Cumming School of Medicine, University of Calgary and O’Brien Institute for Public Health, University of Calgary, Canada; <sup>17</sup>McMaster University, Canada; <sup>18</sup>Kilimanjaro Christian Medical Centre, Kenya; <sup>19</sup>Centre for Practice-Changing Research, Ottawa Hospital Research Institute, Canada; <sup>20</sup>Africa Centre for Systematic Reviews and Knowledge Translation, Kampala, Uganda; <sup>21</sup>Sefako Makgatho Health Sciences University, South Africa; <sup>22</sup>Bureau of Integrated Rural Development, College of Agriculture and Natural Resources, Ghana; <sup>23</sup>Dalhousie University, Canada; <sup>24</sup>International Centre for Eye Health, London School of Hygiene &amp; Tropical Medicine, London, UK; <sup>25</sup>Knowledge Translation Program, Li Ka Shing Knowledge Institute, Unity Health Toronto, Canada; <sup>26</sup>Department of Public Health &amp; Epidemiology, School of Population Health, Ireland; <sup>27</sup>Catalyst for Positive Change, US; <sup>28</sup>Department of Medicine, Faculty of Medicine, University of Ottawa, Canada; <sup>29</sup>University of Ottawa Heart Institute, Canada; <sup>30</sup>Canada’s Drug Agency, Ottawa, Canada; <sup>31</sup>South Africa Medical Research Council, South Africa; <sup>32</sup>University of Newcastle, Australia; <sup>33</sup>Stellenbosch University, South Africa; vwelch@campbellcollaboration.org. </p>
<h4>Conflict of Interest Disclosures</h4>
<p> Kevin Pottie, Lawrence Mbuagbaw, Vivian Welch, and Peter Tugwell declare they are co-convenors of the Cochrane Health Equity Thematic Group. No other disclosures were reported. </p>
<h4>Funding</h4>
<p> This work was funded by the Canadian Institutes of Health Research. The funder had no role in the project design, implementation, or decision to publish. Vivian Welch is funded by a Public Health Agency of Canada and Canadian Institute of Health Research Applied Public Health Chair award. Kevin Pottie is funded by a Dalhousie University Faculty of Medicine Chair in Family Medicine.</p>
<h4>Acknowledgment</h4>
<p> The authors thank all the respondents to the global survey.</p>


Development of a reporting guideline on health equity in observational research (STROBE-Equity)

<p>Stephana Julia Moss,<sup>1</sup> Juliane Kennett,<sup>2</sup> Jeanna Parsons Leigh,<sup>3</sup> Niklas Bobrovitz,<sup>4</sup> Henry T. Stelfox<sup>5</sup></p>
<h4>Objective </h4>
<p>To map the evidence for factors (eg, research practices) associated with the reproducibility of methods and results reported in health sciences research.</p>
<h4>Design </h4>
<p>Five bibliographic databases were searched from January 2000 to May 2023, followed by supplemental searches of high-impact journals and relevant records. We included health science records of observational, interventional, or knowledge synthesis studies reporting data on factors related to research reproducibility. Factors were operationalized as modifiable or nonmodifiable aspects of study conduct relating to individual-, study-, or institutional-level practices, methods, and processes that could impact the reproducibility of research methods or results.<sup>1</sup> Reproducibility was operationalized by 2 mutually exclusive categories: (1) methodological reproducibility (ie, the ability to exactly repeat the methods, including study procedures and data analysis) and (2) results reproducibility (ie, obtaining corroborating results using the same or similar methods).<sup>2</sup> We included studies that used surrogate measures for reproducibility (eg, type 1 or 2 error rates) if they (1) explicitly stated their aim to investigate the reproducibility of research and (2) rationalized their choice of surrogate measure.<sup>3</sup> Data were coded using inductive qualitative content analysis, and empirical evidence was synthesized with evidence and gap maps. Study risk of bias was assessed using the Quality in Prognostic Studies risk-of-bias tool. Statistical tests of the association between factors and reproducibility outcomes were summarized as reported in the included articles.</p>
<h4>Results </h4>
<p>Our review included 148 primarily biomedical and preclinical (n = 62) and clinical (n = 71) studies. Factors were classified into 12 modifiable (eg, sample size and power) and 3 nonmodifiable (eg, publication year) categories. Of 234 reported evaluations of factors, 76 (32%) assessed methodological reproducibility and 158 (68%) assessed results reproducibility. The most frequently reported factor was transparency and reporting (38 of 234 assessments [16%]). A total of 155 factors (66%) were evaluated for statistical associations with reproducibility outcomes (<b>Table 25-0858</b>). Statistical associations were most frequently conducted for analytical methods (24 of 26 reporting significance [92%]), sample size and power (21 of 23 reporting significance [91%]), and participant characteristics and study materials (10 of 12 reporting significance [83%]). Risk-of-bias assessments found low risk of bias for study participation, factor measurement, and statistical analysis, and high risk of bias for confounding.</p>

![](https://assets.underline.io/markdown_image/1/image/b16148b83c9aa6aaf32a839e2bc713d9.png)

<h4>Conclusions </h4>
<p>Our review identified a large body of literature consisting primarily of observational studies of factors associated with the reproducibility of health sciences research. The data suggest that reproducibility may be improved by implementing more stringent statistical testing procedures and thresholds, sample size and power calculations, and improved transparency and completeness of reporting. Experimental studies are needed to test interventions to improve reproducibility. Factors identified in this study with consistent observational support should be prioritized for experimentation. Factors that affect reproducibility in health and social care services and population and public health need to be identified given the paucity of data in these areas.</p>
<h4>References</h4>
<p>1. Goodman SN, Fanelli D, Ioannidis JP. What does research reproducibility mean? <i>Sci Transl Med</i>. 2016;8(341):341ps12-341ps12. doi:10.1126/scitranslmed.aaf5027</p>
<p>2. Niven DJ, McCormick TJ, Straus SE, et al. Reproducibility of clinical research in critical care: a scoping review. <i>BMC Med</i>. 2018;16:1-12. doi:10.1186/s12916-018-1018-6</p>
<p>3. Clemens MA. The meaning of failed replications: a review and proposal. <i>J Econ Surveys</i>. 2017;31(1):326-342. doi:10.1111/joes.12139</p>
<sup>1</sup>Faculty of Medicine, Dalhousie University, Halifax, Nova Scotia, Canada, sj.moss@dal.ca; <sup>2</sup>Department of Critical Care Medicine, University of Calgary, Calgary, Alberta, Canada; <sup>3</sup>Faculty of Health, Dalhousie University, Halifax, Nova Scotia, Canada; <sup>4</sup>Department of Emergency Medicine, University of Calgary, Calgary, Alberta, Canada; <sup>5</sup>Faculty of Medicine &amp; Dentistry, University of Alberta, Edmonton, Alberta, Canada.</p>
<h4>Conflict of Interest Disclosures</h4>
<p> None reported.</p>
<h4>Funding/Support</h4>
<p> This work was funded by the Canadian Institutes of Health Research.</p>
<h4>Role of the Funder/Sponsor </h4>
<p>The funding body had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the abstract; or decision to submit the abstract for presentation.</p>


Factors Associated with the Reproducibility of Health Sciences Research: A Systematic Review and Evidence and Gap Map

<p>Ayu Putu Madri Dewi,<sup>1</sup> Nicholas J. DeVito,<sup>2</sup> Gowri Gopalakrishna,<sup>1,3</sup> Inge Stegeman,<sup>4,5</sup> Mariska Leeflang<sup>1</sup></p>
<h4>Objective</h4>
<p> To improve research reproducibility, referees of grant proposals could assess the expected reproducibility of a proposed project. A simple Open Science checklist may assist referees in assessing whether referees are more capable of predicting the reproducibility of research proposals with an Open Science checklist than without one.<sup>1</sup></p>
<h4>Design</h4>
<p> This is a nonrandomized controlled trial and a comparative prediction study<sup> </sup>using a minimum of 100 granted research proposals and their corresponding published outputs (α = 5%, power = 80%, correlation = 0.6).<sup>2</sup> The study focuses on empirical research with quantitative data across diverse fields. A mock grant review was started on April 7, 2025, with a minimum of 4 referees divided into 2 groups: one using an Open Science checklist and the other not. Each proposal will be assessed under both conditions (<b>Figure 25-0934</b>). The checklist, developed in a previous study<sup>3</sup> and adapted for grant proposal evaluation, serves as the intervention. Referees are blinded to final study outcomes to minimize bias. The primary outcome is predictive accuracy, assessed using the area under the curve (AUC) from receiver operating characteristic (ROC) analysis. Referees’ predictions on the actual reproducibility of the published studies will be compared. Open Science scores are numerical (0-14), while reproducibility outcomes are binary (yes or no). The AUC from logistic regression will be used and visualized in ROC curves. Comparative predictive accuracy will be calculated for both groups. Sensitivity analyses will explore variations in checklist scoring and study reproducibility across different research fields. A database including grant proposal data, Open Science scores, reproducibility status, and analytical code in R will be shared in an open repository. We hypothesize that referees without the checklist will predict reproducibility no better than random chance (with an AUC of approximately 0.50), while those using the checklist will achieve an AUC of approximately 0.65, indicating improvement while acknowledging that the checklist is not a perfect tool.</p>

![](https://assets.underline.io/markdown_image/1/image/17200ac9fe2e7dc26b84dba55df1facc.png)

<h4>Results</h4>
<p> As of June 5, 2025, a total of 15 mock grant reviewers have completed 89 mock reviews across 60 unique grant proposals: 46 reviews in the checklist group and 43 in the control group. Twenty-nine proposals were reviewed in both groups, forming a fully paired subset. Analysis is ongoing.</p>
<h4>Conclusions</h4>
<p> We will share our results with funding agencies to explore integrating the Open Science checklist into grant review processes to enhance reproducibility.</p>
<h4>References</h4>
<p>1. Parker TH, Griffith SC, Bronstein JL, et al. Empowering peer reviewers with a checklist to improve transparency. <i>Nat Ecol Evol</i>. 2018;2(6):929-935. doi:10.1038/s41559-018-0545-z</p>
<p>2. Fraser H, Bush M, Wintle BC, et al. Predicting reliability through structured expert elicitation with the repliCATS (Collaborative Assessments for Trustworthy Science) process. <i>PLoS One</i>. 2023;18(1):e0274429. doi:10.1371/journal.pone.0274429</p>
<p>3. Dewi APM, Rethlefsen M, Schroter S, et al. Measuring the efficacy of an intervention to improve reproducibility of scientific manuscripts: study protocol for a randomized controlled trial. OSFHOME. October 31, 2024. Accessed July 2, 2025. <a href="https://osf.io/b5g6y"><span class="Hyperlink CharOverride-6">https://osf.io/b5g6y</span></a></p>
<sup>1</sup>Amsterdam UMC, Epidemiology and Data Science Department, Amsterdam, The Netherlands; <sup>2</sup>Bennett Institute for Applied Data Science, Nuffield Department of Primary Care Health Sciences, University of Oxford, Oxford, UK, nicholas.devito@phc.ox.ac.uk; <sup>3</sup>Department of Epidemiology, Faculty of Health, Medicine, and Life Sciences Maastricht University, The Netherlands; <sup>4</sup>Department of Otorhinolaryngology and Head &amp; Neck Surgery University Medical Center Utrecht, The Netherlands; <sup>5</sup>Brain Center, University Medical Center Utrecht, The Netherlands.</p>
<h4>Conflict of Interest Disclosures </h4>
<p>None reported.</p>


Perception of Open Science Practices on Reproducibility Among Reviewers of Grant Proposals at Research Funding Organizations

<p>Kyungwan Hong,<sup>1</sup> Mark Basista,<sup>1</sup> Tony Tse<sup>1</sup></p>
<h4>Objective </h4>
<p>Information about how adverse events (AEs) were specified, collected, and assessed for clinical trials (ie, AE methods) is essential to accurately interpret the safety profiles of studied interventions.<sup>1</sup> ClinicalTrials.gov provides a structured format for trialists to report aggregate results information and AE methods that complement the published literature. Prior work compared reporting of AEs in publications and ClinicalTrials.gov in terms of completeness and consistency,<sup>2</sup> examined differences between sources in reporting AE methods for systematic collection,<sup>3</sup> and assessed AE methods reporting in publications and identified methodologic weaknesses.<sup>1</sup> This cross-sectional analysis explored the extent that AE methods reported in ClinicalTrials.gov complemented corresponding publications.</p>
<h4>Design </h4>
<p>ClinicalTrials.gov was searched on August 1, 2024, for drug and biologic trials with primary completion dates between January 18, 2017, and August 1, 2021, to allow 3 years for results posting and publication. A total of 9659 records were initially retrieved. After randomization, the records were reviewed sequentially until a convenience sample of 40 trials with summary results posted on ClinicalTrials.gov and corresponding primary results articles in PubMed were identified. The identified trials were then analyzed. All authors reviewed and extracted text for 5 ClinicalTrials.gov AE method data elements: collection period (time frame), vocabulary source for AE terms (how AE terms were standardized), collection approach (systematic or nonsystematic), analysis population, and whether participants withdrew due to harms. Each set of trial results was independently reviewed by 2 authors, with discrepancies resolved by consensus.</p>
<h4>Results </h4>
<p>Collection period (28 [70.0%]), analysis population (27 [67.5%]), and withdrawal for harm (35 [87.5%]) were described for over half of the 40 sampled trials in ClinicalTrials.gov and/or publications (<b>Table 24-0831</b>). However, consistency varied; for example, of the 35 trials with number of participants withdrawn for harm described in both sources, 20 (57.1%) provided inconsistent information. Collection approach was described in both sources infrequently (4 [10.0%]). Other AE methods (eg, vocabulary source) were reported for some trials in only one or neither source. </p>

![](https://assets.underline.io/markdown_image/1/image/570e4b44fadda46a96c1215d9717877c.png)

<h4>Conclusions </h4>
<p>A few AE methods were not reported on ClinicalTrials.gov and/or publications, indicating a need for improvement. Complete AE methods reporting is needed for accurate safety profiles of study interventions and unbiased risk-benefit assessments. Even when available, reported AE methods were frequently inconsistent between sources, potentially due to variations in time frames used for AE reporting. Limitations included excluding supplementary appendices and study protocols that may provide additional methodologic details on AE methods. Future research is needed to review publicly available study protocols for AE methods information, compare the quality of AE methods reporting by source, and characterize trials in which AE methods have and have not been completely reported (eg, by funding source, study design).</p>
<h4>References</h4>
<p>1. Phillips R, Hazell L, Sauzet O, Cornelius V. Analysis and reporting of adverse events in randomised controlled trials: a review. <i>BMJ Open</i>. 2019;9(2):e024537. doi:10.1136/bmjopen-2018-024537</p>
<p>2. Krešo A, Grahovac M, Znaor L, Marušić A. Safety reporting in trials on glaucoma interventions registered in ClinicalTrials.gov and corresponding publications. <i>Sci Rep</i>. 2024;14(1):27762. doi:10.1038/s41598-024-79394-z</p>
<p>3. Mayo-Wilson E, Fusco N, Li T, Hong H, Canner JK, Dickersin K; MUDS Investigators. Harms are assessed inconsistently and reported inadequately part 1: systematic adverse events. <i>J Clin Epidemiol</i>. 2019;113:20-27. doi:10.1016/j.jclinepi.2019.04.022</p>
<sup>1</sup>National Center for Biotechnology Information, National Library of Medicine, National Institutes of Health, Bethesda, MD, US, kyungwan.hong@nih.gov.</p>
<h4>Conflict of interest Disclosures </h4>
<p>Kyungwan Hong reported being employed by the National Institutes of Health (NIH) as a policy analyst with the ClinicalTrials.gov program. Mark Basista reported conducting this work while contracting with ICF. Tony Tse reported being employed by the NIH as an analyst with the ClinicalTrials.gov program. No other disclosures were reported.</p>
<h4>Funding/Support</h4>
<p> This work was supported by the National Center for Biotechnology Information of the National Library of Medicine, NIH.</p>
<h4>Role of the Funder/Sponsor </h4>
<p>The funder was not directly involved in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the abstract for presentation.</p>
<h4>Disclaimer </h4>
<p>The views expressed in this article are those of the authors and do not necessarily reflect the views or policies of the NIH.</p>


Characterizing Adverse Event Methods Reported in ClinicalTrials.gov and Publications

<p>Randall J. Ellis,<sup>1</sup> Chirag J. Patel<sup>1</sup></p>
<h4>Objective </h4>
<p>New omics modalities hold promise for biomarker discovery for disease prediction. Accessible biobank samples have fueled a massive literature, some of which may be irreproducible, be subject to lack of generalizability, or report inflated results due to data leakage,<sup>1</sup> ie, spurious relationships between input and target variables that arise as artifacts of data collection, sampling, or preprocessing. These biases frequently result in models developed in one context not generalizing to real-world contexts. Here, we demonstrated and quantified bias in biomedical multiomic machine learning results due to irrelevant or unreported baselines, data leakage, and inadequate performance metrics reporting.</p>
<h4>Design </h4>
<p>In November 2024, using data from the UK Biobank, we conducted a cohort study of all 607 disease outcomes defined in the dataset, using blood plasma proteomics data (collected from 2006-2010) and demographics data. We demonstrated how simple demographics (age, sex, education) compare competitively with putatively novel models that incorporate an expansive number of omics input features, how randomly chosen omics features compare with those chosen according to data-driven feature selection methods, the impact of data leakage on performance (eg, normalizing the data before making train-test cross-validation splits), and how presenting the area under the receiver operating characteristic curve (AUROC) gives a biased view of performance. We presented additional case studies of using baseline risk calculators for cardiovascular disease and atherosclerotic cardiovascular disease to assess how omics factors compared with traditional risk scores for associations with 10-year risk of heart disease or stroke. We used a stratified cross-validation approach and assessed the AUROC, sensitivity, specificity, and positive and negative predictive value.</p>
<h4>Results </h4>
<p>Across a majority of 607 disease outcomes, demographic baselines performed competitively (0%-10% difference in mean AUROC) in comparison with the combination of demographics and omics factors when looking at the variability of performance across cross-validation folds, which are underreported in the literature. We found that data leakage influenced predictive performance (mean AUROC increases of 5%-30%). Accounting for disease prevalence and class balance played a significant role in the interpretation of machine learning results, particularly for positive predictive value. Omics factors contributed marginal increases in performance for cardiovascular disease and atherosclerotic cardiovascular disease compared with traditional risk score calculators (less than 5% mean AUROC).</p>
<h4>Conclusions </h4>
<p>The robustness and transparency of biobank-based omics research is enhanced if several practices are followed: (1) rigorous and relevant baselines would provide critical tests of candidate models to justify their clinical use and value, (2) preventing data leakage would preclude inflated results, and (3) comprehensive performance metrics reporting would give transparent measures of progress for researchers in the omics community. Our results complement other work showing the impact of data leakage<sup>1</sup> and the marginal improvements of plasma proteomics beyond demographic baselines.<sup>2</sup></p>
<h4>References</h4>
<p>1. Kapoor S, Narayanan A. Leakage and the reproducibility crisis in machine-learning-based science. <i>Patterns</i>. 2023;4(9):100804. doi:10.1016/j.patter.2023.100804</p>
<p>2. Deng YT, You J, He Y, et al. Atlas of the plasma proteome in health and disease in 53,026 adults. <i>Cell</i>. 2025;188(1):253-271. doi:10.1016/j.cell.2024.10.045</p>
<sup>1</sup>Department of Biomedical Informatics, Harvard Medical School, Boston, MA, US, chirag_patel@hms.harvard.edu.</p>
<h4>Conflict of Interest Disclosures </h4>
<p>None reported.</p>
<h4>Funding/Support </h4>
<p>This work was supported by grants from the National Institute on Aging (RF1AG074372) and National Library of Medicine (5T15LM007092-33).</p>
<h4>Role of the Funder/Sponsor </h4>
<p>The funders were not involved in the development or implementation of this study.</p>
<h4>Acknowledgment </h4>
<p>We acknowledge the support of our funders and Audrey Airaud for early-stage input on these experiments.</p>


Bias in Machine Learning Associated With Weak Baselines, Data Leakage, and Inadequate Measures Reporting

<p>Edith Otalike,<sup>1</sup> Mike Clarke,<sup>2</sup> Ngianga-Bakwin Kandala,<sup>1</sup> Joel J. Gagnier<sup>1,3</sup></p>
<h4>Objective</h4>
<p> Individual-participant data meta-analysis (IPD-MA) is regarded as the criterion standard in evidence synthesis, but it is resource intensive. While there is consensus on reporting items, a consensus-based tool for the critical appraisal of the methodological quality of IPD-MA does not currently exist. We undertook a systematic methodology review as the initial phase in the development of a critical appraisal checklist. This review collated and summarized the available methodological guidance on IPD-MA of randomized and observational studies. </p>
<h4>Design</h4>
<p> We followed the guidelines for Cochrane Methodology Reviews and reported following the PRISMA 2020 guidance. We performed an electronic search of MEDLINE, Embase, CINAHL, Web of Science Scopus, Cochrane Methodology Registry, CONSORT Database of Methodological Papers, Health Technology Assessment Review Database, <i>Research Synthesis Methods</i>, and <i>Journal of the Royal Statistical Society</i> covering publications from 1946 to June 2024. We included studies published in English that addressed any methodological guidance and essential statistical and software requirements for IPD-MA. Data extraction focused on study characteristics, domain of the review process, and the specific recommendations. Risk of bias was assessed using resources relevant to the study design. A thematic synthesis was performed to group recurring themes into domains. For each domain, signalling questions were generated to develop a preliminary checklist for assessment and refinement in a 2-round e-Delphi survey involving international IPD-MA experts.</p>
<h4>Results</h4>
<p> The literature search yielded 13,589 citations. After screening 9436 unique abstracts and reviewing 286 full texts, we included 130 articles that met our inclusion criteria. These articles consisted of narrative reviews, handbooks, critical reviews, empirical studies, and statistical method articles. They were published between 1995 and 2024, with most originating from the UK (62 [48%]), the US (20 [15%]), and the Netherlands (16 [12%]) and 50 (38%) originating from 10 other countries. Most of these studies had a low risk of bias. We identified 14 domains of guidance for conducting and reporting of IPD-MA, and we categorized them into 5 sections (<b>Table 25-0999</b>). This finding informed the initial version of the checklist to be evaluated in the e-Delphi survey.</p>

![](https://assets.underline.io/markdown_image/1/image/ae40b29d7f5b93606f06cb13df104504.png)

<h4>Conclusions</h4>
<p> There are many recommendations in the literature on the general conduct of IPD-MA and on specific aspects of this research, which would benefit from consensus recommendations for all aspects of IPD-MA and critical appraisal of reports. This review provides many suggestions for these recommendations, and our e-Delphi survey will seek consensus on items to include in the critical appraisal tool, which we expect to be completed in 2025.</p>
<sup>1</sup>Department of Epidemiology &amp; Biostatistics, Schulich School of Medicine &amp; Dentistry, Western University, London, Ontario, Canada, eotalike@uwo.ca; <sup>2</sup>Northern Ireland Methodology Hub, Queen’s University Belfast, Northern Ireland, UK; <sup>3</sup>Department of Surgery, Schulich School of Medicine &amp; Dentistry, Western University, London, Ontario, Canada.</p>
<h4>Conflict of Interest Disclosures</h4>
<p> Edith Otalike receives internal funding from Western University and the Dean’s Research Scholarship award. No other disclosures were reported.</p>
<h4>Additional Information</h4>
<p> Joel J. Gagnier is a co–corresponding author (jgagnie4@uwo.ca).</p>


Individual-Participant Data Meta-Analysis Methodological Guidance: A Systematic Review

<p>Luis C. L. Correia,<sup>1</sup> Rafael F. Mascarenhas,<sup>2</sup> Felipe S. C. de Menezes,<sup>2</sup> Jeronimo S. Oliveira Júnior,<sup>2</sup> Marcus V. Almeida,<sup>2</sup> Caio F. Azevedo,<sup>2</sup> Naieli M. de Andrade,<sup>2</sup> Viola Vaccarino,<sup>1</sup> Joseph S. Ross,<sup>3</sup>< Joshua D. Wallach<sup>1</sup></p>
<h4>Objective</h4>
<p> Several approaches exist for selecting confounders for adjustment in observational studies, including subjective judgment, statistical criteria, or the use of a causal model, usually represented by directed acyclic graphs (DAGs). We assessed the characteristics and trends in the reported methods for selecting confounders to control for in observational studies published in the highest impact factor medical and epidemiological journals.</p>
<h4>Design </h4>
<p>We identified the 10 highest Impact Factor medical (n = 5) and epidemiological (n = 5) journals according to InCites Journal Citation Reports. For each journal, we reviewed all PubMed-indexed articles published in 2003, 2013, and 2023 to identify observational studies evaluating exposure-outcome relationships in which confounder adjustment would be expected. We excluded articles that were descriptive, predictive, and quasi-experimental (ie, that primarily address confounding through design-based approaches). We randomly selected half of the articles in each journal and publication year for full-text evaluation and identified key study characteristics. We classified the methods reported by each study to select confounders: no confounder adjustment; adjustment but confounders not specified; confounders selected without justification; confounders selected based on an established association with the outcome; confounders selected based on statistical criteria (ie, imbalance between exposure groups, change-in-estimate strategy, or application of a stepwise regression); or confounders selected based on a causal model, either depicted by a DAG or explained in the text. We followed the STROBE reporting guideline for cross-sectional studies.</p>
<h4>Results</h4>
<p> We identified 623 eligible observational studies, including 197 (31.6%) published in medical journals and 426 (68.4%) published in epidemiological journals. Of these, 22 (3.5%) did not report adjusting for confounders, 18 (2.9%) did not specify which confounders were selected, 281 (45.1%) reported selection of confounders without justification, 139 (22.3%) reported selection of confounders based on an established association with the outcome, 121 (19.4%) reported selection of confounders based on statistical criteria, and 42 (6.7%) reported selection of confounders based on a causal model (35 used a DAG and 7 provided an explanation in their text) (<b>Table 25-1006</b>). The selection of confounders without justification remained relatively stable between 2003 and 2023 (from 111 of 228 [48.7%] to 68 of 164 [41.5%]), while the use of a causal model to identify confounders increased (from 0 of 228 to 37 of 164 [22.6%]). Differences in the methods used to select confounders were observed across journal type and study design, but not exposure type or funding source.</p>

![](https://assets.underline.io/markdown_image/1/image/b3e83419c11953083cbf73a2ad1b02b7.png)

<h4>Conclusions</h4>
<p> Although our analysis was limited to high-impact medical and epidemiological journals, we found that the reporting of causal models—such as DAGs—has increased over time. However, fewer than one-fourth of studies reported such models in 2023, raising concerns about how confounders are selected and justified in observational research.</p>
<sup>1</sup>Department of Epidemiology, Rollins School of Public Health, Emory University, Atlanta, GA, US, joshua.wallach@emory.edu; <sup>2</sup>Bahiana School of Medicine and Public Health, Salvador, BA, Brazil; <sup>3</sup>Department of Medicine, Yale School of Medicine, New Haven, CT, US.</p>
<h4>Conflicts of Interest</h4>
<p> Joseph S. Ross reported receiving grants from the US Food and Drug Administration; Johnson &amp; Johnson; the Medical Device Innovation Consortium; the Agency for Healthcare Research and Quality; the National Heart, Lung, and Blood Institute; and Arnold Ventures outside the submitted work and is also is an expert witness at the request of relator attorneys, the Greene Law Firm, in a qui tam suit alleging violations of the False Claims Act and Anti-Kickback Statute against Biogen Inc that was settled in September 2022. Dr Ross is a Deputy Editor at <i>JAMA</i>. Joshua D. Wallach is supported by Arnold Ventures, Johnson &amp; Johnson through the Yale Open Data Access project, and the National Institute on Alcohol Abuse and Alcoholism of the National Institutes of Health under award 1K01AA028258 and previously served as a consultant to Hagens Berman Sobol Shapiro LLP and Dugan Law Firm APLC. Dr Wallach is a Specialty Associated Editor at <i>JACC</i>.</p>


Reporting of Confounder Selection in Observational Studies in High Impact Medical and Epidemiological Journals, 2003-2023

<p>Ashkan Hashemi,<sup>1</sup> Isaac Dreyfus,<sup>2</sup> Nicholas Varunok,<sup>3</sup> John Burton,<sup>4</sup> Sina Rashedi,<sup>5</sup> Farbod Zahedi Tajrishi,<sup>6</sup> Behnood Bikdeli<sup>5</sup></p>
<h4>Objective </h4>
<p>The role of noninferiority trials in cardiovascular research remains unclear. We sought to determine the proportion of noninferiority vs superiority trials, their likelihood of meeting primary outcomes, and their academic impact and online engagement.</p>
<h4>Design</h4>
<p> We reviewed cardiovascular randomized controlled trials (RCTs) published in <i>The New England Journal of Medicine</i>, <i>JAMA</i>, and <i>The Lancet</i> between January 2014 and December 2019. Cardiovascular trials were defined based on review by 2 independent reviewers for topics that included patients with cardiovascular diseases or tested cardiovascular outcomes. Trials were categorized by design (superiority vs noninferiority), and assessed for primary outcome success, academic impact (Google Scholar citations), and online engagement (Altmetric Attention Score). Linear trends were assessed via linear regression. An interaction term was tested between trial design (superiority vs noninferiority) and outcome success (meeting the primary outcome or not) with respect to citation count.</p>
<h4>Results</h4>
<p> Among 429 cardiovascular RCTs, 100 (23.3%) were noninferiority trials, with their proportion increasing over time (<i>P</i> = .009). Noninferiority trials were significantly more likely to meet their primary outcome than superiority trials (79% vs 54%;<i> P </i>&lt; .001). Median citations were comparable between noninferiority and superiority trials (560 [IQR, 248-851] and 505 [IQR, [274-886], respectively; <i>P</i> = .83), as were Altmetric Attention Scores (226 [IQR, 124-481] and 234 [IQR, 120-542], respectively; <i>P</i> = .70) (<b>Figure 25-1150</b>). Regardless of design, trials that met their primary outcome had higher citation counts than those that did not (superiority trials, 615 [IQR, 280-1200] vs 423 [IQR, 267-734], respectively; <i>P</i> = .003; noninferiority trials, 613 [IQR, 261-998] vs 338 [IQR, 201-635], respectively; <i>P</i> = .049). There was no significant interaction between trial design and outcome success with respect to citation counts (<i>P</i> for interaction = .87).</p>

![](https://assets.underline.io/markdown_image/1/image/f38f193f9c584e236deb6401cf062bab.png)

<h4>Conclusions</h4>
<p> Noninferiority trials are increasingly shaping cardiovascular research and are more likely to meet their primary outcome, with academic engagement and online attention comparable to superiority trials. The higher success rate observed in noninferiority trials may, in part, reflect the discretionary—and at times overly permissive—approach to noninferiority margin selection, warranting further study. Finally, Altmetric Attention Scores reflect online attention but not clinical impact; future work will assess influence on guidelines and US Food and Drug Administration labeling.</p>
<sup>1</sup>Weill Cornell Medicine, New York, NY, US; <sup>2</sup>David Geffen School of Medicine, University of California, Los Angeles, CA, US; <sup>3</sup>Vanderbilt University Medical Center, Nashville, TN, US; <sup>4</sup>Keck School of Medicine, University of Southern California, Los Angeles, CA, US; <sup>5</sup>Brigham and Women’s Hospital, Harvard Medical School, Boston, MA, US, bbikdeli@bwh.harvard.edu; <sup>6</sup>Tulane University School of Medicine, New Orleans, LA, US.</p>
<h4>Conflict of Interest Disclosures</h4>
<p> None reported.</p>


Downloads

Next from The 10th Peer Review Congress

Transparent Reporting of Observational Studies Emulating a Target Trial: The TARGET Guideline

Similar lecture

Transparent Reporting of Observational Studies Emulating a Target Trial: The TARGET Guideline