Spain

&quot;Introduction
In this study, we tested the effect of receiving a short individual feedback report compared to a short general feedback report on the agreement between reviewers.

Methods
A total of 42 reviewers at the Norwegian foundation Dam were randomly assigned to receive either a general feedback report or an individual feedback report. The general feedback group received one report before the start of the reviews that contained general information about the previous call the reviewers took part in. In the individual feedback group, the reviewers received two reports, one before the review period (based on the previous call) and one during the period (based on the current call). In the individual feedback group, the reviewers were presented with detailed information on their scoring compared with the review committee as a whole. The main outcomes were the proportion of agreement in the eligibility assessment and the score difference between pairs of reviewers assessing the same application.

Results
A total of 2398 paired reviews were included in the analysis. There was no difference between the two groups in score difference. There was a significant difference between the two groups in the proportion of absolute agreement on whether the application was eligible for the funding program or not, with the general feedback group demonstrating a higher rate of agreement. However, the overall levels of agreement remained critically low in both groups and for both outcomes.

Conclusion
The results did not support our hypothesis but are in line with related studies in journal peer review. The use of a convenience sample and the limited sample size is are important limitations in this study. The low levels of agreement remain a major concern in grant peer review and research to identify factors contributing to this, as well as development and testing of interventions aiming to increase agreement rates, are still needed.

The study was preregistered at OSF.io/n4fq3&quot;

PEERE

Feedback to improve agreement in grant peer review

agreement

peer review

feedback

Following the success of the 1st PEERE International Conference on Peer Review, the PEERE COST Action follow-up community organised the second edition of the PEERE International Conference on Peer Review  and was celebrated virtually on 29 September – 1 October 2020.

The conference welcomed any contribution (e.g., original research, position papers, literature reviews) that used any method (e.g., quantitative, experimental or qualitative) to investigate peer review in a variety of scientific domains (e.g., scholarly journals, funding agencies, research assessment). Interdisciplinary research is especially welcome.

The program is published here: http://tiny.cc/peere2020 


The second edition of the PEERE International Conference on Peer Review aims to provide a forum for scholars, practitioners and science stakeholders to share research, models, tools and experience on peer review in different fields, e.g., medicine, computer science, social sciences and humanities. It aims to stimulate the use of evidence-based research in the design and implementation of peer review in a variety of fields and encourage more systematic research.

"Introduction
In this study, we tested the effect of receiving a short individual feedback report compared to a short general feedback report on the agreement between reviewers.

Methods
A total of 42 reviewers at the Norwegian foundation Dam were randomly assigned to receive either a general feedback report or an individual feedback report. The general feedback group received one report before the start of the reviews that contained general information about the previous call the reviewers took part in. In the individual feedback group, the reviewers received two reports, one before the review period (based on the previous call) and one during the period (based on the current call). In the individual feedback group, the reviewers were presented with detailed information on their scoring compared with the review committee as a whole. The main outcomes were the proportion of agreement in the eligibility assessment and the score difference between pairs of reviewers assessing the same application.

Results
A total of 2398 paired reviews were included in the analysis. There was no difference between the two groups in score difference. There was a significant difference between the two groups in the proportion of absolute agreement on whether the application was eligible for the funding program or not, with the general feedback group demonstrating a higher rate of agreement. However, the overall levels of agreement remained critically low in both groups and for both outcomes.

Conclusion
The results did not support our hypothesis but are in line with related studies in journal peer review. The use of a convenience sample and the limited sample size is are important limitations in this study. The low levels of agreement remain a major concern in grant peer review and research to identify factors contributing to this, as well as development and testing of interventions aiming to increase agreement rates, are still needed.

The study was preregistered at OSF.io/n4fq3"

technical paper

Introducing Elsevier

Introducing ReviewerCredits 

This talk highlights the main results achieved during the PEERE project from the Springer Nature perspective

Introducing Springer Nature

Background: The poor representation of women and of researchers from low-income and middle-income countries (LMICs) as authors, reviewers, and editorial advisors compromises the quality of the global medical literature, and is a threat to research integrity. The Lancet Group of journals are implementing initiatives to aim to redress the imbalance. 
Methods: We established a baseline of the peer-reviewer and author populations for Lancet journals in terms of gender and country of origin for 2018. In consultation with editors and experts, we developed a set of initiatives to improve the inclusion of women and of LMIC colleagues as authors, reviewers, and editorial advisors. 
Results: At baseline, women comprised about one-third of the authors and about a quarter of peer reviewers at Lancet journals. This is despite an almost 75% female editorial workforce, challenging previous evidence of editor homophily as a way of improving diversity in peer reviewers. Three commitments were made in 2019: (1) aim for at least 50% women and LMIC contributors for Series and Commissions, and exercise positive action in diversifying selected authors for commissioned content; (2) aim to invite 50% women as peer reviewers for all submissions, and recruit additional female statistical advisors; (3) meet a minimum of 50% female editorial board members for all Lancet journals by 2020. To complement these commitments to change in our editorial and peer review processes, the editors made public pledges in the form of a Diversity Pledge and No All-Male Panel Policy. Regular tracking of the impact of these initiatives is being undertaken, and 1-year results and the lessons learned, will be reported. 
Conclusions: Editors at the Lancet have implemented a series of diversity initiatives to increase inclusion of women and LMIC colleagues, aiming to reduce the waste of research resources and improve the quality, representation, and integrity of the evidence base in medicine and global health. Lessons learned can assist other journals and organisations who may wish to adopt gender equity and diversity policies.

Improving gender equity and diversity at Lancet journals

To increase transparency during policy-making process and to stimulate the conversation between stakeholders, the Ministry of Science and Technology (MOST) in Taiwan has held a series of forums and seminars since 2017 to invite researchers and relevant stakeholders to attend, and this action has become one of the approaches of communication in policy-making process in these two years. This paper aims to analyze the questions or comments raised by the participants in these forums to understand what is the most important issue in Taiwanese academia, and the response strategies from the funding organization to illustrate how responsible peer review could be achieved in the future.  Based on the result, the general funding mechanism can be considered the most concerning issue and especially the one with peer review process. The researchers questioned the fairness of single-blind peer review, the criteria of assessment and the quality of reviewing.  Hence, the MOST has implemented action plan to improve the quality and aimed to create the consensus on assessment criteria and standard, including organizing the consensus meeting of reviewers, building the evaluation mechanism of reviewers to assuring the quality, providing the example of good comment, allowing applicants to nominate the black-list of reviewers in written review stage, enlarging the pool of reviewers, providing the comments to applicants and revealing the name of reviewers partially after the result notification. Since the key element of peer review is degree of transparency, the more open, the more responsible peer review may be expected. This action plan in education or consensus buildup during peer review process can be the important approach to improve the quality under the single-blind or double-blind peer review mechanism by emphasizing accountability.

How to improve the quality of grant proposal review? -The exercise of responsible peer review in Taiwan

For peer review to be sustainable the voluntary effort of reviewing should ideally be shared more or less equally within the community. An unequal distribution of effort could reveal practices of over-exploitation and unfairness, potentially undermining the standards of quality of peer review. Unfortunately, across-journal data are rarely accessible to map such a distribution on a large scale. We used a complete dataset from 157 scholarly journals in various fields of research from various publishers, including more than 1 million authors, and measured the ratio between submissions and provided reviews by the same authors. We found that the 'exploiter' identikit is not the expected one but the following: younger scholars with a few co-authors, submitting preferably to journals of highest impact factors. This can be due either to the ''publish or perish'' culture emphasizing the importance of publications or the editors' tendency of relying more on senior reviewers.

Man, senior, targeting prestigious journals: Is this the identikit of the big fish in the peer review pond? An empirical study on 157 scholarly journals

"In the era of data-driven studies, statistics become more important than ever. Political, economic, environmental, or public health decisions – among many others– may be influenced by the results of a given published work. So, these results need to be trustable, reproducible, and replicable.
However, we are experiencing a lack of trust in statistical methods and, particularly, in statistical significance, as shown by many papers and journal editorials that have highlighted the danger of p-values. But p-values are not dangerous themselves. As we see it, the problem is the correct application of this technique as well as any other statistical approach (no mather if Bayesian, frequentist of fiducial).
In this sense, the process of peer review becomes more important than ever, and specialized reviewers with some statistical background are needed to take care of any paper that is proposed to be published. But How can journals take care of this process? Do the editors consider statical advice? Are they taking into account the reproducibility and replicability of the papers? 
In this work, we study if it is possible to measure if there is an improvement in terms of statistical content in a paper during the peer-review process and provide a series of preliminary results about a dataset of nine journals with five article types from The Royal Society."

Feedback to improve agreement in grant peer review

Downloads

Next from PEERE

Introducing Elsevier

Similar lecture

Metrics and peer review agreement at the institutional level

Feedback to improve agreement in grant peer review

.css-70qvj9{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-align-items:center;-webkit-box-align:center;-ms-flex-align:center;align-items:center;}Downloads

Next from PEERE

Introducing Elsevier

Similar lecture

Metrics and peer review agreement at the institutional level

Downloads