Spain

In the absence of gold standard for evaluating quality of peer review, considerable attention has been focused on studying reviewer agreement via inter-rater reliability (IRR) which can be thought of as the correlation between scores of different reviewers given to the same grant proposal. Noting that it is not uncommon for IRR in grant peer review studies to be estimated from some range-restricted subset of submissions, we use statistical methods and data analysis of real peer review data to illustrate behavior of such local IRR estimates when only fractions of top-quality proposal submissions are considered. We demonstrate that local IRR estimates are smaller than those obtained from all submissions and that zero local IRR estimates are quite plausible. We note that, from a measurement perspective, when reviewers are asked to differentiate among grant proposals across the whole range of submissions, only IRR measures that correspond to the complete range of submissions are warranted. We recommend against using local IRR estimates in those situations. Moreover, if review scores are intended to be used for differentiating among top proposals, we recommend peer review administrators and researchers to align review procedures with their intended measurement.

PEERE

Local versus global inter-rater reliability for evaluating the internal validity of grant peer review: Considerations of measurement

assessing review quality

grant peer review

inter-rater reliability

# Welcome everyone to ACL 2022!

The 60th Annual Meeting of the Association for Computational Linguistics is taking place May 22-27, 2022 as a hybrid event, in Dublin and online. We are happy to welcome all of you to this anniversary edition with an almost 50-50 in-person and virtual participation. 
The main conference program features oral presentations, in-person and virtual posters and demo sessions, a plenary session for our best paper presentations and awards, three amazing keynote events and two new initiatives of invited talks: Spotlight Talks for Young Rising Stars (STIRS) and The Next Big Idea Talks. Posters (including Findings of ACL 2022) and demos are grouped by areas for both the in-person and the virtual sessions. For the virtual component, the talks will be on Zoom and the posters and the demos will be in GatherTown. The Student Research Workshop will have an oral session and a poster session as part of Poster Session 1. The program also features eight Tutorials and 28 Workshops. 

 
We wish you a wonderful conference! 
[**The ACL 2022 Organizing Committee**](https://www.2022.aclweb.org/organisers)
 
[**Conference Handbook**](https://drive.google.com/file/d/1_BUCMfhMVrjG9E2e71aHdHeE28KSje0l/view?usp=sharing) 
[**Mini Handbook**](https://drive.google.com/file/d/1qlBKl0wzmlVF1oCeMQl3BahLd9nLP5Ce/view?usp=sharing) 
[**Posters and Demo guides**](https://drive.google.com/file/d/1UucMAoCNncIOaH1rMMDa0owuG9qgvJTG/view?usp=sharing)

ACL 2022

The Association for Computational Linguistics (ACL) is the premier international scientific and professional society for people working on computational problems involving human language, a field often referred to as either computational linguistics or natural language processing (NLP). 

technical paper

Following the success of the 1st PEERE International Conference on Peer Review, the PEERE COST Action follow-up community organised the second edition of the PEERE International Conference on Peer Review  and was celebrated virtually on 29 September – 1 October 2020.

The conference welcomed any contribution (e.g., original research, position papers, literature reviews) that used any method (e.g., quantitative, experimental or qualitative) to investigate peer review in a variety of scientific domains (e.g., scholarly journals, funding agencies, research assessment). Interdisciplinary research is especially welcome.

The program is published here: http://tiny.cc/peere2020 


The second edition of the PEERE International Conference on Peer Review aims to provide a forum for scholars, practitioners and science stakeholders to share research, models, tools and experience on peer review in different fields, e.g., medicine, computer science, social sciences and humanities. It aims to stimulate the use of evidence-based research in the design and implementation of peer review in a variety of fields and encourage more systematic research.

In peer review, reviewers often have formal and informal occasions to interact with other reviewers of the same grant proposal or submission. The former happens, for example, when grant proposals are evaluated by sitting review panels who are tasked with collectively deciding the fate of a particular proposal. Despite the scholarly consensus that interactions (and thus social influence) can alter the outcome of collective decision making, to date social influence has rarely been a theoretical ingredient in simulation models of peer review. In our contribution we use a simple agent-based model of peer review to show that social influence processes can indeed change the dynamics and outcome of peer review. With our simulation experiment we showcase the incorporation of social influence processes in peer review models as a promising direction for future research.

Does social influence matter in peer review?

This paper describes the pilot project to introduce a new Open Data policy to three IOP Publishing journals as part of IOP Publishing’s commitment to increasing transparency and support for open science. For the largest and longest established of these journals an investigation was undertaken to understand the impact of this change on authors and the editorial staff by examining changes in the revised submission processing time and how often manuscripts were returned to the author. The paper identifies an overall increase in revised manuscript submission processing at the start of the trial that is influenced by the method used by authors to share data. The rate of un-submitting manuscripts was also found to be increased showing an impact on authors. The paper also shows that the processing time quickly returned to normal levels once mitigating actions were taken. The increase in the rate at which manuscripts were returned to the author was also reduced but not completely removed.

Introducing a Data Accessibility Policy for journals at IOP Publishing: Measuring the impact on authors and editorial teams

Peer review is a cornerstone of the science system. Previous studies have found that peer review in the social sciences primarily prompted changes in the theoretical framing of papers and contextualisation of results, while the methodological sections were largely untouched, with associated patterns of change in reference lists. We ask whether this focus on theory over methods also occurs in other fields. To assess this, we matched preprints from Dimensions with published articles from Web of Science, PLOS and Scopus to identify 6,002 preprint-publication pairs across all fields. We then compared their reference lists and calculated the proportion of retained, added and removed references, and also identified in which sections of publications the references were cited. We found that the medical sciences added and removed significantly more references between preprint and publication than did the natural sciences, which had more bibliographically stable manuscripts. By proportion, the introduction and discussion sections underwent the most bibliographic change in both fields, however by ratio of added to unchanged references, the methodology and results section had greater change. Our results suggest there are important differences between fields in how reference lists change between preprint and publication, and that the focus of theory over methodology in the social sciences may not extend to the natural sciences or medical sciences.

Changing the shoulders I am standing on: Describing the changes that occurred in publications' reference lists after peer review

Preprints – generally defined as a publicly available version of a research manuscript prior to formal assessment and publication via a journal-mediated peer review process – are becoming a common feature in some research communities. Accelerated sharing of research findings, creating a citable research output to show evidence of research productivity, creating opportunities for community feedback and collaboration, and increased attention and citation are some of the ways preprints can benefit researchers (References 1-4). Preprints also create opportunities for publishers seeking to address pain points associated with lengthy peer review processes. To this end, since October 2018, Research Square has collaborated with Springer Nature to offer In Review, a preprint service that integrates preprint posting with ongoing journal peer review and provides transparency into the peer review process. In Review helps Nature authors rapidly share their research findings while also providing journal-mediated quality control checks on the posted preprint and transparency into the peer-review process. 

1) https://jamanetwork.com/journals/jama/fullarticle/2670247
2) https://arxiv.org/abs/1805.05238
3) https://arxiv.org/abs/astro-ph/0411275
4) https://elifesciences.org/articles/52646

In Review on Research Square: a journal-integrated preprint system for Springer Nature

Sentiment analysis methods have been developed to automatically extract opinions from texts, e.g. online reviews. Here we explore the idea of adopting sentiment analysis to extract grant reviewers' opinions. This paper examines the relationship between the sentiments of grant reviews and reviewed proposals’ funding decisions made by agencies. We define peer reviews’ prediction as to what degree reviews’ positiveness/negativeness, via sentiment extraction, can indicate proposals’ chance of success in receiving funding. Building on a corpus of peer review texts and related documents from an Irish science funding agency, we conduct sentiment extraction by manual and two simple automated coding (TextBlob and VADER). A comparison between the three results has been made. We find the manual and automated coding results to be overall consistent, but manual coding is still the most accurate and reliable for peer review studies. Furthermore, the three coding results show strong prediction of proposals’ funding decisions, which indicates that reviewers' opinions are largely adopted by agencies.

Peer Reviews' Prediction in Proposals' Funding Success: A Sentiment Analysis of Grant Reviews

Peer review is often criticized for being flawed, subjective and biased, but research into peer review is still lacking studies on peer review reports. In our study, in which text-analysis software was used to determine the linguistic characteristics of 472,449 peer review reports we compared a range of characteristics (including analytical tone, authenticity, clout, three measures of sentiment, and morality) as interaction of reviewer recommendation, area of research, type of peer review and reviewer gender. Our main conclusion was that the reviewer recommendation had the biggest impact on the linguistic characteristics of reports, and that area of research, type of peer review and reviewer gender had little or no impact. The lack of influence of research area, type of review or reviewer gender on the linguistic characteristics is a sign of the robustness of peer review.

Large-scale language analysis of peer review reports

About 90% of researchers overall thought the main area of effectiveness for peer review would be in improving the published paper by providing constructive feedback to authors. This begs the question of what constructive feedback is, and even more, how current annotation tools (e.g. Acrobat Reader) can be enhanced with facilities towards quality feedback. We present Review&Go, a Chrome’s browser extension which supports manuscript reviewing lifecycle . Review&Go is available for download at the Chrome’s Web Store.

Software scaffolds for quality feedback in peer review

Peer-review represents the main mechanism of selecting for publishing high-quality content with the purpose of maintaining the quality, validity, and originality of scientific publications. However as peer-review is being performed by humans, it is prone to errors and bias which might underline its incapability to fulfill its purpose. Therefore, we propose to study the peer-review as a cooperation problem that depends heavily on the level of competition between individuals and their interests. We build a new agent-based model where both scientists and journals co-evolve in a competitive environment, but where public good benefits are granted for high-effort scientists. We extend the work of Righi and Takacs (2017) by looking at emergence in a broad set of potential author and reviewer strategies, as well as by exploring the editorial policies that appear from the evolutionary process.
We build an agent-based model to study various scenarios and how they can impact the overall scientific quality of publications. Thereby, we explore how different editorial strategies with various levels of strictness improve or deteriorate the commitment of reviewers, or how different cost of producing high-quality work
changes the attitude of scientists. Moreover, we observe which editorial strategies would survive in the evolution process, both in the case of co-evolution or independent evolution.
We obtained a working agent-based model that produced clear and robust results. Our baseline model can achieve both high-quality papers and reviews from agents, which proves that the introduction of public good benefits stimulates individual commitment toward peer review. The results remain valid even when we considered a different set-up for the scientific social network or when we increased the costs of producing good papers. We also observed that extreme levels of competition would alter the system as agents would start to act according to their interests. Lastly, we saw that editorial strategies converge even in the case of chaotic systems.

Journal Competition and the Sustainability of Peer review: An agent based model

Background: Peer review is regarded as the gold standard decision-making approach for research funding. Around the globe, health research funders use peer review to inform fair and objective decisions to advance science. However, peer review is not without challenges. A large body of evidence ranging from journal articles, opinion letters, editorials, conference abstracts, thesis and blogs, report problems associated with peer review. These problems can be anything from high administrative costs, low reliability, potential biases (e.g. gender, career status), lack of equity, unreliable scoring and ranking, which can then contribute to a high burden to stakeholders involved in the application and decision-making system (1-5). Strategic priorities of the National Institute for Health Research (NIHR), one of the largest UK funders of health research, have endorsed and highlighted the need for further action on improving their peer review during the decision-making process for the allocation of research funding (6, 7). 

Aim: The aim of this study is to identify which elements of the peer review process and/or alternative approaches, support research funding allocation decision-making, for whom, in what contexts, and how. By conducting a realist synthesis, we will be better placed to understand which approaches and/or elements are likely to enhance the decision-making process for the allocation of research funding. 

Methods: A realist synthesis will provide an explanatory analysis of peer-review or alternative approaches to funding allocation that work in specific contexts, such as for different funding organisations, charities, Higher Education Institutes (HEIs), and research disciplines. The search strategy is reported elsewhere (6). Database searches were originally conducted during May-June 2016 and updated to December 2018. Relevant sources were also searched through Google Scholar and included a citation analysis for included studies. Searches of grey literature as well as databases outside health (e.g. IEEE, arXiv.org) were conducted to ensure materials outside of mainstream academic journals were captured.

Results: Electronic searches returned 1824 publications (after removing duplicates) and manual searches returned 36, resulting in 1860 publications. Five researchers independently screened 50 publications each (abstract and/or full text) and inter-rater reliability of paired researchers for agreement to include or exclude reached 80% agreement based on initial eligibility criteria (i.e. studies of any design exploring interventions or approaches for allocating research funds). This agreement validated the first-stage inclusion/exclusion criteria and the remaining publications (n=1610) were screened according to these criteria. This resulted in 215 publications that were subjected iteratively in relation to programme theory statements (A. if there’s peer review (as a process) then there is bias & burden to all stakeholders B. if variations to the typical peer review process are introduced then bias/burden/costs could be reduced, C. Studies have elements of A&B). Two researchers then discussed the assessment of the 215 included publications and made final decisions on inclusion. A total of 111 publications were selected to inform the research questions. All researchers in the team were involved in data extraction. The following section provides a brief description of the initial findings of the synthesis as per the realist methodology. 

Evidence map
Overall, publications were found to fall into two categories: 
1. Studies that generated evidence on known issues with peer review (bias towards early career researchers, innovation, gender; ability of peer review to predict future value of research; burden for funders and researchers to manage and write applications) 
2. Studies elaborating the need to find approaches different from peer review that can be used to make funding decisions and/or that propose novel approaches. 
The extraction process is ongoing, and the current emerging findings are taken from an in-depth analysis of 30 included publications (out of the 111). What we know so far is that only a handful of interventions have measured in some form the efficiency of peer review in terms of time or cost efficiencies or which have implemented slight or radical alternatives to the current peer review process. 

Context mechanisms outcomes
From the 111 included publications, we developed a map of the Context, Mechanism and Outcomes (CMO), which is central in the realist methodology to answer the questions of for whom, in what context and how an intervention works. Our CMO map shows that the contexts in which the studies have taken place are funders and higher education institutions focusing on developing interventions to increase efficiencies. Other studies are situated in academic contexts where research communities have explored peer review processes from an experimental point of view to enhance the discipline of Research on Research. From the funder perspective, interventions that have been developed to enhance peer review fall into three categories: 
1. Incremental changes to the funding process (modifying the number of stages in applications, modifying reviewers’ numbers and changing the size of the application) 
2. Organisation-level changes creating funding streams to address bias or systems that aid the identification and management of peer reviewers 
3. Alternative approaches with the most salient approaches (sandpits, random allocation, golden tickets, Delphi exercises, and community-based grants). 
A brief review of these categories using the existing/emerging evidence is provided below. 



Incremental changes to peer review 
The National Institutes of Health (NIH) recommendations to streamline peer review include shorter applications with focus on impact and innovation, and less on methods and preliminary data, (8) as well as establishing a numerical quota for the number of awards made to new investigators (Kaiser, 2008b), which aimed to match the application success rates of new investigators to those of established investigators. The Australian National Health and Medical Research Council proposed having a more focused application to reduce preparation costs and improving as a result the quality of peer review by reducing workloads (9, 10). However, some changes can have unintended negative effects, demonstrated in two cross-sectional surveys studies that by shortening the application increased the applicants time spent on preparing applications (11). This is an indication that changes in one context may shift the burden onto another, and this needs to be considered when proposing changes to peer review processes. Other areas under consideration include improvements to fairness, effectiveness and robustness of peer review through masking applications (12) increasing transparency (13) or developing peer review criteria to meet the goals of a specific research programme or discipline (14).

Alternative approaches to decision-making 
Novel approaches to funding decision-making that substitute (at least partially) peer review have also been discussed or implemented. The common approaches in the literature include sandpits, partial random allocation of research funds and Delphi’s. Sandpits are interactive workshops involving 20 to 30 participants, a director, a team of experts and independent stakeholders. The group comes together for a multi-day residential workshop to discuss ideas and participate in interactive sessions with the purpose of establishing research collaborations and receive funds. The proposals in the workshop are evaluated in a similar way to funding panels (15). The basic premise of the allocation of funds by chance retain an aspect of peer review where all proposals submitted to a call are assessed against the research programme requirements, and those potentially fundable are randomly allocated funding (16). A modified Delphi’s (or consensus surveys) proved to be an efficient, transparent, and equitable method of reviewing novel grant applications in a specialized field of research, where no local expertise was available. This process may also be useful for other peer review processes, particularly where there is limited access to local experts (17).
Preliminary conclusion: The approaches and interventions identified ebb and flow within the literature, perhaps influenced by multiple external factors including funding availability and the overall research landscape. Therefore, research that generates empirical evidence has an important role in enabling funders to make informed decisions about whether and how peer review processes work best, and the contexts in which alternatives may offer viable alternatives that support decision-making processes. Findings from this synthesis would encourage long-term implementation of successful mechanisms to the allocation of research funding.

Keywords: peer review, science of science, realist synthesis
Acknowledgements: We would like to thank Geoff Frampton, Karen Pickett, Jo Picot and Jonathan Shepherd at the Southampton Health Technology Assessments Centre for their collaboration in this study. The search strategy for this realist synthesis extends the original search used in their systematic map and review. Discussions with the team at the start of the study strengthened the overall design of the review and supported the Adding Value in Research NIHR initiative. We would also like to thank Justin Jagosh, Director of the Centre for Advancement in Realist Evaluation and Synthesis at the University of Liverpool for an insightful Realist Evaluation and Synthesis workshop and for sharing his expertise with us during the design of our study.
References
1. Hodgson C. How reliable is peer review? An examination of operating grant proposals simultaneously submitted to two similar peer review systems. J Clin Epidemiol. 1997;50(11):1189-95.
2. Abdoul H, Perrey C, Tubach F, Amiel P, Durand-Zaleski I, Alberti C. Non-financial conflicts of interest in academic grant evaluation: a qualitative study of multiple stakeholders in France. PLoS ONE. 2012;7(4):e35247.
3. Bollen J, Crandall D, Junk D, Ding Y, Borner K. From funding agencies to scientific agency: Collective allocation of science funding as an alternative to peer review. EMBO Rep. 2014;15(2):131-3.
4. Bishop D. Nature INDEX. 2018. [cited 2019]. Available from: https://www.natureindex.com/news-blog/luck-of-the-draw.
5. Bollen J, Carpenter S, Lubchenco J, Scheffer M. Rethinking Resource Allocation in Science. arXiv preprint arXiv:190407645. 2019.
6. Shepherd J, Frampton GK, Pickett K, Wyatt JC. Peer review of health research funding proposals: A systematic map and systematic review of innovations for effectiveness and efficiency. PLoS ONE. 2018;13(5):e0196914.
7. Turner S, Bull A, Chinnery F, Hinks J, Mcardle N, Moran R, et al. Evaluation of stakeholder views on peer review of NIHR applications for funding: a qualitative study. BMJ Open. 2018;8(12):e022548.
8. Kaiser J. National Institutes of Health. Changes in peer review target young scientists, heavyweights. Science. 2008;320(5882):1404.
9. Herbert DL, Barnett AG, Graves N. Funding: Australia's grant system wastes time. Nature. 2013;495(7441):314.
10. Herbert DL, Graves N, Clarke P, Barnett AG. Using simplified peer review processes to fund research: a prospective study. BMJ Open. 2015;5(7):e008380.
11. Barnett AG, Graves N, Clarke P, Herbert D. The impact of a streamlined funding application process on application time: two cross-sectional surveys of Australian researchers. BMJ Open. 2015;5(1):e006912.
12. Ismail S, Farrands A, Wooding S. Evaluating grant peer review in the health sciences. A review of the literature. Cambridge, UK; 2009.
13. Gurwitz D, Milanesi E, Koenig T. Grant application review: the case of transparency. Plos Biology. 2014;12(12):e1002010.
14. Hawe P, Samis S, Di Ruggiero E, Shoveller JA. Population Health Intervention Research Initiative for Canada: progress and prospects. N S W Public Health Bull. 2011;22(1-2):27-32.
15. Guthrie S, Guérin B, Wu H, Ismail S, Wooding S. Alternatives to peer review in research project funding. Cambridge, UK; 2013.
16. Fang FC, Casadevall A. Research Funding: the Case for a Modified Lottery. mBio. 2016;7(2):e00422-16.
17. Holliday C, Robotin M. The Delphi process: a solution for reviewing novel grant applications. Int J Gen Med. 2010;3:225-30.



Decision-making approaches to grant funding allocation: insights from a realist synthesis

One of the central themes of studies on peer review decisions is whether the decisions select the best applicants. Some studies found that the best-performing unfunded applicants outperformed the funded applicants, and concluded that peer review is unable to select the best applicants. In contrast, our simulation model shows that we should be cautious when drawing conclusions from a comparison between funded applicants and best-performing unfunded applicants. Uncertainty in bibliometric statistics may cause the best-performing unfunded applicants to outperform the funded applicants even if peer review provides perfectly accurate assessments of the performance of applicants. Such observations are insufficient for concluding that peer review is not working properly. A deeper understanding of uncertainty in bibliometric statistics is needed before concrete conclusions can be drawn from comparisons between peer review outcomes and bibliometric statistics.

Local versus global inter-rater reliability for evaluating the internal validity of grant peer review: Considerations of measurement

Downloads

Next from PEERE

Does social influence matter in peer review?

Similar lecture

Inter-annotator agreement is not the ceiling of machine learning performance: Evidence from a comprehensive set of simulations

Local versus global inter-rater reliability for evaluating the internal validity of grant peer review: Considerations of measurement

.css-70qvj9{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-align-items:center;-webkit-box-align:center;-ms-flex-align:center;align-items:center;}Downloads

Next from PEERE

Does social influence matter in peer review?

Similar lecture

Inter-annotator agreement is not the ceiling of machine learning performance: Evidence from a comprehensive set of simulations

Downloads