Spain

Sentiment analysis methods have been developed to automatically extract opinions from texts, e.g. online reviews. Here we explore the idea of adopting sentiment analysis to extract grant reviewers&#39; opinions. This paper examines the relationship between the sentiments of grant reviews and reviewed proposals’ funding decisions made by agencies. We define peer reviews’ prediction as to what degree reviews’ positiveness/negativeness, via sentiment extraction, can indicate proposals’ chance of success in receiving funding. Building on a corpus of peer review texts and related documents from an Irish science funding agency, we conduct sentiment extraction by manual and two simple automated coding (TextBlob and VADER). A comparison between the three results has been made. We find the manual and automated coding results to be overall consistent, but manual coding is still the most accurate and reliable for peer review studies. Furthermore, the three coding results show strong prediction of proposals’ funding decisions, which indicates that reviewers&#39; opinions are largely adopted by agencies.

PEERE

Peer Reviews&#39; Prediction in Proposals&#39; Funding Success: A Sentiment Analysis of Grant Reviews

funding decisions

proposals

grant review

sentiment analysis

Following the success of the 1st PEERE International Conference on Peer Review, the PEERE COST Action follow-up community organised the second edition of the PEERE International Conference on Peer Review  and was celebrated virtually on 29 September – 1 October 2020.

The conference welcomed any contribution (e.g., original research, position papers, literature reviews) that used any method (e.g., quantitative, experimental or qualitative) to investigate peer review in a variety of scientific domains (e.g., scholarly journals, funding agencies, research assessment). Interdisciplinary research is especially welcome.

The program is published here: http://tiny.cc/peere2020 


The second edition of the PEERE International Conference on Peer Review aims to provide a forum for scholars, practitioners and science stakeholders to share research, models, tools and experience on peer review in different fields, e.g., medicine, computer science, social sciences and humanities. It aims to stimulate the use of evidence-based research in the design and implementation of peer review in a variety of fields and encourage more systematic research.

Sentiment analysis methods have been developed to automatically extract opinions from texts, e.g. online reviews. Here we explore the idea of adopting sentiment analysis to extract grant reviewers' opinions. This paper examines the relationship between the sentiments of grant reviews and reviewed proposals’ funding decisions made by agencies. We define peer reviews’ prediction as to what degree reviews’ positiveness/negativeness, via sentiment extraction, can indicate proposals’ chance of success in receiving funding. Building on a corpus of peer review texts and related documents from an Irish science funding agency, we conduct sentiment extraction by manual and two simple automated coding (TextBlob and VADER). A comparison between the three results has been made. We find the manual and automated coding results to be overall consistent, but manual coding is still the most accurate and reliable for peer review studies. Furthermore, the three coding results show strong prediction of proposals’ funding decisions, which indicates that reviewers' opinions are largely adopted by agencies.

Peer Reviews' Prediction in Proposals' Funding Success: A Sentiment Analysis of Grant Reviews

technical paper

Peer review is often criticized for being flawed, subjective and biased, but research into peer review is still lacking studies on peer review reports. In our study, in which text-analysis software was used to determine the linguistic characteristics of 472,449 peer review reports we compared a range of characteristics (including analytical tone, authenticity, clout, three measures of sentiment, and morality) as interaction of reviewer recommendation, area of research, type of peer review and reviewer gender. Our main conclusion was that the reviewer recommendation had the biggest impact on the linguistic characteristics of reports, and that area of research, type of peer review and reviewer gender had little or no impact. The lack of influence of research area, type of review or reviewer gender on the linguistic characteristics is a sign of the robustness of peer review.

Large-scale language analysis of peer review reports

About 90% of researchers overall thought the main area of effectiveness for peer review would be in improving the published paper by providing constructive feedback to authors. This begs the question of what constructive feedback is, and even more, how current annotation tools (e.g. Acrobat Reader) can be enhanced with facilities towards quality feedback. We present Review&Go, a Chrome’s browser extension which supports manuscript reviewing lifecycle . Review&Go is available for download at the Chrome’s Web Store.

Software scaffolds for quality feedback in peer review

Peer-review represents the main mechanism of selecting for publishing high-quality content with the purpose of maintaining the quality, validity, and originality of scientific publications. However as peer-review is being performed by humans, it is prone to errors and bias which might underline its incapability to fulfill its purpose. Therefore, we propose to study the peer-review as a cooperation problem that depends heavily on the level of competition between individuals and their interests. We build a new agent-based model where both scientists and journals co-evolve in a competitive environment, but where public good benefits are granted for high-effort scientists. We extend the work of Righi and Takacs (2017) by looking at emergence in a broad set of potential author and reviewer strategies, as well as by exploring the editorial policies that appear from the evolutionary process.
We build an agent-based model to study various scenarios and how they can impact the overall scientific quality of publications. Thereby, we explore how different editorial strategies with various levels of strictness improve or deteriorate the commitment of reviewers, or how different cost of producing high-quality work
changes the attitude of scientists. Moreover, we observe which editorial strategies would survive in the evolution process, both in the case of co-evolution or independent evolution.
We obtained a working agent-based model that produced clear and robust results. Our baseline model can achieve both high-quality papers and reviews from agents, which proves that the introduction of public good benefits stimulates individual commitment toward peer review. The results remain valid even when we considered a different set-up for the scientific social network or when we increased the costs of producing good papers. We also observed that extreme levels of competition would alter the system as agents would start to act according to their interests. Lastly, we saw that editorial strategies converge even in the case of chaotic systems.

Journal Competition and the Sustainability of Peer review: An agent based model

Background: Peer review is regarded as the gold standard decision-making approach for research funding. Around the globe, health research funders use peer review to inform fair and objective decisions to advance science. However, peer review is not without challenges. A large body of evidence ranging from journal articles, opinion letters, editorials, conference abstracts, thesis and blogs, report problems associated with peer review. These problems can be anything from high administrative costs, low reliability, potential biases (e.g. gender, career status), lack of equity, unreliable scoring and ranking, which can then contribute to a high burden to stakeholders involved in the application and decision-making system (1-5). Strategic priorities of the National Institute for Health Research (NIHR), one of the largest UK funders of health research, have endorsed and highlighted the need for further action on improving their peer review during the decision-making process for the allocation of research funding (6, 7). 

Aim: The aim of this study is to identify which elements of the peer review process and/or alternative approaches, support research funding allocation decision-making, for whom, in what contexts, and how. By conducting a realist synthesis, we will be better placed to understand which approaches and/or elements are likely to enhance the decision-making process for the allocation of research funding. 

Methods: A realist synthesis will provide an explanatory analysis of peer-review or alternative approaches to funding allocation that work in specific contexts, such as for different funding organisations, charities, Higher Education Institutes (HEIs), and research disciplines. The search strategy is reported elsewhere (6). Database searches were originally conducted during May-June 2016 and updated to December 2018. Relevant sources were also searched through Google Scholar and included a citation analysis for included studies. Searches of grey literature as well as databases outside health (e.g. IEEE, arXiv.org) were conducted to ensure materials outside of mainstream academic journals were captured.

Results: Electronic searches returned 1824 publications (after removing duplicates) and manual searches returned 36, resulting in 1860 publications. Five researchers independently screened 50 publications each (abstract and/or full text) and inter-rater reliability of paired researchers for agreement to include or exclude reached 80% agreement based on initial eligibility criteria (i.e. studies of any design exploring interventions or approaches for allocating research funds). This agreement validated the first-stage inclusion/exclusion criteria and the remaining publications (n=1610) were screened according to these criteria. This resulted in 215 publications that were subjected iteratively in relation to programme theory statements (A. if there’s peer review (as a process) then there is bias & burden to all stakeholders B. if variations to the typical peer review process are introduced then bias/burden/costs could be reduced, C. Studies have elements of A&B). Two researchers then discussed the assessment of the 215 included publications and made final decisions on inclusion. A total of 111 publications were selected to inform the research questions. All researchers in the team were involved in data extraction. The following section provides a brief description of the initial findings of the synthesis as per the realist methodology. 

Evidence map
Overall, publications were found to fall into two categories: 
1. Studies that generated evidence on known issues with peer review (bias towards early career researchers, innovation, gender; ability of peer review to predict future value of research; burden for funders and researchers to manage and write applications) 
2. Studies elaborating the need to find approaches different from peer review that can be used to make funding decisions and/or that propose novel approaches. 
The extraction process is ongoing, and the current emerging findings are taken from an in-depth analysis of 30 included publications (out of the 111). What we know so far is that only a handful of interventions have measured in some form the efficiency of peer review in terms of time or cost efficiencies or which have implemented slight or radical alternatives to the current peer review process. 

Context mechanisms outcomes
From the 111 included publications, we developed a map of the Context, Mechanism and Outcomes (CMO), which is central in the realist methodology to answer the questions of for whom, in what context and how an intervention works. Our CMO map shows that the contexts in which the studies have taken place are funders and higher education institutions focusing on developing interventions to increase efficiencies. Other studies are situated in academic contexts where research communities have explored peer review processes from an experimental point of view to enhance the discipline of Research on Research. From the funder perspective, interventions that have been developed to enhance peer review fall into three categories: 
1. Incremental changes to the funding process (modifying the number of stages in applications, modifying reviewers’ numbers and changing the size of the application) 
2. Organisation-level changes creating funding streams to address bias or systems that aid the identification and management of peer reviewers 
3. Alternative approaches with the most salient approaches (sandpits, random allocation, golden tickets, Delphi exercises, and community-based grants). 
A brief review of these categories using the existing/emerging evidence is provided below. 



Incremental changes to peer review 
The National Institutes of Health (NIH) recommendations to streamline peer review include shorter applications with focus on impact and innovation, and less on methods and preliminary data, (8) as well as establishing a numerical quota for the number of awards made to new investigators (Kaiser, 2008b), which aimed to match the application success rates of new investigators to those of established investigators. The Australian National Health and Medical Research Council proposed having a more focused application to reduce preparation costs and improving as a result the quality of peer review by reducing workloads (9, 10). However, some changes can have unintended negative effects, demonstrated in two cross-sectional surveys studies that by shortening the application increased the applicants time spent on preparing applications (11). This is an indication that changes in one context may shift the burden onto another, and this needs to be considered when proposing changes to peer review processes. Other areas under consideration include improvements to fairness, effectiveness and robustness of peer review through masking applications (12) increasing transparency (13) or developing peer review criteria to meet the goals of a specific research programme or discipline (14).

Alternative approaches to decision-making 
Novel approaches to funding decision-making that substitute (at least partially) peer review have also been discussed or implemented. The common approaches in the literature include sandpits, partial random allocation of research funds and Delphi’s. Sandpits are interactive workshops involving 20 to 30 participants, a director, a team of experts and independent stakeholders. The group comes together for a multi-day residential workshop to discuss ideas and participate in interactive sessions with the purpose of establishing research collaborations and receive funds. The proposals in the workshop are evaluated in a similar way to funding panels (15). The basic premise of the allocation of funds by chance retain an aspect of peer review where all proposals submitted to a call are assessed against the research programme requirements, and those potentially fundable are randomly allocated funding (16). A modified Delphi’s (or consensus surveys) proved to be an efficient, transparent, and equitable method of reviewing novel grant applications in a specialized field of research, where no local expertise was available. This process may also be useful for other peer review processes, particularly where there is limited access to local experts (17).
Preliminary conclusion: The approaches and interventions identified ebb and flow within the literature, perhaps influenced by multiple external factors including funding availability and the overall research landscape. Therefore, research that generates empirical evidence has an important role in enabling funders to make informed decisions about whether and how peer review processes work best, and the contexts in which alternatives may offer viable alternatives that support decision-making processes. Findings from this synthesis would encourage long-term implementation of successful mechanisms to the allocation of research funding.

Keywords: peer review, science of science, realist synthesis
Acknowledgements: We would like to thank Geoff Frampton, Karen Pickett, Jo Picot and Jonathan Shepherd at the Southampton Health Technology Assessments Centre for their collaboration in this study. The search strategy for this realist synthesis extends the original search used in their systematic map and review. Discussions with the team at the start of the study strengthened the overall design of the review and supported the Adding Value in Research NIHR initiative. We would also like to thank Justin Jagosh, Director of the Centre for Advancement in Realist Evaluation and Synthesis at the University of Liverpool for an insightful Realist Evaluation and Synthesis workshop and for sharing his expertise with us during the design of our study.
References
1. Hodgson C. How reliable is peer review? An examination of operating grant proposals simultaneously submitted to two similar peer review systems. J Clin Epidemiol. 1997;50(11):1189-95.
2. Abdoul H, Perrey C, Tubach F, Amiel P, Durand-Zaleski I, Alberti C. Non-financial conflicts of interest in academic grant evaluation: a qualitative study of multiple stakeholders in France. PLoS ONE. 2012;7(4):e35247.
3. Bollen J, Crandall D, Junk D, Ding Y, Borner K. From funding agencies to scientific agency: Collective allocation of science funding as an alternative to peer review. EMBO Rep. 2014;15(2):131-3.
4. Bishop D. Nature INDEX. 2018. [cited 2019]. Available from: https://www.natureindex.com/news-blog/luck-of-the-draw.
5. Bollen J, Carpenter S, Lubchenco J, Scheffer M. Rethinking Resource Allocation in Science. arXiv preprint arXiv:190407645. 2019.
6. Shepherd J, Frampton GK, Pickett K, Wyatt JC. Peer review of health research funding proposals: A systematic map and systematic review of innovations for effectiveness and efficiency. PLoS ONE. 2018;13(5):e0196914.
7. Turner S, Bull A, Chinnery F, Hinks J, Mcardle N, Moran R, et al. Evaluation of stakeholder views on peer review of NIHR applications for funding: a qualitative study. BMJ Open. 2018;8(12):e022548.
8. Kaiser J. National Institutes of Health. Changes in peer review target young scientists, heavyweights. Science. 2008;320(5882):1404.
9. Herbert DL, Barnett AG, Graves N. Funding: Australia's grant system wastes time. Nature. 2013;495(7441):314.
10. Herbert DL, Graves N, Clarke P, Barnett AG. Using simplified peer review processes to fund research: a prospective study. BMJ Open. 2015;5(7):e008380.
11. Barnett AG, Graves N, Clarke P, Herbert D. The impact of a streamlined funding application process on application time: two cross-sectional surveys of Australian researchers. BMJ Open. 2015;5(1):e006912.
12. Ismail S, Farrands A, Wooding S. Evaluating grant peer review in the health sciences. A review of the literature. Cambridge, UK; 2009.
13. Gurwitz D, Milanesi E, Koenig T. Grant application review: the case of transparency. Plos Biology. 2014;12(12):e1002010.
14. Hawe P, Samis S, Di Ruggiero E, Shoveller JA. Population Health Intervention Research Initiative for Canada: progress and prospects. N S W Public Health Bull. 2011;22(1-2):27-32.
15. Guthrie S, Guérin B, Wu H, Ismail S, Wooding S. Alternatives to peer review in research project funding. Cambridge, UK; 2013.
16. Fang FC, Casadevall A. Research Funding: the Case for a Modified Lottery. mBio. 2016;7(2):e00422-16.
17. Holliday C, Robotin M. The Delphi process: a solution for reviewing novel grant applications. Int J Gen Med. 2010;3:225-30.



Decision-making approaches to grant funding allocation: insights from a realist synthesis

One of the central themes of studies on peer review decisions is whether the decisions select the best applicants. Some studies found that the best-performing unfunded applicants outperformed the funded applicants, and concluded that peer review is unable to select the best applicants. In contrast, our simulation model shows that we should be cautious when drawing conclusions from a comparison between funded applicants and best-performing unfunded applicants. Uncertainty in bibliometric statistics may cause the best-performing unfunded applicants to outperform the funded applicants even if peer review provides perfectly accurate assessments of the performance of applicants. Such observations are insufficient for concluding that peer review is not working properly. A deeper understanding of uncertainty in bibliometric statistics is needed before concrete conclusions can be drawn from comparisons between peer review outcomes and bibliometric statistics.

Challenges in Using Bibliometric Indicators to Assess Peer ReviewDecisions: A Simulation Model

Several countries partly fund universities on the basis of scientific performance. In this context, peer review is often used to assess the quality of scientific products. In large research assessment exercises peer review may become costly, and bibliometric indicators have been suggested as an alternative. A recurrent question is whether peer review and metrics tend to yield similar outcomes. In this paper, we study the agreement between bibliometric indicators and peer review at the institutional level. Additionally, we also quantify the internal agreement between two peer reviewers. We find that the level of agreement is generally higher at the institutional level than at the publication level. Overall, the agreement between metrics and peer review is on par with the internal agreement among two peer reviewers. This suggests that bibliometric indicators may indeed be considered as an alternative to peer review for large institutional research assessment exercises.

Metrics and peer review agreement at the institutional level

A Content Exploration of Reviewers' Comments in FP7 Marie Curie ITN evaluation reports 
Darko Hren1, David G. Pina2, Cristopher Norman3, Ana Marušić4

1University of Split Faculty of Humanities and Social Sciences, Split, Croatia
2Research Executive Agency, European Commission, Brussels, Belgium
3LIMSI, CNRS, Université Paris Saclay, Paris, France
4University of Split School of Medicine, Split, Croatia

Objectives:
To identify themes in the content of reviewers' evaluation reports about proposals submitted to the FP7 Marie Curie Initial Training Networks calls and to explore the content areas in reviewers' comments that differentiate successful from unsuccessful proposals.

Methods:
We qualitatively analyzed the contents of reviewers' comments to obtain the most common themes within evaluation reports. Given a large amount of textual data, we used a two-step approach.
The first step was automated to help reduce the data to the manageable size. We used K-Means clustering to identify the clusters of statements within sections of the evaluation reports. Clusters were visualized in 2-dimensional space using TensorBoard Embedding Projector (https://projector.tensorflow.org/). Principal components analysis (PCA) was used for the visualization of clusters and cosine distances between clusters as a reference to cluster distances.
In the second step, the clusters were further grouped into themes using empirical estimates from PCA in step 1 and qualitative analysis based on cluster terms and content. To identify clusters that could potentially be grouped into themes, we used cosine distances lower than 1 as an arbitrary indicator of cluster proximity. Next, we inspected the content of the obtained clusters (terms and example statements) to verify whether and how groups of clusters could be interpreted as broader themes. Inspection of content had precedence over empirical data, i.e. if there was no meaningful overlap in cluster terms and content, even clusters that had cosine distance below 1 were not combined.
Finally, we registered the frequencies of obtained themes within each evaluation report (i.e. a number of sentences containing terms belonging to a specific theme) to inspect potential differences between proposals with different decision status.

Results:
We analyzed the content of reviews of 3666 grant proposals submitted to Marie Curie Initial Training Networks from the FP7 PEOPLE programme in 2008 (n=886), 2010 (n=857), 2011 (n=909), and 2012 (n=1013).
Each review had four sections of criteria (C1-Science & Technology Quality, C2-Training, C3-Implementation, and C4-Impact), and each section was divided into two parts (Strengths and Weaknesses). Therefore, 12 separate K-Means cluster analyses were performed in the first step, and we identified between 9 and 12 clusters explaining between 32.7 and 51.6% of the variance. 
In the second step, we identified between 29 themes within different sections of the evaluation reports (Figure 1A-D). Evaluation reports of proposals from the main and reserve lists included more themes related to strengths and fewer themes related to weaknesses than those of rejected proposals. Themes related to strengths were generally more frequent than those related to weaknesses. Most prominent themes related to strengths were: Research programme and objectives and Multidisciplinary context (C1), Consistency of the research and training programme and Career development (C2), Capacities of research and training (C3), and Outreach activities and Career prospects (C4). Most prominent themes related to weaknesses were: Research programme and objectives and Research methodology (C1), Engagement of ESRs and Training characteristics (C2), Capacities of research and training (C3), and Exploitation and dissemination (C4).

Conclusions:
We used a novel approach which combined machine learning and qualitative inquiry to identify the most common themes of the evaluation reports of proposals submitted to the FP7 Marie Curie Initial Training Networks calls. The themes reflected general evaluation criteria applied by the European Commission, but also specific themes that distinguished successful from unsuccessful proposals thus demonstrating the validity of the selection process.
The obtained results may be used by researchers preparing to apply for grants.



A Content Exploration of Reviewers' Comments in FP7 Marie Curie ITN evaluation reports

eLife launched a trial peer-review process in June 2018, designed to give authors greater control over the decision to publish. Authors opted into the trial process when submitting their manuscript for consideration and, once the eLife editors decided that a submission would be sent for in-depth peer review, the journal was committed to publishing the paper. Authors chose how to address the concerns raised during review and, upon receiving the revised submission, editors rated whether these concerns had been addressed. This editorial rating was published with the paper, as were the decision letter after peer review, the separate reviews, and the accompanying author responses. The trial process was available until more than 300 authors opted in (in total, 313 opted in) and we compare these with 616 papers submitted into the regular process during the same period of time. Overall, the trial review process resulted in a significantly higher acceptance rate (20.8% compared with 14.9% in the regular process), with all the issues raised by reviewers addressed by the authors in the vast majority (88%) of revised submissions. We observed no significant difference in acceptance rates for male and female last authors in the trial process (21% and 20.6% respectively). However, we observed marginally higher acceptance rates for late-career researchers (32%) in the trial compared to their early- (22%) and mid-career (19%) colleagues and significantly higher acceptance rates for last authors based in North America (29.2%) and Europe (18.1%) compared to last authors based in Asia (11.9%). Trial submissions took significantly longer to be accepted (median = 187 days, IQR = 130.5-235.5) compared to the regular process (median = 138 days, IQR =114-170).

Comparison of two types of collaborative peer review at eLife

A comparative secondary data analysis was conducted on reviewers‘ answers to peer review questionnaires from two different journals from the life sciences (5382 reports in total). The correspondence of the answering patterns to peer review criteria sets with the reviewers’ recommendations was investigated, as well as the peer reviewers’ disagreement about manuscript quality in the respective dimensions. The analyses show a high internal consistency for the individual opinion questionnaires with the resulting recommendation. Furthermore, high reviewers’ disagreement was observed on the given criteria as well as the overall recommendation. Taken all together, the results indicate, that the norm of disinterestedness is reflected in peer reviewers' behavior and further depends on editorial decisions.

Consistency and disagreement in journal peer review and the norm of disinterestedness

Peer-review through our viewpoint is understood to be characterised by heterogeneity. In particular, reviewers are generally seen to be moved by a variety of reasons. We argue that some may have a burning quest for better science. In such cases, if it is seen worth doing so, they might try to convince the authors to steer the paper in the directions they deem necessary, with their reviews. In a quest for ever-better science they may, for example, produce extremely detailed and long reviews where they attempt at showing the “right” way to science to the authors. In this way, we may say, they attempt to ‘hack’ the paper through their review. This work studies the effect of this hacker-reviewer phenomenon using a modified version of the PRIF agent-based model (Secchi & Cowley, 2019). The aim of the model is that of exploring the effects of reviewer-triggered influence on submitted journal articles when the reviewer acts as a hacker.


Downloads

Next from PEERE

Large-scale language analysis of peer review reports

Similar lecture

Large-scale language analysis of peer review reports

.css-70qvj9{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-align-items:center;-webkit-box-align:center;-ms-flex-align:center;align-items:center;}Downloads

Next from PEERE

Large-scale language analysis of peer review reports

Similar lecture

Large-scale language analysis of peer review reports

Downloads