Spain

A Content Exploration of Reviewers&#39; Comments in FP7 Marie Curie ITN evaluation reports 
Darko Hren1, David G. Pina2, Cristopher Norman3, Ana Marušić4

1University of Split Faculty of Humanities and Social Sciences, Split, Croatia
2Research Executive Agency, European Commission, Brussels, Belgium
3LIMSI, CNRS, Université Paris Saclay, Paris, France
4University of Split School of Medicine, Split, Croatia

Objectives:
To identify themes in the content of reviewers&#39; evaluation reports about proposals submitted to the FP7 Marie Curie Initial Training Networks calls and to explore the content areas in reviewers&#39; comments that differentiate successful from unsuccessful proposals.

Methods:
We qualitatively analyzed the contents of reviewers&#39; comments to obtain the most common themes within evaluation reports. Given a large amount of textual data, we used a two-step approach.
The first step was automated to help reduce the data to the manageable size. We used K-Means clustering to identify the clusters of statements within sections of the evaluation reports. Clusters were visualized in 2-dimensional space using TensorBoard Embedding Projector (https://projector.tensorflow.org/). Principal components analysis (PCA) was used for the visualization of clusters and cosine distances between clusters as a reference to cluster distances.
In the second step, the clusters were further grouped into themes using empirical estimates from PCA in step 1 and qualitative analysis based on cluster terms and content. To identify clusters that could potentially be grouped into themes, we used cosine distances lower than 1 as an arbitrary indicator of cluster proximity. Next, we inspected the content of the obtained clusters (terms and example statements) to verify whether and how groups of clusters could be interpreted as broader themes. Inspection of content had precedence over empirical data, i.e. if there was no meaningful overlap in cluster terms and content, even clusters that had cosine distance below 1 were not combined.
Finally, we registered the frequencies of obtained themes within each evaluation report (i.e. a number of sentences containing terms belonging to a specific theme) to inspect potential differences between proposals with different decision status.

Results:
We analyzed the content of reviews of 3666 grant proposals submitted to Marie Curie Initial Training Networks from the FP7 PEOPLE programme in 2008 (n=886), 2010 (n=857), 2011 (n=909), and 2012 (n=1013).
Each review had four sections of criteria (C1-Science &amp; Technology Quality, C2-Training, C3-Implementation, and C4-Impact), and each section was divided into two parts (Strengths and Weaknesses). Therefore, 12 separate K-Means cluster analyses were performed in the first step, and we identified between 9 and 12 clusters explaining between 32.7 and 51.6% of the variance. 
In the second step, we identified between 29 themes within different sections of the evaluation reports (Figure 1A-D). Evaluation reports of proposals from the main and reserve lists included more themes related to strengths and fewer themes related to weaknesses than those of rejected proposals. Themes related to strengths were generally more frequent than those related to weaknesses. Most prominent themes related to strengths were: Research programme and objectives and Multidisciplinary context (C1), Consistency of the research and training programme and Career development (C2), Capacities of research and training (C3), and Outreach activities and Career prospects (C4). Most prominent themes related to weaknesses were: Research programme and objectives and Research methodology (C1), Engagement of ESRs and Training characteristics (C2), Capacities of research and training (C3), and Exploitation and dissemination (C4).

Conclusions:
We used a novel approach which combined machine learning and qualitative inquiry to identify the most common themes of the evaluation reports of proposals submitted to the FP7 Marie Curie Initial Training Networks calls. The themes reflected general evaluation criteria applied by the European Commission, but also specific themes that distinguished successful from unsuccessful proposals thus demonstrating the validity of the selection process.
The obtained results may be used by researchers preparing to apply for grants.



PEERE

A Content Exploration of Reviewers&#39; Comments in FP7 Marie Curie ITN evaluation reports

grant evaluation

peer review

qualitative analysis

Following the success of the 1st PEERE International Conference on Peer Review, the PEERE COST Action follow-up community organised the second edition of the PEERE International Conference on Peer Review  and was celebrated virtually on 29 September – 1 October 2020.

The conference welcomed any contribution (e.g., original research, position papers, literature reviews) that used any method (e.g., quantitative, experimental or qualitative) to investigate peer review in a variety of scientific domains (e.g., scholarly journals, funding agencies, research assessment). Interdisciplinary research is especially welcome.

The program is published here: http://tiny.cc/peere2020 


The second edition of the PEERE International Conference on Peer Review aims to provide a forum for scholars, practitioners and science stakeholders to share research, models, tools and experience on peer review in different fields, e.g., medicine, computer science, social sciences and humanities. It aims to stimulate the use of evidence-based research in the design and implementation of peer review in a variety of fields and encourage more systematic research.

A Content Exploration of Reviewers' Comments in FP7 Marie Curie ITN evaluation reports 
Darko Hren1, David G. Pina2, Cristopher Norman3, Ana Marušić4

1University of Split Faculty of Humanities and Social Sciences, Split, Croatia
2Research Executive Agency, European Commission, Brussels, Belgium
3LIMSI, CNRS, Université Paris Saclay, Paris, France
4University of Split School of Medicine, Split, Croatia

Objectives:
To identify themes in the content of reviewers' evaluation reports about proposals submitted to the FP7 Marie Curie Initial Training Networks calls and to explore the content areas in reviewers' comments that differentiate successful from unsuccessful proposals.

Methods:
We qualitatively analyzed the contents of reviewers' comments to obtain the most common themes within evaluation reports. Given a large amount of textual data, we used a two-step approach.
The first step was automated to help reduce the data to the manageable size. We used K-Means clustering to identify the clusters of statements within sections of the evaluation reports. Clusters were visualized in 2-dimensional space using TensorBoard Embedding Projector (https://projector.tensorflow.org/). Principal components analysis (PCA) was used for the visualization of clusters and cosine distances between clusters as a reference to cluster distances.
In the second step, the clusters were further grouped into themes using empirical estimates from PCA in step 1 and qualitative analysis based on cluster terms and content. To identify clusters that could potentially be grouped into themes, we used cosine distances lower than 1 as an arbitrary indicator of cluster proximity. Next, we inspected the content of the obtained clusters (terms and example statements) to verify whether and how groups of clusters could be interpreted as broader themes. Inspection of content had precedence over empirical data, i.e. if there was no meaningful overlap in cluster terms and content, even clusters that had cosine distance below 1 were not combined.
Finally, we registered the frequencies of obtained themes within each evaluation report (i.e. a number of sentences containing terms belonging to a specific theme) to inspect potential differences between proposals with different decision status.

Results:
We analyzed the content of reviews of 3666 grant proposals submitted to Marie Curie Initial Training Networks from the FP7 PEOPLE programme in 2008 (n=886), 2010 (n=857), 2011 (n=909), and 2012 (n=1013).
Each review had four sections of criteria (C1-Science & Technology Quality, C2-Training, C3-Implementation, and C4-Impact), and each section was divided into two parts (Strengths and Weaknesses). Therefore, 12 separate K-Means cluster analyses were performed in the first step, and we identified between 9 and 12 clusters explaining between 32.7 and 51.6% of the variance. 
In the second step, we identified between 29 themes within different sections of the evaluation reports (Figure 1A-D). Evaluation reports of proposals from the main and reserve lists included more themes related to strengths and fewer themes related to weaknesses than those of rejected proposals. Themes related to strengths were generally more frequent than those related to weaknesses. Most prominent themes related to strengths were: Research programme and objectives and Multidisciplinary context (C1), Consistency of the research and training programme and Career development (C2), Capacities of research and training (C3), and Outreach activities and Career prospects (C4). Most prominent themes related to weaknesses were: Research programme and objectives and Research methodology (C1), Engagement of ESRs and Training characteristics (C2), Capacities of research and training (C3), and Exploitation and dissemination (C4).

Conclusions:
We used a novel approach which combined machine learning and qualitative inquiry to identify the most common themes of the evaluation reports of proposals submitted to the FP7 Marie Curie Initial Training Networks calls. The themes reflected general evaluation criteria applied by the European Commission, but also specific themes that distinguished successful from unsuccessful proposals thus demonstrating the validity of the selection process.
The obtained results may be used by researchers preparing to apply for grants.



A Content Exploration of Reviewers' Comments in FP7 Marie Curie ITN evaluation reports

technical paper

eLife launched a trial peer-review process in June 2018, designed to give authors greater control over the decision to publish. Authors opted into the trial process when submitting their manuscript for consideration and, once the eLife editors decided that a submission would be sent for in-depth peer review, the journal was committed to publishing the paper. Authors chose how to address the concerns raised during review and, upon receiving the revised submission, editors rated whether these concerns had been addressed. This editorial rating was published with the paper, as were the decision letter after peer review, the separate reviews, and the accompanying author responses. The trial process was available until more than 300 authors opted in (in total, 313 opted in) and we compare these with 616 papers submitted into the regular process during the same period of time. Overall, the trial review process resulted in a significantly higher acceptance rate (20.8% compared with 14.9% in the regular process), with all the issues raised by reviewers addressed by the authors in the vast majority (88%) of revised submissions. We observed no significant difference in acceptance rates for male and female last authors in the trial process (21% and 20.6% respectively). However, we observed marginally higher acceptance rates for late-career researchers (32%) in the trial compared to their early- (22%) and mid-career (19%) colleagues and significantly higher acceptance rates for last authors based in North America (29.2%) and Europe (18.1%) compared to last authors based in Asia (11.9%). Trial submissions took significantly longer to be accepted (median = 187 days, IQR = 130.5-235.5) compared to the regular process (median = 138 days, IQR =114-170).

Comparison of two types of collaborative peer review at eLife

A comparative secondary data analysis was conducted on reviewers‘ answers to peer review questionnaires from two different journals from the life sciences (5382 reports in total). The correspondence of the answering patterns to peer review criteria sets with the reviewers’ recommendations was investigated, as well as the peer reviewers’ disagreement about manuscript quality in the respective dimensions. The analyses show a high internal consistency for the individual opinion questionnaires with the resulting recommendation. Furthermore, high reviewers’ disagreement was observed on the given criteria as well as the overall recommendation. Taken all together, the results indicate, that the norm of disinterestedness is reflected in peer reviewers' behavior and further depends on editorial decisions.

Consistency and disagreement in journal peer review and the norm of disinterestedness

Peer-review through our viewpoint is understood to be characterised by heterogeneity. In particular, reviewers are generally seen to be moved by a variety of reasons. We argue that some may have a burning quest for better science. In such cases, if it is seen worth doing so, they might try to convince the authors to steer the paper in the directions they deem necessary, with their reviews. In a quest for ever-better science they may, for example, produce extremely detailed and long reviews where they attempt at showing the “right” way to science to the authors. In this way, we may say, they attempt to ‘hack’ the paper through their review. This work studies the effect of this hacker-reviewer phenomenon using a modified version of the PRIF agent-based model (Secchi & Cowley, 2019). The aim of the model is that of exploring the effects of reviewer-triggered influence on submitted journal articles when the reviewer acts as a hacker.


Hacking one's way to peer-review

The Horizon 2020 Marie Skłodowska-Curie Actions (MSCA) Individual Fellowships (IF) funding programme is one of the largest and most competitive EU research programme (around 15% success rate). It would be expected that the applications of funded proposals have detected ethical issues well. The aim of this study was to describe the ethics evaluation process in the MSCA IF funding scheme . We first assessed the ethical issues identified by the applicants when preparing their proposals and compared these to the ethics issues identified by ethics experts during the ethics evaluation performed for those proposals retained for funding. In total, we analyzed 3074 proposals from three MSCA IF calls ( 2016, 2017 and 2018). Applicants identified almost two times fewer ethical issues compared to the experts performing the ethics review. Most of the ethics issues were identified in proposals from the Life Sciences and Social Sciences and Humanities research panels, which represented more than 60% of all proposals. Out of all 3074 proposals which were main listed and accepted for funding, 2225 (72.4%) received conditional ethics clearance, pending satisfaction of requirements defined by the ethics experts. The most prevalent requirements were within the Protection of personal data, Environment, Health and Safety, Humans, and non-EU countries categories. Our results emphasize that successful applicants to the MSCA IF programme, consisting mostly of postdoctoral researchers, are not adequately trained or knowledgeable about the ethics issues related to their proposed research, and that problem persists over time. Training in research ethics and proper guidance by the host institutions are paramount for narrowing the gap between the applicants’ ethics self-assessments and the results of the ethics review procedure. 


Ethics assessment of H2020 Marie Skłodowska-CUrie individual fellowships

ARCADIA (Assessment of Review reports with a Checklist Available to eDItors and Authors) constitutes the first tool that has been systematically developed to assess the quality of peer review reports in biomedical research. It is a checklist which consists of five domains covering 14 items. In the presentation, we will show the psychometric properties of the newly developed ARCADIA tool.

Psychometric testing of ARCADIA, a tool for assessing peer review report quality in biomedical research

Effect of an editorial intervention to improve the completeness of reporting of randomised trials: a randomised controlled trial

A peer review programme for mental health service users
Objective
Service user involvement should be an essential component of medical research, and many funders are requiring evidence in funding applications that expertise by experience is included. As editors, while we highly value peer review by researchers with lived experience of mental illness, we are concerned that individuals without formal training in research methods are not given the opportunity to express their opinion on the work we publish. Our objective was to conduct a pilot programme to explore how people with lived experience but without research qualifications or an employment contract at a university can contribute to peer review.
Methods
The Lancet Psychiatry launched in 2018 a peer review programme for mental health service users in collaboration with the McPin Foundation, a charity based in London, UK that champions expertise from experience in mental health research. Participants who have used mental health services in the UK were recruited via on-line advertisements, The Lancet Psychiatry and service user news bulletins. Applications were reviewed by a panel from the McPin Foundation and Lancet Psychiatry. Selected individuals received training in peer review in London, UK, and online from The Lancet Psychiatry editorial team supported by McPin staff. Travel costs for an annual meeting are funded by the McPin Foundation, and The Lancet Psychiatry pays for each review. 
Results
48 people applied. McPin staff produced a short list of 20 candidates. Lancet Psychiatry editors and McPin staff then selected ten participants: eight women and two men. 9 participants attended a training day at the Lancet office in December 2018. All participants were encouraged to complete the online training in peer review provided by Publons with a Lancet Psychiatry editor as their mentor. 7 participants (6 women, 1 man) had completed this by mid-August 2019 and have started reviewing manuscripts for The Lancet Psychiatry. They are being invited to review manuscripts in areas such as clinical trials, service delivery, observational studies, qualitative research, health policy, systematic reviews, and opinion pieces. 


A peer review programme for mental health service users

We explore which characteristics of a reviewer affect disagreement with the other reviewers in research proposal evaluation, studying 52,488 proposals submitted to the Horizon 2020 Marie Skłodowska-Curie Actions (MSCA) and 1,939 proposals to the European Cooperation in Science and Technology Actions (COST). We find that experience in previous calls significantly reduces disagreement, and in MSCA - IF, an inverted U relationship between the number of proposals evaluated in a given call and disagreement, with a remarkable decrease of disagreement above thirteen proposals evaluated. The results suggest that reviewing experience improve reliability, curbing unwarranted disagreement by gradually fine-tuning reviewers’ evaluation.

Does reviewers' traits affect disagreement in proposal evaluation?

The organization of peer review processes in grant proposals evaluation is rarely studied. We analysed 77 thousand proposals from the Marie Curie programme, submitted over a period of 12 years, with the aim to get insights on the reliability of the process. Mean scores, Average Deviation (AD) indices, used as a measure of inter-rater agreement, and interrupted time series analysis were calculated to study changes in reviewers’ scores and level of agreement. Our results indicate that scores given by and agreement among reviewers may be more dependent on the type of grants or scientific panels than on the way the evaluation process is organised, with important economic and environmental implications for policy-making in research grants evaluation.

Reviewers' agreement and scores in Marie Curie proposals from 2007 to 2018

Background: Every year, thousands of research proposals are submitted to funding organisations in the hope of securing funding for research. These applications go through a number of assessment stages in which decisions are made about whether to decline the application, progress the application or fund the application (1–3). Funding organisations often have limited funds, large numbers of application submissions, and a need to demonstrate impactful spending, and so many of these applications go through multiple iterations and not all will be funded. 
Feedback to the applicants about these decisions are a vital part of the scientific process. Applicants rely on feedback to improve the quality of their application for securing successful funding and/or as a learning opportunity to facilitate the chance of success in future applications. It is therefore important that applicants receive constructive feedback in order to improve the quality of their application, change the content of their application or to understand why their application was or was not successful (was it competitive but there were stronger applications, or were significant improvements required?). A lack of constructive feedback could not only impact on professional development for researchers but could introduce unnecessary burden and negative experiences of researchers as applicants, in what is already a labour intensive research activity (1,4–7). 
For the National Institute for Health Research (NIHR) Evaluation, Trials and Studies Coordinating Centre (NETSCC), applicants receive written feedback by the research programme funding committees at both stage 1 and stage 2 of the application process. Stage 1 feedback is provided for applications that are progressed to stage 2, as well as for those that were unsuccessful. For applications that successfully progress to stage 2, feedback is again provided for both successful and unsuccessful applications at this second stage. Committees are encouraged to consider providing targeted feedback for each application rather than using generic phrases. Feedback points are drafted in the funding committee meetings and are provided to the applicants in written bullet point format. NIHR applicants also have access to guidance on what to include in their applications and the assessment criteria against which the application will be assessed. 

In order to support applicants throughout the funding process, it is vital that the guidance and feedback provided is of high quality, constructive and consistent. It is therefore important to understand the key components of the feedback provided to applicants at Stage 1 and 2 and how these are perceived by applicants. This will lead to increased knowledge on the quality and value of the feedback provided to applicants, whether improvements are needed regarding the type of feedback given and whether this is consistent across the NIHR research programmes managed by NETSCC, promoting the adding value in research approach and transparency agenda. 

Objectives: The aim is to identify and compare the key components of feedback that are provided to applicants by NIHR research programmes, whether these components are consistent across different sources of feedback (external peer review, funding committees at stage 1, funding committees at stage 2, guidance documents) and how the feedback is perceived by applicants.

Methods: To gain a NIHR funder perspective of feedback provided, qualitative content analysis of feedback and guidance documents from the Health Technology Assessment (HTA), Public Health Research (PHR), Efficacy and Mechanism Evaluation (EME), and Health Services and Delivery Research (HSDR) funding programmes was conducted. Following ethical approval, purposeful sampling was used to select commissioned-led and researcher-led, and successful and unsuccessful research applications from March 2018 and March 2019. To date, a total of 84 applicants across the four funding programmes have been contacted to gain approvals to access the feedback data. Using a phased approach, (1) a coding framework was created using inductive content analysis of external peer reviews, stage 1 and stage 2 funding committee feedback from the HTA programme, and (2) then applied to the stage 2 funding committee feedback from the remaining funding programmes using deductive content analysis. (3) The final phase will compare the key components across funding programmes and to the guidance criteria provided for each programme. 
To gain an NIHR applicant perspective about feedback received from funding committees, an online survey was conducted in October 2019. Following ethical approval, the survey used purposeful sampling and targeted chief investigators of applications submitted to HTA, EME, PHR and HSDR research programmes during March 2018 and March 2019 to ensure the sample reached commissioned-led, researcher-led and successful and unsuccessful applicants. The survey was distributed via email to 770 applicants and was also published on Twitter. The survey used both closed and open questions to explore how useful the feedback was and considerations for improvements for both stages 1 and 2. A mixed methods analysis using descriptive statistics and inductive thematic analysis was conducted on closed and open question responses.
Results: From the NETSCC perspective, qualitative content analysis of feedback and guidance documents is on-going. To date, feedback documents relating to 25 HTA applications have been analysed. From this, a coding framework has been created. Seven key components (with sub-components) were identified from the feedback given to HTA applicants:
(1) Acceptability to patients and professionals: feedback on patient and public involvement in the study design.
(2) Team and infrastructure: feedback on justification of the roles and time of the team as well as potential conflicts of interest for the team.
(3) Justification of project: feedback on the scientific rationale, relevance and need of the project.
(4) Study design: feedback was primarily around clearly defining and justifying study parameters, outcomes and sample size.
(5) Risks and contingencies: feedback highlighted the need for consideration to be given to recruitment, retention and attrition, and feasibility of the project.
(6) Value for money and costings: feedback around cost elements of the application.
(7) Pathway to impact and implementation: feedback around implementation plans and whether project will have clinical impact.
The most frequently reported components were study design; risks and contingencies; and team and infrastructure. The majority of the components were consistently reported across the funding committee and external peer reviewer comments, except for the justification of project component, which was only provided in feedback to applicants at stage 1. Recruitment for documents in the remaining research programmes commenced in October 2019. To date, we have recruited 23 applicants across PHR (8), EME (6) and HSDR (9), and the coding framework is being applied to stage 2 funding committee feedback and guidance documents. 
From the applicant perspective, 147 applicants completed the online survey (response rate of 19%). After removing data from applications under review, non-NETSCC research programmes and incomplete datasets, a final sample of 115 applicant data was analysed. Overall, more than half of applicants agreed that the feedback provided by the funding committee was useful (52%) and of benefit (53%). However, more than half of applicants also disagreed that the feedback was of good quality (54%) or that it was helpful for developing future applications or work (57%). Responses differed depending on whether the applicant had been successful in securing funding, with at least 77% of funded applicants responding positively about the feedback and less than 37% of unsuccessful applicants responding positively. Moreover, feedback at each stage was more likely to be seen positively if the application was moved on to the next stage/funded whilst feedback at the point of rejection was more likely to be considered negatively. For example, unsuccessful applicants at stage 2 regarded stage 1 feedback positively, but regarded stage 2 feedback negatively, finding it unhelpful and unclear on areas for development. The open-ended responses also suggested that feedback could be more detailed and directive to facilitate changes and that it would be helpful to get some indication of how important each comment was (is it mandatory and which to focus more attention on). In contrast to the findings from the NIHR funder perspective, feedback from the applicant perspective showed that feedback is not always seen as being consistent across stages, with guidance, briefs or within the same feedback document. 
Discussion: The findings will lead to increased knowledge on the quality, transparency and consistency of feedback provided to applicants, whether this is consistent across research programmes and how useful the feedback is to applicants. The preliminary findings show that key components provided in feedback by the different NIHR research programmes were consistently reported across both the funding committee and external peer reviewer comments. However, applicants reported that the content of the feedback was not always consistent across both stages and within stages, and that although applicants found this feedback useful overall, there are a number of areas in which feedback and guidance could be improved. Once the full analysis has been completed, recommendations can then be made for NIHR research programme guidance and feedback to applicants that supports the chance of a successful application, promoting the adding value in research approach and transparency agenda. 
This work is part of a wider series of studies being undertaken by the NIHR Research on Research team on peer review and decision-making in funding allocation. This wider work includes the studies reported here, a realist synthesis to identify contexts in which different decision-making processes work, a survey with international funders to understand current funding practices, qualitative analysis to understand stakeholders’ expectations and an observational study of funding committees and will provide much needed evidence to help inform funders in peer review and decision-making activities.

References:
1. Guthrie S, Ghiga I, Wooding S. What do we know about grant peer review in the health sciences? An updated review of the literature and six case studies. Santa Monica, CA; 2018. 
2. Ismail S, Farrands A, Wooding S, Europe R. Evaluating grant peer review in the health sciences. A review of the literature. Cambridge, UK; 2009. 
3. Publons part of the Web of Science Group. Grant Review in Focus. Global State of Peer Review Series. 2019. 
4. Gluckman P. Which science to fund: time to review peer review? 2012;(December):11. 
5. Graves N, Barnett AG, Clarke P. Cutting random funding decisions. Nature. 2011;469(7330):299. 
6. Herbert DL, Coveney J, Clarke P, Graves N, Barnett AG. The impact of funding deadlines on personal workloads, stress and family relationships: a qualitative study of Australian researchers. BMJ Open. 2014;4(3):e004462. 
7. Herbert DL, Barnett AG, Clarke P, Graves N. On the time spent preparing grant proposals: an observational study of Australian researchers. BMJ Open. 2013;3(5):1–6. 



A Content Exploration of Reviewers' Comments in FP7 Marie Curie ITN evaluation reports

Downloads

Next from PEERE

Comparison of two types of collaborative peer review at eLife

Similar lecture

Feedback to improve agreement in grant peer review

A Content Exploration of Reviewers' Comments in FP7 Marie Curie ITN evaluation reports

.css-70qvj9{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-align-items:center;-webkit-box-align:center;-ms-flex-align:center;align-items:center;}Downloads

Next from PEERE

Comparison of two types of collaborative peer review at eLife

Similar lecture

Feedback to improve agreement in grant peer review

Downloads