Spain

Between 21 May 2016 and 9 September 2019, 1,983 bioRxiv preprints received a single public comment on the bioRxiv website. More than two thirds of those comments were posted by non-authors (n=1,366, 69%) while the remainder were posted by the preprint’s authors (n=617, 31%). Overall, the non-author comments were longer than comments posted by the authors (median number of words was 38, IQR 17 to 83, for non-authors vs 18, IQR 11 to 32, for authors, Mann-Whitney test, P&lt;0.001), and they were posted a median of 23 days (IQR 3 to 117) after the preprints. In comparison, authors’ comments were posted after a median of 91 days (IQR 3 to 23, Mann-Whitney test, P&lt;0.001). Twelve percent of non-author’s comments (n=168) were full review reports resembling those traditionally submitted during the journal peer review process. Comments not resembling full peer review reports most commonly praised the preprint (n=577, 42%), made suggestions on how to improve it (n=399, 29%), or criticized some aspect of the preprint (n=226, 17%). Authors’ comments most commonly contained updates about the preprint’s publication status (n=354, 57%), additional information on the study (n=158, 26%), or solicited feedback for the preprint (n=65, 11%).

PEERE

From amazing work to I beg to differ: analysis of bioRxiv preprints that received one public comment till September 2019

biorxiv

preprints

comments

Following the success of the 1st PEERE International Conference on Peer Review, the PEERE COST Action follow-up community organised the second edition of the PEERE International Conference on Peer Review  and was celebrated virtually on 29 September – 1 October 2020.

The conference welcomed any contribution (e.g., original research, position papers, literature reviews) that used any method (e.g., quantitative, experimental or qualitative) to investigate peer review in a variety of scientific domains (e.g., scholarly journals, funding agencies, research assessment). Interdisciplinary research is especially welcome.

The program is published here: http://tiny.cc/peere2020 


The second edition of the PEERE International Conference on Peer Review aims to provide a forum for scholars, practitioners and science stakeholders to share research, models, tools and experience on peer review in different fields, e.g., medicine, computer science, social sciences and humanities. It aims to stimulate the use of evidence-based research in the design and implementation of peer review in a variety of fields and encourage more systematic research.

Between 21 May 2016 and 9 September 2019, 1,983 bioRxiv preprints received a single public comment on the bioRxiv website. More than two thirds of those comments were posted by non-authors (n=1,366, 69%) while the remainder were posted by the preprint’s authors (n=617, 31%). Overall, the non-author comments were longer than comments posted by the authors (median number of words was 38, IQR 17 to 83, for non-authors vs 18, IQR 11 to 32, for authors, Mann-Whitney test, P<0.001), and they were posted a median of 23 days (IQR 3 to 117) after the preprints. In comparison, authors’ comments were posted after a median of 91 days (IQR 3 to 23, Mann-Whitney test, P<0.001). Twelve percent of non-author’s comments (n=168) were full review reports resembling those traditionally submitted during the journal peer review process. Comments not resembling full peer review reports most commonly praised the preprint (n=577, 42%), made suggestions on how to improve it (n=399, 29%), or criticized some aspect of the preprint (n=226, 17%). Authors’ comments most commonly contained updates about the preprint’s publication status (n=354, 57%), additional information on the study (n=158, 26%), or solicited feedback for the preprint (n=65, 11%).

technical paper

Introducing Elsevier

Legal scholarship has an ambiguous identity, somewhere between the humanities and the social sciences, having features in common with both. It shares many of the characteristics of social sciences since the law is a social phenomenon, but when normativity and legal certainty are concerned, legal scholarship is probably closer to humanities (Peruginelli & Faro, 2018). At the same time law is distinct from both. There is an interesting debate on the role of academic legal research in which consideration is given to 'law as a practical discipline', 'law as humanities' and 'law as social sciences' (Siems & Mac Sithigh, 2012). Because of the peculiarities of legal scholarship, it is not easy to assess legal journals according to the standard quantitative metrics applied in social sciences, and qualitative assessment of the published content is more appropriate. Therefore, we compared the peer review process employed by legal journals in three countries, Croatia, Italy and Spain, to find out their characteristics. 
In this study, 34 Croatian, 36 Spanish and 40 Italian law journals (selected randomly from the 153 top ranking Italian law journals) were analysed. For each journal, we collected basic set of metadata, as well as a document providing the description of the peer-review process. We collected 107 documents in four languages (English, Croatian, Italian and Spanish) which were analysed in order to find out characteristics of the peer review process. For text analysis, we developed an extensive categorization dictionary. Nine main categories in the Categorization dictionary are peer review (process and reviewers), reviewer's characteristics (academic level, autonomy, competence), reviewer's provenance (external, internal), peer review blindness/openness (single-blind, double-blind, open), number of reviewers (one, two), evaluation criteria for submissions (originality, methods, relevance, clarity, accuracy, etc.), peer review outcome (report, acceptance, rejection), ethical issues (editorial standards and codes, conflict of interests, confidentiality, research integrity), manuscript type (original scientific article, professional article, etc.).
The most frequent categories are "peer review", "submissions' evaluation criteria" and "manuscript type", present in more than 90% of the collected documents. "Reviewer's competences" are the most represented subcategory in "“reviewer's characteristics“, as high level of reviewers expertise in the legal sciences is important for law journals. 73% of analyzed journals are declaring anonymous peer review, 25% double-blind peer review, and only one journal employ single-blind peer review. 42% of documents contained information on "two reviewers", 8% "one reviewer" and 7% "three or more reviewers".
Ethical issues are certainly under-represented in documents describing peer review process and editorial policies issued by Italian, Spanish and Croatian law journals. Only 32% of the documents mention conflict of interests, and all other subcategories are present even less than 30%. 
Croatian legal journals are addressing research misconduct and different publication types more than the other two countries. Also, they consider manuscript originality as very important, but some important submission evaluation criteria, reviewer characteristics and peer review outcomes are not present. On the other hand, Spanish legal journals are addressing more reviewer's competences, but some important ethical issues are missing. Italian legal journals take care about the reviewer's competences, academic level and objectivity, as well as anonymity of the peer review process, but expected peer review outcomes and ethical issues should be mentioned in a greater extent.
References
Gutwirth, S. (2009). The evaluation of legal science. The Vl.I.R.-model for integral quality assessment of research in law : what next ? In H. Eisendrath & J. P. Van Bendegem (Eds.), It takes two to do science. The puzzling interactions between science and society (pp. 69–80). VUBPress.
Peruginelli, G., & Faro, S. (2018). Research quality evaluation: The case of legal studies. In The Evaluation of Research in Social Sciences and Humanities (pp. 103–129). Springer Cham.
Siems, M., & Mac Sithigh, D. (2012). Mapping Legal Research. Cambridge Law Journal, 71(3), 651–676.




Peer Review in Legal journals

The received wisdom of online user experience is that text must be kept to an absolute minimum if you are to hold your users’ attention. It came as a shock to us, then, when the long-winded, legally-worded question asking referees to allow us to pass their reviewer activity for Springer Nature journals on to Publons produced an opt-in rate of 56% — substantially higher than the opt-in of 37% for the same service for ORCID offered more succinctly. Based on user experience studies carried out for ORCiD opt-in, we believe that part of the reason for this is that the greater detail given during opt-in improves the opt-in rate.

Do Words Matter? Increasing buy-in by building trust with peer reviewers on Publons and ORCID

Peer review has multiple functions. On the one hand, it helps screening the quality of manuscripts for publication. On the other, it allows authors to increase the value of their manuscripts by constructive feedback. However, measuring the developmental value of peer review is difficult as it requires complete, large-scale manuscript data rarely available. With complete data from ten journals from the Royal Society from 2006 to 2017 in various fields of research, we estimated the effect of the number of reviewers and the reviewer score on manuscript changes throughout peer review. We found that peer review improves manuscripts of moderate initial quality often leading them to receive more citations than manuscripts initially receiving higher reviewer scores. Most beneficial changes include the last sections of manuscripts, i.e., discussion/conclusions, and especially references, whose changes were beneficial in terms of citations. The developmental effect of peer review is marginal whenever more than three reviewers assess the same manuscript. Our findings suggest that peer review can essentially work both as a quality screening and a developmental process.

Peer Review Improves Manuscripts of Moderate Initial Quality. A Study on Ten Journals from the Royal Society (2006-2017)

Finding good peer reviewers is a difficult task. In Decentralized Science1project we are designing and developing a tool to improve the quality, fairness and reliability of academic peer reviewing. Our approach relies in opening peer review, giving transparency to the peer reviewing process using decentralized technologies such as Blockchain. During our ongoing product design research we gained interesting insights about the peer reviewing selection process, and how editors currently deal with it. Our research methods are oriented towards the development of a software tool. We use Lean Design and Agile development principles, favoring fast iterative learning over the precision and completeness of more formal approaches. This contribution shares what we learned in the process about how editors deal with peer reviewer selection: from their need sand complains to their tricks, including some of their confessions. It also explains how we embraced this insights to improve our current prototype design.

The challenges of finding peer reviewers: Insights from our product design research

A presentation of a taxonomy for peer review that will be used to describe the peer review process to authors and readers in a consistent and transparent way.

A Standard Taxonomy for Peer Review

Academic journals provide a key quality-control mechanism in science. Yet, information asymmetries and conflicts of interests incentivize scientists to deceive journals about the quality of their research. How can honesty be ensured, despite incentives for deception? Here, we address this question by applying the theory of honest signaling to the publication process. Our models demonstrate that several mechanisms can ensure honest journal submission, including differential benefits, differential costs, and costs to resubmitting rejected papers. Without submission costs, scientists benefit from submitting all papers to high-ranking journals, unless papers can only be submitted a limited number of times. Counterintuitively, our analysis implies that inefficiencies in academic publishing (e.g., arbitrary formatting requirements, long review times) can serve a function by disincentivizing scientists from submitting low-quality work to high-ranking journals. Our models provide simple, powerful tools for understanding how to promote honest paper submission in academic publishing.

Honest signaling in academic publishing

"Introduction
In this study, we tested the effect of receiving a short individual feedback report compared to a short general feedback report on the agreement between reviewers.

Methods
A total of 42 reviewers at the Norwegian foundation Dam were randomly assigned to receive either a general feedback report or an individual feedback report. The general feedback group received one report before the start of the reviews that contained general information about the previous call the reviewers took part in. In the individual feedback group, the reviewers received two reports, one before the review period (based on the previous call) and one during the period (based on the current call). In the individual feedback group, the reviewers were presented with detailed information on their scoring compared with the review committee as a whole. The main outcomes were the proportion of agreement in the eligibility assessment and the score difference between pairs of reviewers assessing the same application.

Results
A total of 2398 paired reviews were included in the analysis. There was no difference between the two groups in score difference. There was a significant difference between the two groups in the proportion of absolute agreement on whether the application was eligible for the funding program or not, with the general feedback group demonstrating a higher rate of agreement. However, the overall levels of agreement remained critically low in both groups and for both outcomes.

Conclusion
The results did not support our hypothesis but are in line with related studies in journal peer review. The use of a convenience sample and the limited sample size is are important limitations in this study. The low levels of agreement remain a major concern in grant peer review and research to identify factors contributing to this, as well as development and testing of interventions aiming to increase agreement rates, are still needed.

The study was preregistered at OSF.io/n4fq3"

Feedback to improve agreement in grant peer review

For peer review to be sustainable the voluntary effort of reviewing should ideally be shared more or less equally within the community. An unequal distribution of effort could reveal practices of over-exploitation and unfairness, potentially undermining the standards of quality of peer review. Unfortunately, across-journal data are rarely accessible to map such a distribution on a large scale. We used a complete dataset from 157 scholarly journals in various fields of research from various publishers, including more than 1 million authors, and measured the ratio between submissions and provided reviews by the same authors. We found that the 'exploiter' identikit is not the expected one but the following: younger scholars with a few co-authors, submitting preferably to journals of highest impact factors. This can be due either to the ''publish or perish'' culture emphasizing the importance of publications or the editors' tendency of relying more on senior reviewers.

Man, senior, targeting prestigious journals: Is this the identikit of the big fish in the peer review pond? An empirical study on 157 scholarly journals

"In the era of data-driven studies, statistics become more important than ever. Political, economic, environmental, or public health decisions – among many others– may be influenced by the results of a given published work. So, these results need to be trustable, reproducible, and replicable.
However, we are experiencing a lack of trust in statistical methods and, particularly, in statistical significance, as shown by many papers and journal editorials that have highlighted the danger of p-values. But p-values are not dangerous themselves. As we see it, the problem is the correct application of this technique as well as any other statistical approach (no mather if Bayesian, frequentist of fiducial).
In this sense, the process of peer review becomes more important than ever, and specialized reviewers with some statistical background are needed to take care of any paper that is proposed to be published. But How can journals take care of this process? Do the editors consider statical advice? Are they taking into account the reproducibility and replicability of the papers? 
In this work, we study if it is possible to measure if there is an improvement in terms of statistical content in a paper during the peer-review process and provide a series of preliminary results about a dataset of nine journals with five article types from The Royal Society."

From amazing work to I beg to differ: analysis of bioRxiv preprints that received one public comment till September 2019

Downloads

Next from PEERE

Introducing Elsevier

Similar lecture

Changing the shoulders I am standing on: Describing the changes that occurred in publications' reference lists after peer review

From amazing work to I beg to differ: analysis of bioRxiv preprints that received one public comment till September 2019

.css-70qvj9{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-align-items:center;-webkit-box-align:center;-ms-flex-align:center;align-items:center;}Downloads

Next from PEERE

Introducing Elsevier

Similar lecture

Changing the shoulders I am standing on: Describing the changes that occurred in publications' reference lists after peer review

Downloads