In:  Peer Review  

A new gold standard of peer review is needed

How can something exclusive, secretive, and irreproducible be considered to be objective? How can something exclusive, secretive, and irreproducible be considered as a ‘gold standard’ of any sort?

Traditional, closed peer review has these traits, but yet for some reason held in esteem as the most rigorous and objective standard of research and knowledge generation that we have. Peer review fails peer review, and its own test of integrity and validation, and is one of the greatest ironies of the academic world.

What we need is a new standard of peer review that is suitable for a Web-based world of scholarly communication. This is to help accommodate the increasingly rapid communication of research and new sources of information, and bring peer review out of the dark (literally) ages and into one which makes sense in a world of fast, open, digital knowledge dissemination.

What should a standard for peer review look like in 2017?

The big test for peer review, and any future version of it, is how does the scientific community apply its stamp of approval?

At the moment, management of peer review is done through a journal-based system controlled by publishers and rarely by the researchers themselves. Peer review and the validation of research should not be a tool wielded by private corporations with an enormous conflict of interest (i.e., financial gain), but instead should be a system managed by and for the research community.

We as the scientific community should be thinking more about the management, functionality, and process of peer review. It is our collective responsibility that peer review is held to the same high standards of integrity that we do for research. This cannot be done if it is a process out of our control. All of the symptoms of traditional peer review, and its failing as any objective standard, can be, at least in part, traced back to this issue.

We are in a time where demonstrations like ScienceMarch are showing that research communities are powerful all around the world in making real changes. They are standing up for the things that are important to them, like research funding, Open Access, fairness and equality in research, greater integration of research into policy.

So why not make a stand for peer review too? We can do better as a global community.

Is ‘open peer review’ the solution?

Much ink has been spilled about the various aspects of ‘open peer review’. Different parts of this include publishing the referee reports, making the process more inclusive to contribution, and whether or not referees should be identified or anonymous. The dust has by no means settled on these discussions yet, but in terms of a standard, open peer review seems to fit the bill better than the traditional model.

An example of open peer review on ScienceOpen (link)

Open peer review is inclusive, as anyone can contribute to it, should they wish to. Open peer review is non-secretive, as you can see what is written and by whom. Open peer review is reproducible, because you can track the process from manuscript, through review sessions, and updated versions, as part of an iterative process.

The whole process is more valid because you can see what referees say, who said it, what the changes made were as a result, if any. Any good peer review is a critical dialogue between authors, referees, and the Editor or editorial board. That discussion adds important context to research, irrespective of the differences between a submitted article and the final published version, and deserves to be published alongside articles.

The main result of this, however, is that the stamp of approval for research is not just provided in secret by an exclusive dialogue, and with journal title acting as some sort of vague proxy for this. It is provided by the community who decides to contribute to a piece of work. The value and prestige of research is conferred through an objective, accountable, and community-wide process.

That sounds much more like the sort of standard that peer review should be aiming for. And the great thing is, it already exists, in one form or another.

What can ScienceOpen bring to the table?

We are not the only pioneers of open peer review. Other platforms like F1000 Research and Publons are all helping to reform parts of the peer review process. There most likely isn’t one single way to fix what many consider to be a ‘broken’ peer review system, and we are happy to be part of the broader ecosystem that is working towards that.

Some key things we already offer are:

  • Integration with and moderation through ORCID
  • A four-part star-rating system for articles
  • A corpus of 29 million published research articles, including preprints, all available for open peer review
  • An ‘Invite to review’ function on all articles (see below)
  • A requirement for referees to be identified (via ORCID)
  • Publication of all referee reports
  • Published reports all have a CC BY license and a DOI, making them fully equivalent to any Open Access research articles
  • Users can create thematic collections, like a journal, to provide community sub-spaces for peer review
You can even peer review President Barack Obama, should you wish (link)

But what more can we be doing for you, the research communities? We are upgrading and innovating our platform all the time, and more than happy to take on suggestions on how we can be part of improving peer review for researchers, for scholarly communication, and to make an overall more objective system of knowledge generation for everyone.

2 thoughts on “A new gold standard of peer review is needed”

  1. “Any good peer review is a critical dialogue between authors, referees, and the Editor or editorial board.” —
    “The main result of this, however, is that the stamp of approval for research is not just provided in secret by an exclusive dialogue, and with journal title acting as some sort of vague proxy for this.”

    To my 25+ years of taking part in this process, both as author and as reviewer, traditional peer review is not even an “exclusive dialogue”, it is an “exclusive monologue”. (After a delay of many weeks), the standard stance taken by the anonymous reviewer is “I am right, and the author is wrong”, and (again several weeks later) the standard stance of the author is “I was wrong, the reviewer is right, here are my corrections”. My personal experience is that 95 out of 100 attempts by the author to argue why his/her original opinion really is correct, despite objections by the reviewer, never get anywhere, particularly not to a change of opinion on the side of the reviewer. Typically, the editor (and this is the only significant action I ever see from editors) then calls for a third, adjudicative reviewer, and this then settles a debate which never was a debate but simply a clash of different opinions. (For a true debate to happen here, also the time delays are far too long.)
    Why is this so? Simple: The situation is highly asymmetric by construction.The author meticulously has to justify every single statement (to convince a probably-not-so-well-meaning reviewer, and to get the paper published), while the anonymous reviewer can make statements that are backed up by little or no evidence (in the style of “this is commonly known”, “this is textbook knowledge”) — w/o any consequences, because of anonymity. And there is no incentive for the reviewer anyway, to put in more effort (e.g. in terms of providing actual arguments or even references), since the review is done in secret, for free, and in the “spare time” of the reviewer.

  2. Hi Bernd,

    Thanks for your comment. While I don’t think this happens *all* the time, it is certainly symptom of the system which you and this post describe. This is just one of the reasons why we believe a new standard of peer review is needed,


Leave a Reply

Your email address will not be published. Required fields are marked *