Peer review: open sesame?
Open peer review has many different aspects, and is not simply about removing anonymity from the process. Open peer review forms part of the ongoing evolution of an open research system, and the transformation of peer review into a more constructive and collaborative process. The ultimate goal of traditional peer review remains the same – to make sure that the work of authors gets published to an acceptable standard of scientific rigour.
There are different levels of bi-directional anonymity throughout the peer review process, including whether or not the referees know who the authors are but not vice versa (single blind review), or whether both parties remain anonymous to each other (double blind review). Open peer review is a relatively new phenomenon (initiated in 1999 by the BMJ) one aspect of which is that the authors and referees names are disclosed to each other. The foundation of open peer review is based on transparency to avoid competition or conflicts born out through the fact that those who are performing peer review will often be the closest competitors to the authors, as they will tend to be the most competent to assess the research.
The publishing of referee names is a distinct, but related, issue to that of whether or not to make reviewer reports open, which we have a post on tomorrow. In fact, the two are often conflated during discussions of open peer review, and for clarity should be distinguished. Therefore, it is more useful to call the actual process of publishing referee reports ‘open peer review’, while the disclosure of referee names could be termed ‘signed peer review’ to avoid conflation.
To name or not to name?
The debate between signed or not-signed reviews is not to be taken lightly. Early career researchers are perhaps the most conservative in this arena as they may feel afraid that by signing overly critical reviews (i.e., those which investigate the research more thoroughly), they will become targets for retaliatory backlashes from senior figures. The traditional double blind process therefore offers, in theory, a sort of protection for those in junior positions. As well as referee anonymity, author anonymity might be viewed as saving junior authors taking their first steps into the world of academia from public humiliation from more established members of the research community. These potential issues are part of the cause towards a general attitude of conservatism from the research community towards opening up the peer review process.
In a perfect world, we would expect that strong, honest and constructive feedback would be well received by senior researchers, but there is an apprehension that this would not be the case. However, retaliations to referees in such a negative manner are serious cases of academic misconduct, and likely to be dealt with as such. In fact, increasing transparency can only make mitigating or tackling this potential issue easier. Therefore, this perceived danger is highly unlikely to be acceptable or ever actually occur in the current academic system, and if it does can be dealt with through increased transparency.
Openness and accountability
It is not clear how this widely-exclaimed but poorly documented potential abuse of signed-reviews is any different from what would occur anyway in a closed system, as anonymity provides a potential mechanism for referee abuse. On the other hand, by reviewing in a public manner, one could argue that such backlashes are prevented as researchers will not want to do anything to tarnish their reputations in a public forum. In these circumstances, openness becomes a method to hold both referees and authors accountable for their public discourse, as well as makes public the decisions which Editors make on referee and publishing choice.
It has been argued that by disclosing both the author and referee names, detection of scientific misconduct will becomes easier, and that over time the quality of submitted articles will improve. Perhaps most importantly, by the names of those involved in the process being open a measure of transparency is gained into the peer review process, and can provide important context into the subject matters being discussed. Furthermore, by signing reviews, referees are able to obtain credit for their work and for contribution to the development of the scientific record.
Such a system provides a middle-ground between protecting authors from any potential misconduct while making sure that the review content is available for evaluation and reuse. Whether or not referees are identified to the author has been shown in one study to have little effect on the quality of peer reviews, the recommendation regarding publication, or the time taken to review. However, in this instance, reviewers were much more likely to decline to review if they had to disclose their names. The concern that signed peers review leads to less critical reviews is not supported by current analyses, and in fact some have found that the quality of peer reviews actually increases through anonymity. We look forward to seeing continued research into the impact of signed peer review in the future, and in the mean time, see this great review of studies by Hilda Bastian on the advantages versus disadvantages of signed peer review.
At ScienceOpen, we expect peer review to be conducted in a professional and courteous manner, and therefore we publish both signed reviews and author names. This facilitates a continuing dialogue subsequent to the publishing of research, and for us represents an important aspect of providing transparency into the peer review process. We expect the community to be self-regulating in this regard, and to be civil throughout the review process.
This is the third part in a series on the evolution of peer review, with Part 1 being a general overview and Part 2 discussing how to appropriately credit the peer review process.
Research on this important topic is necessary and welcomed. However, in the end, it is an issue of public scientific policy, meaning that ‘we’ – the stakeholders – don’t have the power to decide how things will change in the near future, nor for that sake in the far future. I expect that the current system will prevail for a long time. Perhaps a minority of genuinely open science journals and publishers will establish itself in the long run. The reason being that there are too many non-scientific things on stake for the stakeholders (cf. social science studies).
Hey Jon
Yes, with ‘we stakeholders’ I meant here scholars or scientists of any type, not just empirical or experimental researchers. I am aware, that this is still a somewhat myopic and one-sided view, because on a higher level publishers of books or journals (also the electronic ones) are stakeholders as well. The problem I see is that most scientists don’t get trained well, perhaps aren’t even interested so much in the rules of the (social) game of publishing, leaving free room for the entrepeneurial type of researcher (see the case of Garfield) who defines the stakes. Furthermore, ‘we stakeholders’ are really a very diverse bunch of mostly friendly people (according to my limited experiences, also e.g. on ResearchGate) who nevertheless have quite different views on how open the business of scientific research should be and by what rules (if any, cf. Feyerabend) this game should be played. As long as there will be such a vast diversity – and really I don’t see any convergence of views on the horizon -, it will be easy enough for anyone with the ‘right’ inclination to make profit out of it. It may be that once upon a time there was a fair balance (give and take) between producers and distributors of scientific knowledge and facts, but it seems to me that we lost this balance. Worse: there are clear signs, that some of the tools which have been originally developed by and for the distributors of knowledge, are now being used unreflectively by well-meaning managers of science to control certain parts of the production process (of course, I am especially thinking here of how citation analysis is used nowadays as a source of all kinds of metrics to ‘measure the quality of individual research and researchers’). Isn’t that funny?
I don’t want to make this a generational thing, but I suspect we will see the inherent conservatism (small c) of science make itself felt on this issue. I think we (early career researchers) may have to wait a generation before we start to see meaningful changes. What this means in terms of practice is it might be more productive in the long term to seek to persuade early career researchers of the benefits of the approach and over time it will just become the majority view/practice.
I try and do my part by signing all my reviews. My simple justification (to myself more than anything else) is that I am willing to stand by everything I say in my reviews and would willingly say it, as written, to the author personally. (If I wasn’t I should probably look at what and how I am writing in the review!)
I also take the view that if we try and be fair, constructive and polite in our reviewing then it is actually in ECR’s (and indeed anyone’s ) interests to sign their reviews. I have faith in the scientific community that doing so it will actually benefit the reviewer’s career/reputation to been known as a fair and constructive reviewer more often than it will develop vendettas/negativity from knowing the name of the person who has contributed to a rejection.
I think too often (in my field at least) reviewers hide behind single blind reviewing to make negative, cruel or non-constructive comments. Almost without fail, as a author, I have (eventually!) accepted *what* reviewers have said about my papers, but the things that linger are *how* the reviewers have written comments (bitchy/sarcastic/demeaning) which I would like to think people would not say to someone, in such a way, to their face.
I think signed peer reviews help eliminate what I think of as a peer reviewer being under the mistaken impression they are writing a humorous, scathing restaurant review in a Sunday newspaper supplement and make the whole process more constructive, and *less* personal.
I couldn’t more agree with you. I have been a prolific reviewer for three years, and although I wasn’t forced to sign my reviews, once in a while I did (for good reasons), and I wouldn’t have had any reservations to do it in all cases (both accept and reject). Yes, reviewers should stand for what they say. Nevertheless I understand those younger ones, who are in the midst of building up their carreer (see original contribution) in an ever more competitive environment …
I regret not be able to share your optimism expressed by your statement “… may have to wait a generation before we start to see meaningful changes.” In the best of all worlds it may well take several generations, and it won’t happen without a lot of ‘evangelizing’ at the workfloor, at conferences etc.. Few of the dreams popular in my early career have come true, although personally I thought then they were quite realistic and in some way inevitable. We all underestimated the power of inherent inertia in any big system …
Even in the narrow field of peer reviewing systems and practices we all have quite disparate opinions and recommendations, not all of which can be implemented at the same time (there is only place for one system at a time, all other ones deemed to remain in experimental stage or on someone’s wish list).
By the way, if you want to see the diversity of opinions and recommendation, have a look at Elsevier’s contest, now a couple of years ago. Many submissions, a couple of very good propositions. Alas, since then, I didn’t hear anymore of this initiative, did you?
One factor that you need to take account of is that many journals typically find it difficult to recruit competent reviewers. Any initiatives that place greater demands on the reviewers will make it even more difficult to persuade people to provide what is essentially a free consulting service. My reaction to a journal editor ‘inviting’ me to review a manuscript within two weeks and telling me that I had to sign my review would probably be a two-fingered gesture sometimes associated with the English and Welsh longbowmen at Agincourt.
Indeed recognized and accommodated by several award-winning proposals to Elsevier’s Peer Review Challenge competition from 2012 (cf. http://tinyurl.com/zewxwot). The usual problem will be to get rid of the current systems (John Maynard Keynes), which may take a very long time. See the educational grading systems in use all over the world, some of them (e.g. the German one) dating back to the middle ages (the guilds), although everybody with a modest acquaintance with measurement and statistics will recognize its outdatedness and absurdities. Another example, even more striking to everybody: the layout of the world-wide standard keyboard for computers now even mimicked on smart phones and the like – based on initial considerations which don’t apply any more – and in the face of existing alternative keyboards which provenly allow a 3-fold input rate. My modest conclusion: people learn, society doesn’t, business only if it creates cash. Anybody able to offer a counterexample?
“(R)etaliations to referees in such a negative manner are serious cases of academic misconduct, and likely to be dealt with as such.”
Sexual harassment is a much worse form of misconduct (if not an outright crime) and there are many, many cases in academia where this has not been adequately dealt with. E.g., http://www.nytimes.com/2016/02/03/us/chicago-professor-resigns-amid-sexual-misconduct-investigation.html These sorts of examples make it difficult for the community to have trust that misconduct investigations offer adequate protection.
Many decisions in academia are concentrated in relatively small numbers of scientists (e.g., hiring and funding committees), where individuals with a desire to retaliate could potentially have significant influence, perhaps years after the fact. But that the decision is ultimately made by committee obscures the retaliatory motives. It’s not immediately clear how revealing reviewer’s names prevents that.
The desire to have some level of protection from retaliation is not a crazy one.
Related post on the role of editors in this process: http://neurodojo.blogspot.com/2016/02/editors-are-responsible-when-you-get.html
I feel quite uneasy when I read about this one issue of abuse of identity disclosure of peer reviews as part of open science. We can’t and shouldn’t restrict the discussion of the merits or not of peer reviews to this single issue. IMHO there is much more on stake here. As several scholars (including myself) have already extensively developed at other places, the whole idea and process of peer reviewing is utterly complex, sometimes non-productive and part of non-so-social contract between scholars and publishers. That it is nevertheless working correctly (but very slowly), is a miracle. Well, it doesn’t work correctly as by magic. Everybody knows the experiment with 10-12 reviewers who didn’t remember their own reviews after just one year, and so wrote new reviews of the same papers partly completely contradicting their first reviews. And the social contract I was referring to is a non-zero-sum-game, because publishers let us do the dirty work (of reviewing), how clever! Furthermore, in former days it was common for scientists to travel and visit each other in order to discuss scientific issues. Certainly they didn’t agree all the time, but anyway those discussions took place, unsettled issues were kept alife until theory or experiment clearly voted for one or the other position, etc. Nowadays we have internet, we don’t need to travel to contact any colleague we like, but we prefer not to speak with each other because of fear of abuse!? What world are we living in?