Blog
About

In:  Peer Review  

Peer review: open sesame?

Open peer review has many different aspects, and is not simply about removing anonymity from the process. Open peer review forms part of the ongoing evolution of an open research system, and the transformation of peer review into a more constructive and collaborative process. The ultimate goal of traditional peer review remains the same – to make sure that the work of authors gets published to an acceptable standard of scientific rigour.

There are different levels of bi-directional anonymity throughout the peer review process, including whether or not the referees know who the authors are but not vice versa (single blind review), or whether both parties remain anonymous to each other (double blind review). Open peer review is a relatively new phenomenon (initiated in 1999 by the BMJ) one aspect of which is that the authors and referees names are disclosed to each other. The foundation of open peer review is based on transparency to avoid competition or conflicts born out through the fact that those who are performing peer review will often be the closest competitors to the authors, as they will tend to be the most competent to assess the research.

The publishing of referee names is a distinct, but related, issue to that of whether or not to make reviewer reports open, which we have a post on tomorrow. In fact, the two are often conflated during discussions of open peer review, and for clarity should be distinguished. Therefore, it is more useful to call the actual process of publishing referee reports ‘open peer review’, while the disclosure of referee names could be termed ‘signed peer review’ to avoid conflation.

To name or not to name?

The debate between signed or not-signed reviews is not to be taken lightly. Early career researchers are perhaps the most conservative in this arena as they may feel afraid that by signing overly critical reviews (i.e., those which investigate the research more thoroughly), they will become targets for retaliatory backlashes from senior figures. The traditional double blind process therefore offers, in theory, a sort of protection for those in junior positions. As well as referee anonymity, author anonymity might be viewed as saving junior authors taking their first steps into the world of academia from public humiliation from more established members of the research community. These potential issues are part of the cause towards a general attitude of conservatism from the research community towards opening up the peer review process.

In a perfect world, we would expect that strong, honest and constructive feedback would be well received by senior researchers, but there is an apprehension that this would not be the case. However, retaliations to referees in such a negative manner are serious cases of academic misconduct, and likely to be dealt with as such. In fact, increasing transparency can only make mitigating or tackling this potential issue easier. Therefore, this perceived danger is highly unlikely to be acceptable or ever actually occur in the current academic system, and if it does can be dealt with through increased transparency.

Openness and accountability

It is not clear how this widely-exclaimed but poorly documented potential abuse of signed-reviews is any different from what would occur anyway in a closed system, as anonymity provides a potential mechanism for referee abuse. On the other hand, by reviewing in a public manner, one could argue that such backlashes are prevented as researchers will not want to do anything to tarnish their reputations in a public forum. In these circumstances, openness becomes a method to hold both referees and authors accountable for their public discourse, as well as makes public the decisions which Editors make on referee and publishing choice.

Source
Source

It has been argued that by disclosing both the author and referee names, detection of scientific misconduct will becomes easier, and that over time the quality of submitted articles will improve. Perhaps most importantly, by the names of those involved in the process being open a measure of transparency is gained into the peer review process, and can provide important context into the subject matters being discussed. Furthermore, by signing reviews, referees are able to obtain credit for their work and for contribution to the development of the scientific record.

How can conflicts of interest be detected if referees and authors are anonymous? Image: Hilda Bastian, CC NC
How can conflicts of interest be detected if referees and authors are anonymous? Image: Hilda Bastian, CC NC

Such a system provides a middle-ground between protecting authors from any potential misconduct while making sure that the review content is available for evaluation and reuse. Whether or not referees are identified to the author has been shown in one study to have little effect on the quality of peer reviews, the recommendation regarding publication, or the time taken to review. However, in this instance, reviewers were much more likely to decline to review if they had to disclose their names. The concern that signed peers review leads to less critical reviews is not supported by current analyses, and in fact some have found that the quality of peer reviews actually increases through anonymity. We look forward to seeing continued research into the impact of signed peer review in the future, and in the mean time, see this great review of studies by Hilda Bastian on the advantages versus disadvantages of signed peer review.

At ScienceOpen, we expect peer review to be conducted in a professional and courteous manner, and therefore we publish both signed reviews and author names. This facilitates a continuing dialogue subsequent to the publishing of research, and for us represents an important aspect of providing transparency into the peer review process. We expect the community to be self-regulating in this regard, and to be civil throughout the review process.

This is the third part in a series on the evolution of peer review, with Part 1 being a general overview and Part 2 discussing how to appropriately credit the peer review process.

23 thoughts on “Peer review: open sesame?”

  1. Research on this important topic is necessary and welcomed. However, in the end, it is an issue of public scientific policy, meaning that ‘we’ – the stakeholders – don’t have the power to decide how things will change in the near future, nor for that sake in the far future. I expect that the current system will prevail for a long time. Perhaps a minority of genuinely open science journals and publishers will establish itself in the long run. The reason being that there are too many non-scientific things on stake for the stakeholders (cf. social science studies).

    1. Hey Paul,

      Thanks for your comment, What is it precisely that you think researchers don’t have the power to change? (is that who you meant by ‘we’?) It’s an interesting comment, and I was wondering if you could elaborate further?

      Jon

  2. Hey Jon
    Yes, with ‘we stakeholders’ I meant here scholars or scientists of any type, not just empirical or experimental researchers. I am aware, that this is still a somewhat myopic and one-sided view, because on a higher level publishers of books or journals (also the electronic ones) are stakeholders as well. The problem I see is that most scientists don’t get trained well, perhaps aren’t even interested so much in the rules of the (social) game of publishing, leaving free room for the entrepeneurial type of researcher (see the case of Garfield) who defines the stakes. Furthermore, ‘we stakeholders’ are really a very diverse bunch of mostly friendly people (according to my limited experiences, also e.g. on ResearchGate) who nevertheless have quite different views on how open the business of scientific research should be and by what rules (if any, cf. Feyerabend) this game should be played. As long as there will be such a vast diversity – and really I don’t see any convergence of views on the horizon -, it will be easy enough for anyone with the ‘right’ inclination to make profit out of it. It may be that once upon a time there was a fair balance (give and take) between producers and distributors of scientific knowledge and facts, but it seems to me that we lost this balance. Worse: there are clear signs, that some of the tools which have been originally developed by and for the distributors of knowledge, are now being used unreflectively by well-meaning managers of science to control certain parts of the production process (of course, I am especially thinking here of how citation analysis is used nowadays as a source of all kinds of metrics to ‘measure the quality of individual research and researchers’). Isn’t that funny?

  3. I don’t want to make this a generational thing, but I suspect we will see the inherent conservatism (small c) of science make itself felt on this issue. I think we (early career researchers) may have to wait a generation before we start to see meaningful changes. What this means in terms of practice is it might be more productive in the long term to seek to persuade early career researchers of the benefits of the approach and over time it will just become the majority view/practice.
    I try and do my part by signing all my reviews. My simple justification (to myself more than anything else) is that I am willing to stand by everything I say in my reviews and would willingly say it, as written, to the author personally. (If I wasn’t I should probably look at what and how I am writing in the review!)
    I also take the view that if we try and be fair, constructive and polite in our reviewing then it is actually in ECR’s (and indeed anyone’s ) interests to sign their reviews. I have faith in the scientific community that doing so it will actually benefit the reviewer’s career/reputation to been known as a fair and constructive reviewer more often than it will develop vendettas/negativity from knowing the name of the person who has contributed to a rejection.
    I think too often (in my field at least) reviewers hide behind single blind reviewing to make negative, cruel or non-constructive comments. Almost without fail, as a author, I have (eventually!) accepted *what* reviewers have said about my papers, but the things that linger are *how* the reviewers have written comments (bitchy/sarcastic/demeaning) which I would like to think people would not say to someone, in such a way, to their face.
    I think signed peer reviews help eliminate what I think of as a peer reviewer being under the mistaken impression they are writing a humorous, scathing restaurant review in a Sunday newspaper supplement and make the whole process more constructive, and *less* personal.

    1. That is an excellent perspective to have, Simon, imo. I always try and think that too – would I say this if the person were standing right there in front of me? I see no reason why peer review shouldn’t be conducted to the same degree of professionalism.

      I feel the same way about how being open reflects upon an individual too. I think there are rare cases where there are backlashes, but we shouldn’t let outliers in these cases hold sway over something in which the benefits are so clear.

  4. I couldn’t more agree with you. I have been a prolific reviewer for three years, and although I wasn’t forced to sign my reviews, once in a while I did (for good reasons), and I wouldn’t have had any reservations to do it in all cases (both accept and reject). Yes, reviewers should stand for what they say. Nevertheless I understand those younger ones, who are in the midst of building up their carreer (see original contribution) in an ever more competitive environment …

    1. I actually see open peer review as a way of building my reputation, as a junior researcher. I want all of the outputs of my work as a researcher to be recognised, and I can’t do that in a closed system. I agree that referees should stand by what they say – I see openness as a mechanism to increase the quality of reviews by based on this, in fact.

  5. I regret not be able to share your optimism expressed by your statement “… may have to wait a generation before we start to see meaningful changes.” In the best of all worlds it may well take several generations, and it won’t happen without a lot of ‘evangelizing’ at the workfloor, at conferences etc.. Few of the dreams popular in my early career have come true, although personally I thought then they were quite realistic and in some way inevitable. We all underestimated the power of inherent inertia in any big system …

    Even in the narrow field of peer reviewing systems and practices we all have quite disparate opinions and recommendations, not all of which can be implemented at the same time (there is only place for one system at a time, all other ones deemed to remain in experimental stage or on someone’s wish list).

    By the way, if you want to see the diversity of opinions and recommendation, have a look at Elsevier’s contest, now a couple of years ago. Many submissions, a couple of very good propositions. Alas, since then, I didn’t hear anymore of this initiative, did you?

  6. One factor that you need to take account of is that many journals typically find it difficult to recruit competent reviewers. Any initiatives that place greater demands on the reviewers will make it even more difficult to persuade people to provide what is essentially a free consulting service. My reaction to a journal editor ‘inviting’ me to review a manuscript within two weeks and telling me that I had to sign my review would probably be a two-fingered gesture sometimes associated with the English and Welsh longbowmen at Agincourt.

    1. Hi Peter, I’m the peer review manager for ScienceOpen, and I couldn’t agree more with your comments. I have this to add. As Jon has done a great job in pointing out through this series, one of the motivating ideals behind ScienceOpen is to provide a community sourced public post publication peer review model. Instead of publishers deciding who reviews what, authors and researchers with stakes in making research move swiftly in the direction of truth, would come onto ScienceOpen, provide their expertise, and review the things that they can. In doing so, they could develop a reputation for their good work. If the model took off, the ScienceOpen development team could implement cool features for reviewers such as badges for competency, and our editorial staff could reward reviewers with heavily discounted APCs. Unfortunately, it’s not in the power of the ScienceOpen editorial staff to magically make people contribute and use the tools that our software development team built. That means that we spend a good amount of time and energy trying to persuade people to review a manuscript. And again, the idea is that by publishing first, and letting the community care for the peer review, research and development could more more quickly in the right direction.

    2. Yes, I agree that finding appropriate referees for articles can be a difficult task, and one compounded by the fact that there is little incentive system. So a dual approach to resolving this is to not force people to review papers they don’t want to, and instead let them community decide what to do in an open system! Layer an incentive structure like that described in the second post of this series, and you might actually see an increase in the ‘recruitment’ of competent referees.

  7. Indeed recognized and accommodated by several award-winning proposals to Elsevier’s Peer Review Challenge competition from 2012 (cf. http://tinyurl.com/zewxwot). The usual problem will be to get rid of the current systems (John Maynard Keynes), which may take a very long time. See the educational grading systems in use all over the world, some of them (e.g. the German one) dating back to the middle ages (the guilds), although everybody with a modest acquaintance with measurement and statistics will recognize its outdatedness and absurdities. Another example, even more striking to everybody: the layout of the world-wide standard keyboard for computers now even mimicked on smart phones and the like – based on initial considerations which don’t apply any more – and in the face of existing alternative keyboards which provenly allow a 3-fold input rate. My modest conclusion: people learn, society doesn’t, business only if it creates cash. Anybody able to offer a counterexample?

  8. There is a person in my research area, let’s call him Bob. Bob is more senior than me. Bob sits on many grant panels and reviews for high profile journals. I do not do either of those. I am junior (as people seem to love to remind me). It is far more likely that Bob will evaluate my work than vice versa. Be it for a grant or a publication. So let’s say I am reviewing one of Bob’s papers. Bob knows me, and knows from conversations that I tend to disagree with him about our shared science interest. So I write a review of Bob’s paper listing all the issues I have with that paper. What’s the upside to signing my name? Is Bob a vengeful person? Maybe. If he was and decided to take it out on a grant application of mine later, would I even know? Would anyone know? The paylines are so low and stock critiques so numerous how would one even be caught “unfairly” reviewing a paper or grant? Maybe that means I shouldn’t worry about it and sign my name. Or perhaps being on the business end of a power differential I should play things cautiously.

    1. This ‘backlash’ or ‘retaliatory’ aspect has come up now multiple times as a response to why we shouldn’t have open peer review. I ask the simple question: why is this being used as an argument against open peer review, instead of an argument against facilitating a system where vengeful people are allowed to have any sort of power? I am aware that such figures exist, although the extent to which they might unfairly govern peer review has not been documented to my knowledge, but we should be building structures where [hypothetical] people like this are removed from positions of power, rather than being allowed to impose vengeance from above.

  9. “(R)etaliations to referees in such a negative manner are serious cases of academic misconduct, and likely to be dealt with as such.”

    Sexual harassment is a much worse form of misconduct (if not an outright crime) and there are many, many cases in academia where this has not been adequately dealt with. E.g., http://www.nytimes.com/2016/02/03/us/chicago-professor-resigns-amid-sexual-misconduct-investigation.html These sorts of examples make it difficult for the community to have trust that misconduct investigations offer adequate protection.

    Many decisions in academia are concentrated in relatively small numbers of scientists (e.g., hiring and funding committees), where individuals with a desire to retaliate could potentially have significant influence, perhaps years after the fact. But that the decision is ultimately made by committee obscures the retaliatory motives. It’s not immediately clear how revealing reviewer’s names prevents that.

    The desire to have some level of protection from retaliation is not a crazy one.

    1. Hey Zen,

      Thanks for your comment. I agree entirely that sexual harassment is misconduct at its worst, and should be dealt with as a criminal case. I think what you discuss though aren’t potential issues with open peer review, but more problems in that we have a system which allows shitty behaviour like this to go un-checked.

      In terms of how signing reviews can prevent retaliation, I don’t think it will work in a secretive system. However, the question is to what extent do these retaliatory backlashes happen in the current system, and should we let potentially fringe issues prevent what is otherwise a progressive systemic move with clear benefits (those outlined in this post and others). We should also consider what this means for our higher echelons of science – why are we allowing fear of retaliation from senior colleagues to govern what we do? In what way is these even marginally acceptable? We should all be working so that such vengeance-based power gradients are eliminated, and those who abuse them are removed from their positions.

      tl,dr: I agree that protection is important, but by pointing the finger at openness, we’re asking the wrong questions and blaming the victim and the process, not the perpetrators.

  10. B and Zen raise the issue that concerns me when I hear proposals for 100% open, named reviews. There are grey areas where proof is impossible; nearly everyone knows someone they won’t work with, or be alone with, or whatever it might be — even though they couldn’t *prove* a thing in a formal context. I think we need to retain the option of anonymity, and live with the fact that it will sometimes be abused*, in order not to lose marginalized voices.

    *Abuse of anonymity can be reigned in rather better than, say, abusive PIs. If anonymous reviews are not professional and do not restrict themselves to the science, they can be flagged/deleted/whatever. I’d rather see more effort put into building a culture in which it’s not necessary to attach someone’s home address to their comments in order to get them to play by the rules.

    1. What I don’t like about this, and as I’ve mentioned elsewhere, is that it seems to say that a culture of vengeance, retaliation, and power dynamics is allowed. If this system is so prevalent, then why aren’t we doing something about it? And if it’s so uncommon that it’s more of a fringe issue, then why are we letting it stand in the way of progress? There are always going to be grey areas, and even more in a closed world, but the way of dealing with them isn’t to let them continue unchecked. Yes, some people are going to be assholes, but they should be exposed and dealt with as such, while we generate support structures to avoid issues with them as much as possible.

  11. I feel quite uneasy when I read about this one issue of abuse of identity disclosure of peer reviews as part of open science. We can’t and shouldn’t restrict the discussion of the merits or not of peer reviews to this single issue. IMHO there is much more on stake here. As several scholars (including myself) have already extensively developed at other places, the whole idea and process of peer reviewing is utterly complex, sometimes non-productive and part of non-so-social contract between scholars and publishers. That it is nevertheless working correctly (but very slowly), is a miracle. Well, it doesn’t work correctly as by magic. Everybody knows the experiment with 10-12 reviewers who didn’t remember their own reviews after just one year, and so wrote new reviews of the same papers partly completely contradicting their first reviews. And the social contract I was referring to is a non-zero-sum-game, because publishers let us do the dirty work (of reviewing), how clever! Furthermore, in former days it was common for scientists to travel and visit each other in order to discuss scientific issues. Certainly they didn’t agree all the time, but anyway those discussions took place, unsettled issues were kept alife until theory or experiment clearly voted for one or the other position, etc. Nowadays we have internet, we don’t need to travel to contact any colleague we like, but we prefer not to speak with each other because of fear of abuse!? What world are we living in?

  12. Pingback: Café com LSD

Leave a Reply

Your email address will not be published. Required fields are marked *