A post-publication peer review success story

In 2016, Dr. Joel Pitt and Prof. Helene Hill published an important paper in ScienceOpen Research. In their paper, they propose new statistical methods to detect scientific fraudulent data. Pitt and Hill demonstrate the use of their method on a single case of suspected fraud. Crucially, in their excellent effort to combat fraud, Pitt and Hill make the raw data on which they tested their method publicly available on the Open Science Framework (OSF). Considering that a single case of scientific fraud can cost institutions and private citizens a huge amount of money, their result is provocative, and it emphasizes how important it is to make the raw data of research papers publicly available.

The Pitt and Hill (2016) article was read and downloaded almost 100 times a day since its publication on ScienceOpen. More importantly, it now has 7 independent post-publication peer reviews and 5 comments. Although this is a single paper in ScienceOpen’s vast index of 28 million research articles (all open to post-publication peer review!), the story of how this article got so much attention is worth re-telling.

Peer review history

The manuscript was submitted and published in January 2016, and the final typeset version of the article was available for download on March 1st. Shortly after this in May 2016, PhD student Chris Hartgerink publicly peer reviewed the article, summarising it as “Interesting research, but in need of substantial rewriting.”

It was after this that the article came to the attention of Prof. Philip B. Stark, an Associate Dean at the University of California, Berkeley, and author of the most highly read article on our platform with over 39,000 views to date!

Prof. Stark runs a course on the theory and application of statistical models. In his course, groups of students replicate and critique the statistical analyses of published research articles using the article’s publicly available raw data. Obviously, for this course to work, Prof. Stark needs rigorous research articles and the raw data used in the article. In this sense, Pitt and Hill’s article on ScienceOpen was an ideal candidate..

The groups of students started their critical replication of the Hill and Pitt article in the Fall semester of 2016 and finished right before the new year.  By getting students to actively engage with research, they gain the confidence and expertise to critically analyse published research.

The Post-Publication Peer Review function on ScienceOpen is usually only open to researchers with more than 5 published articles. This would have normally barred Stark’s groups from publishing their critical replications. However, upon hearing about his amazing initiative, ScienceOpen opened their review function to each of Prof. Stark’s vetted early career researchers. And importantly, since each peer review on ScienceOpen is assigned a CrossRef DOI along with a CC-BY license, after posting their reviews, each member of the group has officially shared their very own scientific publication. 

This also means that each peer review can be easily imported into any user’s ORCID, Publons, and even ImpactStory profiles – the choice is yours!

Public, post-publication peer review works

All of the complete peer reviews from the groups of students can be found below. They all come with highly detailed statistical analyses of the research, and are thorough, constructive, and critical, as we expect an open peer review process to be.

Furthermore, unlike almost every other Post Publication Peer Review function out there, the peer reviews on ScienceOpen are integrated with graphics and plots. This awesome feature was added specifically for Prof. Stark’s course, but note that it is now available for any peer review on ScienceOpen.

• Maurer and Mohanty, who stated that the work was an important demonstration of the use of statistical methods for detecting fraud;
• Hejazi, Schiffman and Zhou, who evaluated the work as comprehensible but largely incomplete;
• Dwivedi, Hejazi, Schiffman and Zhou, who note that the research is a strong advocate for detecting scientific fraud and the use of reproducible statistical methods;
• Stern, Gong and Zhou call the research clever in the application of the techniques t uses to address a pressing problem in science;
• Bertelli, DeGraaf and Hicks think the analysis is convincing and valuable, but with a methodology that could be refined;
• Hung, Sheehan, Chen and Liu evaluated the paper, finding a few minor discrepancies between their own results on those of the published research.

So overall a large variety of findings drawn from the critical replication project, and each of which individually greatly enhance the published research.

Aftermath

Many of the peer reviews focused on a specific assumption that the Pitt and Hill article made about how sets of numbers are distributed. We talked to Dr. Helene Hill for comment. She noted that Dr. Pitt was working on the distribution issue noted by many reviewers, and that she was happy to see her research received such critical attention.

She notes that critical reception was: 

just what I was hoping for — for our paper to be a model for posting and analyzing data.

Prof. Stark said there are several things that this project accomplished for him in terms of getting students actively involved in peer review:

• Get students thinking about alternative models for scholarly publication;
• Get students thinking about reproducibility and open science;
• Get students to work collaboratively on a data analysis project that involves thinking hard about the underlying science;
• Get students to register with ORCID;
• Get students to post their analyses on GitHub so that their own work is reproducible/extensible;
• Get students their first scientific publication.

For another step of Open Science brilliance, the reviews themselves sought to be completely reproducible, with the code for all the students’ calculations is available on GitHub (eg here and here)!

Prof Stark said:

I think it’s remarkable that ScienceOpen extended the platform and your process to make this possible, including allowing people with fewer than 5 publications associated with their ORCID to submit reviews, figuring out how to allow figures in reviews, and that you are working on allowing reviews with multiple authors. ScienceOpen was really a partner in making this exercise possible.

We also asked some of the students how they found the peer review exercise, many of whom praised Pitt and Hill’s efforts on making the research as reproducible as possible.

Stephanie DeGraaf: “The Hill and Pitt paper made the data publicly available and explained their analyses thoroughly enough so that we could reproduce all of their results. The paper focused on a really fascinating topic of testing for fraudulent data, and I really enjoyed thinking about how to tackle the problem in a statistically valid way. I found it really interesting to see that even though all of us in the class were reviewing this same paper, we all had different perspectives, criticisms, and ideas for other ways to investigate the researchers’ claims.”

Aaron Stern: “The Hill and Pitt paper was a great choice for the purpose of the course; not only did the authors employ interesting and novel statistical methods for us to critique, but they also were tackling a very important issue in science — namely, fraud. While we agreed with the paper’s conclusions, we found a number of scenarios where their approach applied to new datasets could result in false positives; i.e., their methods could impugn an innocent researcher. Thus, it’s important to validate these methods thoroughly in order to avoid hurting innocent scientists.”

Kenneth Hung: “It is not very agreed, among statisticians, what constitutes replications and reproductions. In writing this review, it gave my new and broader perspectives, in comparison to the post-selection inference background I came from, as well as common tools in practice for detecting scientific frauds.”

Nima Hejazi: “Constructing a review of the paper required extensive collaboration, the use of open source software tools, and the leveraging of statistical and domain knowledge for the purpose of detecting fraudulent science – all in all an experience that demonstrated quite well the challenges of working with real-world data and making use of open-access publishing platforms.”

Great success!

So we definitely count this as a major success story on several levels.

1. Students gained the experience in performing analyses for the sake of reproducible research.
2. Students also gained the skills and confidence to perform rigorous and constructive peer reviews in public.
3. Post-publication peer review works just as well, if not better, than traditional peer review.
4. Openness facilitates recognition and reward for peer review, which is crucial for those just starting their research careers.
5. This whole exercise shows that just because research has been published, it does not mean that critical evaluation of it should stop.

So, what is the next step? Well, anyone who has an ORCID can peer review any of 28 million research articles on our platform. They don’t have to be detailed statistical analyses – they can be critical commentaries, additional notes and context, or what your own related research says.

The point is the choice is yours. The reason is that you help to improve the context and progress of your research field in the open, while improving your research skills and receiving recognition and credit for doing so.