A Plan to Improve the Peer Review Process
Peer review is foundational to academic research and the influence of each academic. The essence of peer review is that one scholars work must be vetted by other scholars, usually two plus an editor, before it can be published. Peer-reviewed publications thus carry appropriately more weight in the eyes of those who are in each field or who are looking into the field as outsiders.
As the peer review process becomes mingled with other drivers, such as the drive to have commercially viable journals and the drive for scholars to have resumes that contain as many peer-reviewed publications as possible, the integrity of the process could be compromised. Indeed, the problems with peer review have been discussed by many authors, and “Peer Review” is even the title of a novel [https://www.amazon.com/dp/B08QDJ7GLZ] whose cover shows a person approaching a foreboding castle above a banner marked with omissa spe, Latin for “abandon hope”.
Ideas to Prompt Iteration and Evolution of the Peer Review Process
Below are four ideas that Academic Influence’s President, Jed Macosko, and two of his colleagues (Rob Sheldon and Robert Preisser) have proposed as the first steps toward improving the peer review process. Others have suggested these same ideas or slight variants, but we at AcademicInfluence.com have found this collection of ideas helpful.
- Right now, peer review usually hides the reviewer and exposes the authors. A better system (which a handful of journals use, e.g. https://www.semantic-web-journal.net/) is to expose both the reviewers (with their reviews) and the authors. The transparency of exposing both parties may lower the number of reviewers willing to review a given paper, but it improves (in our opinion) the quality of each review and adds a body of literature (the sum of all reviews) that has not yet been available to the general public. This body of literature can be very helpful for scholars and non-scholars alike as they try to better understand any controversies in a given field.
- An even more radical idea is to only expose the reviewer (not the authors). In the end, obviously, when a paper is published the authors’ names and affiliations will become public knowledge, but there is an argument that keeping this information hidden until after the review process will improve the fairness of each review. On one hand, famous authors will not be given a free pass and unknown authors will not be given an abnormally high bar to hurdle.
- All papers, even rejected papers, and their reviews should be available in some searchable format. There could be a time delay in this process if authors feel like they need to keep their rejected results under wraps until they have time to re-write their paper and resubmit it (obviously, we don’t want an author to get scooped just because their rejected paper ends up online). But, perhaps a time delay is not that essential since assigning priority of invention/idea/innovation would be easy if the original authors’ results are plainly visible in a rejected paper database.
- Perhaps mandate that any verifications or invalidations of any article previously published in that same journal must be given first priority for review and publication. This would help with the problem that such confirmation studies aren’t “sexy” so don’t get published often (if ever). It would also make authors think twice about fudging data to make the paper more shocking to get it published, while knowing the odds of getting caught are slim. Publications that have such a policy that encourages other scientists to confirm their published results should have a higher rating amongst the more predatory journals.
Responses to Criticism
A common objection to the idea of exposing the reviewer (our second idea listed above) is that this plan makes it harder for reviewers to be critical, particularly when they feel the manuscript they are reviewing has been submitted by a powerful person in their field yet does not deserve publication in the journal where it was submitted, here are some responses:
- There will always be journals that still have anonymous reviewers and those who are concerned about repercussions and who still want to help in the peer review process can review for those journals. But the goal is to have as many journals as possible move away from the anonymous-reviewer model.
- It’s true that an untenured professor (or someone seeking a position or someone who requires grants to keep their job) would risk their employment by pointing out a flaw in a more senior scientist’s manuscript. However, the bulk of reviewers have jobs that won’t disappear when a grant application is rejected, or already have tenure, or are in jobs that don’t come up for tenure and thus don’t require outside people who write letters of support. So, a simple solution is to give an untenured, or similarly vulnerable person, a choice of remaining anonymous. Certainly, this would mean that a more senior scientist who receives an unfavorable, anonymous review will be more able to narrow down the identity of the reviewer, but the untenured professor will have to take into consideration how easy they would be to identify (based on how many other untenured professors are in their field) before they accept the job of reviewing a paper. There’s not much harm in doing fewer reviews until one gets through the tenure decision. A system with two reviewers per manuscript relies on people reviewing two papers for every one they submit, but that’s just a lifetime average and untenured professors can plan to make up for a lower level of reviewing by reviewing more papers after their tenure decision. And the proof that one is “vulnerable” is straightforward: are you untenured, on “soft” money, or someone who is currently looking for their first job as a professor or principal investigator? These are objective questions that are fairly easy to verify, and thus the exceptions to the policy of disclosing reviewers are easy to contain.
Conclusion
These four ideas for peer review (and two responses to a common criticism of our second idea) are, as we mentioned, not unique to us. But we present them here in hopes that more people will read about these ideas and begin to implement them or insist that the gatekeepers of peer-reviewed journals will implement them. Peer review is a wonderful and important process, but it would benefit from changes that help it keep in line with the new drivers that have sprung up alongside peer-reviewed publications over the last several decades.